assumptions about the build environment.
 

On Fri, Sep 21, 2012 at 09:07:57PM +0100, Ben Hutchings wrote:
> On Fri, Sep 21, 2012 at 08:26:24PM +0100, peter green wrote:
> > Some time ago I found that a package (I think it was openjdk but I
> > don't remember for sure) which relied on uname -r such that linux32
> > had to be used to build it in an i386 chroot on an amd64. However
> > since then I'm pretty sure i've seen similar cases with other
> > packages on other architectures being treated as bugs.
>
> I think confusion between kernel vs userland architecture is so
> widespread that we should consider this a necessary part of doing a
> native build.

Please, don't. This breaks cross building. Any use of uname during build
other than for logging purposes is a bug.

> (It should preferably be fixed upstream, to benefit users who build
> from either Debian or upstream source on a 32/64 environment. But I
> don't know a simple way to find the 'primary userland architecture'
> that is not distribution-specific.)

There's more than just 32/64.

Try this: amd64 kernel, i386 host (and compiler), armel chroot, multiarch
with armhf. What should uname report?

And this is not even an artificial example: you can't really buy a real i386
machine anymore, yet most semi-proprietary (maemo, android) SDKs ship only
i386 binaries, if you want to develop a Debian guest you'd better use armhf
instead of armel, and semi-emulated scratchbox style builds can take 44
seconds compared to 8 hours native[1].

I don't do android so never went 4-way, but played with wine on armhf
earlier this very week... thanks to how wine is set up in wheezy, that's
3-way. The result of uname wouldn't be too meaningful.

In a multiarch world, you can't speak of "environment" for more than a
single executable[2].

One of portable ways to check your target platform is:
$CC -dumpmachine
(assuming a typical build scheme).



[1]. A not so new amd64 box with make -j6 vs armel -j1, on C++ code that
caused a swappeathon on the latter.

[2]. And even that only because ELF doesn't support fat binaries.

--
Copyright and patents were never about promoting culture and innovations;
from the very start they were legalized bribes to give the king some income
and to let businesses get rid of competition. For some history, please read
https://en.wikipedia.org/wiki/Statute_of_Monopolies_1623

assumptions about the build environment.
 

On Fri, Sep 21, 2012 at 08:26:24PM +0100, peter green wrote:
> I just discovered that on my beagleboard XM (under armhf sid) nacl
> (which previously build on a debian experimental armhf buildd but
> not a debian unstable armhf buildd) will build if /sys is mounted
> but will not build if it is not mounted. Can packages assume that
> /sys will be mounted in the build environment or not?

By default, you get /proc, /dev/pts and /sys mounted. Unless the
buildd admin specifically configured they system differently than
the defaults (/etc/schroot/buildd/fstab).

> IIRC it is generally established that packages are not allowed to
> rely on an internet connection during build but if one is present
> are they allowed to assume it's non-broken. I recently came accross
> a package ( sslh ) which fails to build in the presense of nxdomain
> hijacking. Is that a bug?

You are not supposed to rely on any network connectivity during a
package build. If it's present, that's just happenstance; it's
not guaranteed to be present and/or functional, and you should not
be using it under any circumstances. Local loopback is OK though
for e.g. unit testing services.

Just for the record, I'm planning on adding support for
unshare(CLONE_NEWNET) in schroot post wheezy. This will allow
the buildd (sbuild) to request that networking be explicitly
turned off (bar localhost) during a package build. This will
break any buggy packages which are relying on networking during
a build.


Regards,
Roger

--
.'`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools
`- GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120921211314.GB18133@codelibre.net">http://lists.debian.org/20120921211314.GB18133@codelibre.net

assumptions about the build environment.
 

On Fri, Sep 21, 2012 at 08:26:24PM +0100, peter green wrote:
> While working on debian one thing I have not managed to find is
> documentation on what packages can and can't assume about the build
> environment. Does such documentation exist and if not should it be
> created.

One thing that is at least documented in policy is that the only
packages that it can rely on being installed are essential,
build-essential and the packages's build-depends.

> Some specific cases i'm wondering about:
>
> I just discovered that on my beagleboard XM (under armhf sid) nacl
> (which previously build on a debian experimental armhf buildd but
> not a debian unstable armhf buildd) will build if /sys is mounted
> but will not build if it is not mounted. Can packages assume that
> /sys will be mounted in the build environment or not?

As far as I know, on all the buildds /sys, /proc, /dev/pts and
/dev/shm are available and we get complaints if some of them
aren't there. At least /proc and /dev/pts will frequently results
in errors, I think there are also at least some packages that
require /dev/shm/. I don't remember anything about /sys.

I think it would also be a good idea to have this documented in
policy if it's not already.

> IIRC it is generally established that packages are not allowed to
> rely on an internet connection during build but if one is present
> are they allowed to assume it's non-broken. I recently came accross
> a package ( sslh ) which fails to build in the presense of nxdomain
> hijacking. Is that a bug?

It basicly comes down to if they try to download something as
source to be build. In that case it's clearly a violation of
policy because the source is not in the archive. Some packages
then fall back to the source file that's in the package, but
they should have always used that in the first place.

I know there are also some packages that have a test suite that
connects to something on the internet. This doesn't result in
changes to the binaries, it just checks that it works properly.
You can argue wheter that should be allowed or not, or that the test
server should run on localhost. In any case packages doing that
shouldn't fail to build because of that and should just skip that
test in case the network is down.

> Some time ago I found that a package (I think it was openjdk but I
> don't remember for sure) which relied on uname -r such that linux32
> had to be used to build it in an i386 chroot on an amd64. However
> since then I'm pretty sure i've seen similar cases with other
> packages on other architectures being treated as bugs.

They probably should try to use the output of dpkg-architecture to
select the arch. Then should never check that output of uname -m.

On the buildds we work around this by using linux32 because there
were a lot of packages that were broken, and it was easier to work
around it. Maybe we should stop doing that.


Kurt


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120921222336.GA31270@roeckx.be">http://lists.debian.org/20120921222336.GA31270@roeckx.be

assumptions about the build environment.
 

* peter green <plugwash@p10link.net> [120921 21:26]:
> I just discovered that on my beagleboard XM (under armhf sid) nacl
> (which previously build on a debian experimental armhf buildd but
> not a debian unstable armhf buildd) will build if /sys is mounted
> but will not build if it is not mounted. Can packages assume that
> /sys will be mounted in the build environment or not?

I'm quite suprised to see /sys to be mounted in chroots. Wasn't one
of the reasons to start /sys and not put the info there in /proc to
not have to have it available in chroots? Shouldn't that information
about hardware usually be kept away from chroots?

Bernhard R. Link


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120921232531.GA20907@client.brlink.eu">http://lists.debian.org/20120921232531.GA20907@client.brlink.eu

assumptions about the build environment.
 

On Sat, 2012-09-22 at 01:25 +0200, Bernhard R. Link wrote:
> * peter green <plugwash@p10link.net> [120921 21:26]:
> > I just discovered that on my beagleboard XM (under armhf sid) nacl
> > (which previously build on a debian experimental armhf buildd but
> > not a debian unstable armhf buildd) will build if /sys is mounted
> > but will not build if it is not mounted. Can packages assume that
> > /sys will be mounted in the build environment or not?
>
> I'm quite suprised to see /sys to be mounted in chroots. Wasn't one
> of the reasons to start /sys and not put the info there in /proc to
> not have to have it available in chroots?

I've never heard that claimed.

> Shouldn't that information about hardware usually be kept away from
> chroots?

Chroots aren't containers. A chrooted environment can use all CPUs and
all network devices, and programs may expect to find information about
them under sysfs.

If you're concerned about leaking sensitive information to untrusted
processes then procfs is a far, far bigger problem (somewhat mitigated
by hidepid or pid namespaces).

Ben.

--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

assumptions about the build environment.
 

On Fri, 2012-09-21 at 23:06 +0200, Adam Borowski wrote:
> On Fri, Sep 21, 2012 at 09:07:57PM +0100, Ben Hutchings wrote:
> > On Fri, Sep 21, 2012 at 08:26:24PM +0100, peter green wrote:
> > > Some time ago I found that a package (I think it was openjdk but I
> > > don't remember for sure) which relied on uname -r such that linux32
> > > had to be used to build it in an i386 chroot on an amd64. However
> > > since then I'm pretty sure i've seen similar cases with other
> > > packages on other architectures being treated as bugs.
> >
> > I think confusion between kernel vs userland architecture is so
> > widespread that we should consider this a necessary part of doing a
> > native build.
>
> Please, don't. This breaks cross building. Any use of uname during build
> other than for logging purposes is a bug.

Certainly build systems should support cross-building and should not
rely on uname -m. And cross-building is useful for derivatives.

But since not all build systems do that, and Debian does not attempt to
cross-build its own packages, we should not compromise the reliability
of the native build environments. An i386 chroot on an x86_64 kernel
should look as much like a plain i386 environment as possible.

[...]
> semi-emulated scratchbox style builds can take 44
> seconds compared to 8 hours native[1].
[...]

And scratchbox makes cross-build more reliable by presenting a native-
looking environment... e.g. by making uname report whatever you want:

http://scratchbox.org/documentation/user/scratchbox-1.0/html/installdoc.html#AEN707

Ben.

--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

assumptions about the build environment.
 

On Sat, Sep 22, 2012 at 12:23:36AM +0200, Kurt Roeckx wrote:
> On Fri, Sep 21, 2012 at 08:26:24PM +0100, peter green wrote:
> > Some time ago I found that a package (I think it was openjdk but I
> > don't remember for sure) which relied on uname -r such that linux32
> > had to be used to build it in an i386 chroot on an amd64. However
> > since then I'm pretty sure i've seen similar cases with other
> > packages on other architectures being treated as bugs.
>
> They probably should try to use the output of dpkg-architecture to
> select the arch. Then should never check that output of uname -m.

That's living on the assumption that there's never any upstream build
system which is both complex (thus difficult to patch correctly) and
relying on uname -m in one or more places. I'm not saying we should
necessarily support such systems, but if running inside something akin
to linux32 is not causing too many problems, it would make that easier.
What's the harm?

--
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120922142832.GK29416@grep.be">http://lists.debian.org/20120922142832.GK29416@grep.be

assumptions about the build environment.
 

On Sat, Sep 22, 2012 at 04:28:32PM +0200, Wouter Verhelst wrote:
> On Sat, Sep 22, 2012 at 12:23:36AM +0200, Kurt Roeckx wrote:
> > They probably should try to use the output of dpkg-architecture to
> > select the arch. Then should never check that output of uname -m.
>
> That's living on the assumption that there's never any upstream build
> system which is both complex (thus difficult to patch correctly) and
> relying on uname -m in one or more places. I'm not saying we should
> necessarily support such systems, but if running inside something akin
> to linux32 is not causing too many problems, it would make that easier.
> What's the harm?

That it breaks multiarch builds. You'd need a separate chroot for every
arch you want to be able to compile for.

Just like you can compile Windows crap by CC=i686-w64-mingw32-gcc, you
should be able to CC=i486-linux-gnu-gcc, and, with a bit of effort that is
supposed to be unnecessary in jessie, already are. So you can with
CC=armel-linux-gnueabi-gcc, without any ugly hacks like scratchbox.
xapt allowed this in squeeze already, adding a suffix -armel-cross where we
add :armel in wheezy. It still doesn't work for every package, but for
stuff YOU maintain, YOU can fix it. It's nice for whoever will be hacking
on your code.

And an interesting tidbit:
checking for suffix of executables... .exe
checking whether we are cross-compiling... no

So the concept of "cross-compiling" is pretty fuzzy :)

As for complex "hard to patch" build systems, could you give me an example?
I can't quite think of a case where patching references to uname could be
tricky.

--
Copyright and patents were never about promoting culture and innovations;
from the very start they were legalized bribes to give the king some income
and to let businesses get rid of competition. For some history, please read
https://en.wikipedia.org/wiki/Statute_of_Monopolies_1623

assumptions about the build environment.
 

On Sat, 22 Sep 2012 19:06:57 +0200, Adam Borowski <kilobyte@angband.pl> wrote:
> And an interesting tidbit:
> checking for suffix of executables... .exe
> checking whether we are cross-compiling... no
>
> So the concept of "cross-compiling" is pretty fuzzy :)

That typically happens if you have wine and binfmt-support installed, since
the "cross-compiling" test is "can the system run an executable built using
the chosen compiler"!

Regards,

Stephen

assumptions about the build environment.
 

On Sat, Sep 22, 2012 at 07:06:57PM +0200, Adam Borowski wrote:
> On Sat, Sep 22, 2012 at 04:28:32PM +0200, Wouter Verhelst wrote:
> > On Sat, Sep 22, 2012 at 12:23:36AM +0200, Kurt Roeckx wrote:
> > > They probably should try to use the output of dpkg-architecture to
> > > select the arch. Then should never check that output of uname -m.
> >
> > That's living on the assumption that there's never any upstream build
> > system which is both complex (thus difficult to patch correctly) and
> > relying on uname -m in one or more places. I'm not saying we should
> > necessarily support such systems, but if running inside something akin
> > to linux32 is not causing too many problems, it would make that easier.
> > What's the harm?
>
> That it breaks multiarch builds.

Multiarch builds are not done on our buildd machines.

> You'd need a separate chroot for every arch you want to be able to compile
> for.

Buildd machines typically have only one chroot for each distribution
they build, as they don't build for multiple architectures.

[...]
> As for complex "hard to patch" build systems, could you give me an example?
> I can't quite think of a case where patching references to uname could be
> tricky.

I can't give you an example of *this* particular issue; but given the
amount of complex build systems I've seen, I don't think it's too
unreasonable to assume they exist.

I'm not saying we should change policy to mandate that i386 builds in
i386 chroots should be done with linux32 running or something similar.
But I also do happen to think that tuning buildd machines to do a bit
more than what policy requires them to do is usually a good thing. For I
used to keep debhelper installed in my buildd chroots, even though it's
not part of build-essential; but since almost all packages use it in
some form or another, it was being installed and deinstalled enough that
I could gain some time by having it installed by default. The result
was of course that packages who mistakenly failed to add debhelper to
their build-depends would successfully build on my buildd hosts, but I
didn't think that was much of a problem.

Similarly, if adding linux32 to the environment on amd64 hosts building
inside a chroot means the success rate for packages built will go up, I
don't think we should refrain from doing so -- unless, of course, doing
so would cause more problems than it would solve, which was the point of
my question.

After all, the autobuilder network is not meant to do QA; we have
resources for doing full-archive rebuilds for that purpose. Instead, the
autobuilder network is meant to make sure we build as many packages as
possible. If we can avoid some issues in building packages that isn't
really worth spelling out in policy by just doing some minor
configuration tweak on a buildd host, I think we should do so -- and
this certainly qualifies, IMO.

--
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120924111419.GQ17514@grep.be">http://lists.debian.org/20120924111419.GQ17514@grep.be