Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Debian dpkg (http://www.linux-archive.org/debian-dpkg/)
-   -   Bug#594179: dpkg armhf patch acceptance status? (http://www.linux-archive.org/debian-dpkg/490581-bug-594179-dpkg-armhf-patch-acceptance-status.html)

Loc Minier 02-17-2011 10:27 AM

Bug#594179: dpkg armhf patch acceptance status?
 
Hey

Trying to kick the dust a bit as having the triplet "in the air" is
kind of an unhappy situation for armhf :-)

On Wed, Sep 08, 2010, Guillem Jover wrote:
> We currently need something like this in dpkg-dev because the mappings
> need to be bidirectional, as dpkg-dev needs to be able to convert
> from GNU triplet to Debian architecture and the other way around.
>
> Steve wondered why this is the case, and that's because for
> cross-compiling purposes dpkg-architecture infers the host architecture
> from the CC environment variable through the -dumpmachine option.
> Chaning this is possible but, would break a current way of
> cross-compiling which has been around for a long time.

So this use case is "CC=arm-linux-gnueabi-gcc dpkg-buildpackage" and it
just guesses the -a flag that you should have set?

> The toolchain has the same triplet for binary incompatible producing
> objects, which seems like a bug/misfeature to me, and that's one of
> the assumptions dpkg-dev has relied on, in the same way as multiarch
> when deciding to use the triplet as a unique token for the installation
> directories.

You describe this as a bug/misfeature, that might be true but I don't
think we can challenge this usage of triplets in the upstream
toolchain, and multiarch is moving to having its own tuples instead now
(http://wiki.debian.org/Multiarch/Tuples).

> This also causes issue with not being able to have installed two
> cross-toolchains for say armel and armhf as they share triplet,
> although you can use the armel toolchain with few options to build for
> armhf, but then you'd need to specify those as part of the CC variable.

Even if we could co-install them, I'm not sure how
/some-armel-path/arm-linux-gnueabi-gcc and
/some-armhf-path/arm-linux-gnueabi-gcc could helpfully be installed on
the same system :-/

It's a real limitation of the upstream toolchain.

> Also that also happens with biarch, you can produce i386 binaries from
> an x86_64 toolchain, yet they both have their own triplet.

Yes but x86 goes to the other extreme, with separate triplets for
compatible ABIs ({i386,i486,i586,i686}-linux-gnu); the fact is the
triplet is not a good ABI specifier.

> Anyway, ideally I'd rather see this addressed by giving armhf a real
> triplet, which would also make multiarch life way easier as there'd not
> be any need to define an artificial set of neutral architecture names
> to be able to place objects in the file system. But if this is not going
> to happen, and thus the assumptions dpkg-dev is making have been just
> wrong, then I'd rather drop the bidirectional mappings, and be done
> with it. This will need careful consideration though, as it's breaking
> a current interface, but something that could be done for dpkg 1.16.x.

Having a separate triplet (not using the vendor field) like e.g. lpia
would mean a lot of pain IMO, and we'd have to fix many packages to
cope with it, including reaching out to upstream etc. It was pain for
lpia for sure.

We could have an additional set of preprocessor macros which dpkg
checks for to decide which Debian architecture we're talking about; the
the would only be done if there is more than one architecture using the
same triplet in dpkg tables. This alone is a bit fragile though, as it
means that if a new ABI is introduced to an existing triplet and is not
immediately added as a new dpkg architecture, then people might be
picking the wrong dpkg architecture when using a toolchain defaulting
to thenew ABI.

I wonder whether it would be a good idea to use multiarch tuples
internally; dpkg would still have to map to/from GNU triplets, but it
would force implementors to think about their ABI. I am not sure how
we can ensure that we've mapped to the right tuple though. Neither am
I sure that the multiarch tuples are frozen already, so it might be too
early for that either.

Bye
--
Loc Minier


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110217112730.GA19562@bee.dooz.org">http://lists.debian.org/20110217112730.GA19562@bee.dooz.org

Guillem Jover 02-18-2011 09:13 AM

Bug#594179: dpkg armhf patch acceptance status?
 
[ CCing Matthias, as I'd like your opinion on my proposed solution
involving some Debian gcc changes. ]

Hi!

On Thu, 2011-02-17 at 12:27:30 +0100, Loïc Minier wrote:
> Trying to kick the dust a bit as having the triplet "in the air" is
> kind of an unhappy situation for armhf :-)

I think it's fair to assume several of you have already peeked into the
notes I circulated on IRC the other day, :) I'm reincluding them here
although slightly redacted and extended, for the benefit of the rest
of the people. I'll still reply to specific points you rised below.


GNU triplets
------------

* gcc upstream claims GNU triplets do not fully encode ABI:
- Thus cannot be used properly to map deb arch ←→ GNU triplet.
- Thus cannot be used for multiarch paths.
- Thus cannot be used for default cross-toolchains sharing GNU triplet.
- Thus we'd need yet another ABI encoding and mapping for multiarch
and dpkg arches, instead of being able to just use GNU triplets.
The one being currently proposed can be found at
<http://wiki.debian.org/Multiarch/Tuples>.

* But the GNU triplets have always encoded the ABI, it's even
extremely explicit in some cases, for example with the arm EABI,
executable formats (elf, aout, ecoff, etc) or stuff like
powerpc-*-eabispe* or powerpc-*-eabialtivec* (supported by gcc
upstream).

* And we already use GNU triplets for powerpcspe (powerpc-linux-gnuspe)
and lpia (i386-linux-gnulp) arches, and these do not need gcc upstream
support as are handled transparently by the -gnu* patterns, and any
differing ABI options are selected by the Debian gcc packaging.

* There's other gcc options which can change the ABI of the default
compiler besides the --with-float one, like --with-long-double-128,
etc, and still they are not a problem mainly for two reasons:

1) In case they do not need to coexist, they can be handled like
other transitions, like it was done in Debian, by renaming
shared library packages or with versioned symbols and handling
both ABIs at the same time.

2) No one seems to have needed to create coexisting default toolchains
with diverging ABI options at the same time until now, or they have
done so privately.

* I think it's important to be able to consistently support co-installing
different default cross-compilers for different ABIs which would
otherwise share the same triplet. And while we could avoid the problem
for multiarch libraries co-installations by using the multiarch tuples,
we would not be able to avoid it for the cross-compilers.

* The assumption that each GNU triplet denotes a different ABI is so
entrenched in the GNU build system, that we have things like the
following all over the place to properly support cross-building:

,---
ifeq ($(DEB_BUILD_GNU_TYPE), $(DEB_HOST_GNU_TYPE))
confflags += --build $(DEB_HOST_GNU_TYPE)
else
confflags += --build $(DEB_BUILD_GNU_TYPE) --host $(DEB_HOST_GNU_TYPE)
endif
`---

This would not hold true any longer if we didn't use a unique triplet
per arch, and would thus break in quite annoying ways. For further
details why that's needed check the autotools-dev section
“Calling GNU configure properly” in its README.Debian.

* Ignoring the bi-directional 1:1 mapping would be a PITA, as it implies
not being able to automatically bootstrap dpkg on a new port or
dpkg package builds on foreign distros. The former is annoying but not
done frequently, the latter is going to be a problem for each non-dpkg
based distribution, as they'll have to bootstrap dpkg on each arch
where dpkg is built anew. It ends up being a matter of off-loading the
knowledge of the arch and build system from the dpkg/gcc combo to the
porters/maintainers, which seems rather unappealing.

* Other packaging systems seem to not have made quite the same assumption
about the GNU triplet as dpkg/Debian does. Gentoo's portage at least
relies on the GNU triplet being unique per different arch as itseems to
be using it on the path somehow. rpm and conary for example use uname
which is not enough by itself, and quite unsatisfactory as it's lacking
quite a bit of the ABI information, which probably has not presented as
a problem for them as they do not have the amount of supported arches
as Debian does, neither do they seem to have the same amount of
mixes/convinations we do. So the other approaches seem to be actually
inferior.

* The remaining problem at least for multiarch is the use of more
specialized cpu names for the i386 triplets, i486-linux-gnu on Debian,
which might change depending on the base instruction set to support,
for example i686-linux-gnu on Ubuntu.

Due to #611741, it already crossed my mind (but removed it from the
list of solutions before sending the reply as at the time it seemed
slighly preposterous just to fix the divergence with Ubuntu) that we
should switch back our i386 arch to use i386 as the base cpu name for
the triplet (i386-linux-gnu) in the same way we use it in other
arches, like on arm where we use arm-linux-gnu instead of something
like armv4tel-linux-gnueabi, for example. (I mentioned this already to
Steve and Raphaël on some private conversation.)


Solution(s)
-----------

So while some of the previous points _could_ be considered corner-cases,
there are several of them piling up, which work just fine with our (and
GNU build system) current assumptions, and breaking them seems
gratuituous and way worse than not doing it, as it adds future confusion,
breakage and complexity, instead of the extremely “simple” and more
correct solution (in contrast to the claimed dpkg lameness/limitation,
which I disagree with, and I think I've proven incorrect above) of just
using a new triplet.


- Use additional pre-processor macros for armhf, inccurs an additional
pre-processor call for the arm arches. It implies the Debian arches
cannot be fully retrieved w/o a cross-compiler for each of the arches
lacking information, which means we lose the univeral bi-directional
mapping.

This one is a no go.

- Abuse the vendor field, a hack upstream does not like either, although
strangely enough they seem to prefer it instead of a proper triplet.

Although viable, I don't really see the point, we might as well just
do the right thing and use a proper triplet then.

+ Just use a new triplet arm-*-*-gnueabihf or whatever in a similar vein
as our own -gnulp and -gnuspe.

Would not need to be explicitly supported by gcc upstream to perform
the automatic selection of the ABI based on the triplet for different
variants, which seems to be the biggest reason they have against it.
This should be satisfactory for upstream as the magic will be kept
in the Debian packaging, and they can be kept oblivious of it.

Package build systems might be matching against stuff like
arm*-*-linux-gnueabi, so it might require changes to match on -gnueabi*
instead, but is more immediate, and not like propagating config.guess
all over the place, which we should not need as debian/rules should
be passing --build to configure anyway, and config.sub can already
canonicalize something like arm-linux-gnueabihf by way of the current
patterns.

Switch also i486-linux-gnu (Debian) and i686-linux-gnu (Ubuntu) to
simply i386-linux-gnu (but still preserving their respective cpu
tunnings). This also fixes this inconsistency in the i386 arch.

So Matthias would you be amenable to including something like the
attached patch to Debian gcc (regardless of upstream accepting it,
although the patch is pretty minimal and non-intrusive, which would
seem to have good chances of being accepted there)?
What about switching the triplet for i386, I could prepare patches
for that if you want (I think it's a one-liner though)?

Konstantinos, would you be willing test such patch with an accompanying
triplet in dpkg (patch attached too)? If you don't like
arm-linux-gnueabihf, then something like arm-linux-gnueabivfp or
whatever you want, I don't think I particularly care much, it would
need to be prefixed arm-linux-gnueabi though so that the matches work
transparently.

Beware, both patches completely untested!

> On Wed, Sep 08, 2010, Guillem Jover wrote:
> > We currently need something like this in dpkg-dev because the mappings
> > need to be bidirectional, as dpkg-dev needs to be able to convert
> > from GNU triplet to Debian architecture and the other way around.
> >
> > Steve wondered why this is the case, and that's because for
> > cross-compiling purposes dpkg-architecture infers the host architecture
> > from the CC environment variable through the -dumpmachine option.
> > Chaning this is possible but, would break a current way of
> > cross-compiling which has been around for a long time.
>
> So this use case is "CC=arm-linux-gnueabi-gcc dpkg-buildpackage" and it
> just guesses the -a flag that you should have set?

That and for automatic bootstrapping purposes too.

> > The toolchain has the same triplet for binary incompatible producing
> > objects, which seems like a bug/misfeature to me, and that's one of
> > the assumptions dpkg-dev has relied on, in the same way as multiarch
> > when deciding to use the triplet as a unique token for the installation
> > directories.
>
> You describe this as a bug/misfeature, that might be true but I don't
> think we can challenge this usage of triplets in the upstream
> toolchain, and multiarch is moving to having its own tuples instead now
> (http://wiki.debian.org/Multiarch/Tuples).

Oh, but I think I just did. :) Also given the above I don't think it
makes sense to invent a new set of tuples, the triplets should work
just fine. In addition those tuples end up relying partly on the
definition of the ABI the default gcc has for a given target, which
should not change incompatibly for a given Debian architecture, or we
are screwed anyway. So I don't see that it buys us much.

> > This also causes issue with not being able to have installed two
> > cross-toolchains for say armel and armhf as they share triplet,
> > although you can use the armel toolchain with few options to build for
> > armhf, but then you'd need to specify those as part of the CC variable.
>
> Even if we could co-install them, I'm not sure how
> /some-armel-path/arm-linux-gnueabi-gcc and
> /some-armhf-path/arm-linux-gnueabi-gcc could helpfully be installed on
> the same system :-/

Well for cross-toolchains you have them in the PATH, and they just get
prefixed with the triplet so you'd end up with something like:

/usr/bin/arm-linux-gnueabi-gcc-4.4
/usr/lib/gcc/arm-linux-gnueabi/4.4.5

and

/usr/bin/arm-linux-gnueabihf-gcc-4.4
/usr/lib/gcc/arm-linux-gnueabihf/4.4.5

which work just fine, as they should.

> It's a real limitation of the upstream toolchain.

Not really, it's only a limitation if the same triplet is used for
incompatible ABIs.

> > Also that also happens with biarch, you can produce i386 binaries from
> > an x86_64 toolchain, yet they both have their own triplet.
>
> Yes but x86 goes to the other extreme, with separate triplets for
> compatible ABIs ({i386,i486,i586,i686}-linux-gnu); the fact is the
> triplet is not a good ABI specifier.

Sure, but as mentioned above, I'm now convinced the correct solution is
to switch back to i386-linux-gnu.

> > Anyway, ideally I'd rather see this addressed by giving armhf a real
> > triplet, which would also make multiarch life way easier as there'd not
> > be any need to define an artificial set of neutral architecture names
> > to be able to place objects in the file system. But if this is not going
> > to happen, and thus the assumptions dpkg-dev is making have been just
> > wrong, then I'd rather drop the bidirectional mappings, and be done
> > with it. This will need careful consideration though, as it's breaking
> > a current interface, but something that could be done for dpkg 1.16.x.
>
> Having a separate triplet (not using the vendor field) like e.g. lpia
> would mean a lot of pain IMO, and we'd have to fix many packages to
> cope with it, including reaching out to upstream etc. It was pain for
> lpia for sure.

If we have to fix lots of packages that means they are already buggy,
at least for the packaging bits, the packaging should not generally
be matching against the GNU triplets. For their upstream build systems
that might possibly mean changing from *-gnueabi to *-gnueabi*, but I
doubt there are many upstream build systems matching against -gnueabi
at all? Most should be matching against *-gnu* if at all.

> We could have an additional set of preprocessor macros which dpkg
> checks for to decide which Debian architecture we're talking about; the
> the would only be done if there is more than one architecture using the
> same triplet in dpkg tables. This alone is a bit fragile though, as it
> means that if a new ABI is introduced to an existing triplet and is not
> immediately added as a new dpkg architecture, then people might be
> picking the wrong dpkg architecture when using a toolchain defaulting
> to thenew ABI.

As stated above, it does not solve many of the stated problems, so a no
go, also as a counter example, arm EABI actually uses such a macro to
define its own triplet (see config.guess).

> I wonder whether it would be a good idea to use multiarch tuples
> internally; dpkg would still have to map to/from GNU triplets, but it
> would force implementors to think about their ABI. I am not sure how
> we can ensure that we've mapped to the right tuple though. Neither am
> I sure that the multiarch tuples are frozen already, so it might be too
> early for that either.

Given the above conclusion, that the GNU triplets must be already unique
per arch and cannot arbitrarily change ABI or it'd break stuff, I don't
really see the point in switching the internal representation, because I
don't really see the point in the multiarch tuples, when the GNU triplets
seems to do the job just fine (modulo the armhf and i386 issues, which we
should just fix).

thanks,
guillem

Matthias Klose 02-18-2011 11:30 AM

Bug#594179: dpkg armhf patch acceptance status?
 
On 18.02.2011 11:13, Guillem Jover wrote:

[ CCing Matthias, as I'd like your opinion on my proposed solution
involving some Debian gcc changes. ]


The armhf patch for gcc looks ok, however I would like to see this better
addressed in Linaro and/or upstream.



Yes but x86 goes to the other extreme, with separate triplets for
compatible ABIs ({i386,i486,i586,i686}-linux-gnu); the fact is the
triplet is not a good ABI specifier.


Sure, but as mentioned above, I'm now convinced the correct solution is
to switch back to i386-linux-gnu.


No, this doesn't work. the triplet currently is set to i586-linux-gnu. Switching
back to to i386 would loose the sync primitives and a not working gomp. There
is too much relying on >= i586 in many configure's of other packages. Not only
for the good, as the switch in Ubuntu to i686 did show, because many configure
files assume sse with i686-linux-gnu.


Matthias


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4D5E665B.1060400@debian.org">http://lists.debian.org/4D5E665B.1060400@debian.org

Steve Langasek 02-18-2011 05:09 PM

Bug#594179: dpkg armhf patch acceptance status?
 
On Fri, Feb 18, 2011 at 01:30:19PM +0100, Matthias Klose wrote:
> On 18.02.2011 11:13, Guillem Jover wrote:
> >[ CCing Matthias, as I'd like your opinion on my proposed solution
> > involving some Debian gcc changes. ]

> The armhf patch for gcc looks ok, however I would like to see this
> better addressed in Linaro and/or upstream.

I'm not sure how Linaro could better address this, short of persuading
upstream to allocate a separate triplet for armhf - which has been
explicitly refused on the upstream mailing list. Do you have something else
in mind here?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org

Steve Langasek 02-19-2011 12:36 AM

Bug#594179: dpkg armhf patch acceptance status?
 
Hi Guillem,

Thanks for letting us know your thoughts.

On Fri, Feb 18, 2011 at 11:13:11AM +0100, Guillem Jover wrote:
> * The assumption that each GNU triplet denotes a different ABI is so
> entrenched in the GNU build system, that we have things like the
> following all over the place to properly support cross-building:

> ,---
> ifeq ($(DEB_BUILD_GNU_TYPE), $(DEB_HOST_GNU_TYPE))
> confflags += --build $(DEB_HOST_GNU_TYPE)
> else
> confflags += --build $(DEB_BUILD_GNU_TYPE) --host $(DEB_HOST_GNU_TYPE)
> endif
> `---

But then we also have biarch and triarch toolchains in the archive, where
the host triplet doesn't change at all and instead gcc multilib is used to
designate what ABI you want.

The upstream argument seems to be that armel/armhf isn't "cross-compiling",
it's merely ABI selection within an architecture. This sort of "cross"
compiling doesn't yield binaries that can't be run on the build machine, and
therefore most of the cross-compilation logic in autotools doesn't apply.

(Never mind that upstream does not apply this reasoning consistently; the
fact that there's precedent means it's going to be harder to argue the point
with them.)

> * The remaining problem at least for multiarch is the use of more
> specialized cpu names for the i386 triplets, i486-linux-gnu on Debian,
> which might change depending on the base instruction set to support,
> for example i686-linux-gnu on Ubuntu.

> Due to #611741, it already crossed my mind (but removed it from the
> list of solutions before sending the reply as at the time it seemed
> slighly preposterous just to fix the divergence with Ubuntu) that we
> should switch back our i386 arch to use i386 as the base cpu name for
> the triplet (i386-linux-gnu) in the same way we use it in other
> arches, like on arm where we use arm-linux-gnu instead of something
> like armv4tel-linux-gnueabi, for example. (I mentioned this already to
> Steve and Raphal on some private conversation.)

The challenge, as Matthias points out, is that these triplets are already so
entrenched and there is so much software that handles x86 specially - even
if incorrectly! - that it's prohibitive to switch back to i386-linux-gnu as
our triplet because of all the re-porting work we'll have to do to get
properly functioning packages on x86.

> Package build systems might be matching against stuff like
> arm*-*-linux-gnueabi, so it might require changes to match on -gnueabi*
> instead, but is more immediate, and not like propagating config.guess
> all over the place, which we should not need as debian/rules should
> be passing --build to configure anyway, and config.sub can already
> canonicalize something like arm-linux-gnueabihf by way of the current
> patterns.

Yes, package build systems do match on stuff like arm*-*-linux-gnueabi; my
understanding is that this was the pattern match that was propagated when
EABI was introduced on ARM, and as a result there's a fair amount of
software that will misbuild for eabi+hf if we go with -gnueabihf. So
basically, it's the same problem we have as for switching to i386-linux-gnu
on i386, just on a different set of software in the archive.

How do we locate the software that will be affected by such a change? I
think this has proven non-trivial in the past.

But ultimately that's for the porters and the GCC maintainers to decide
whether they're happy with that answer; it's an aside to the main point of
my mail, coming up below. :)

> > > The toolchain has the same triplet for binary incompatible producing
> > > objects, which seems like a bug/misfeature to me, and that's one of
> > > the assumptions dpkg-dev has relied on, in the same way as multiarch
> > > when deciding to use the triplet as a unique token for the installation
> > > directories.

> > You describe this as a bug/misfeature, that might be true but I don't
> > think we can challenge this usage of triplets in the upstream
> > toolchain, and multiarch is moving to having its own tuples instead now
> > (http://wiki.debian.org/Multiarch/Tuples).

> Oh, but I think I just did. :) Also given the above I don't think it
> makes sense to invent a new set of tuples, the triplets should work
> just fine. In addition those tuples end up relying partly on the
> definition of the ABI the default gcc has for a given target, which
> should not change incompatibly for a given Debian architecture, or we
> are screwed anyway. So I don't see that it buys us much.

The underlying intent of the tuples proposal is that we stop pretending that
GNU triplets provide a valid global one-to-one mapping to ABIs, because we
know from several concrete, high profile examples that this is not true.

You have proposed a patch that Matthias has said "looks ok", but if he adds
it to the Debian/Ubuntu gcc, that only solves the problem for Debian and
Ubuntu. Multiarch is intended as a cross-vendor standard, fixing the FHS's
inadequate lib<qual> directories and providing binary interoperability for
anyone choosing to implement it. How can we achieve that if we're using
strings in our paths that are dependent on distro-specific patches? Do we
expect to tell other implementors they have to use our GCC?

The multiarch tuple proposal would externalize this problem by establishing
a central, standard, one-to-one mapping between ABIs and strings *for this
precise purpose*. I'm perfectly happy for Matthias to accept your gcc patch
in parallel so we establish a distinct GNU-ish triplet for armhf, but unless
this can get upstreamed, I don't think it solves the problem we have with
using GNU triplets for multiarch paths.

Do you agree, or do you still think multiarch should continue to be
dependent on GNU triplets, which - as we've seen - are not managed in a
consistent manner?

> Given the above conclusion, that the GNU triplets must be already unique
> per arch and cannot arbitrarily change ABI or it'd break stuff, I don't
> really see the point in switching the internal representation, because I
> don't really see the point in the multiarch tuples, when the GNU triplets
> seems to do the job just fine (modulo the armhf and i386 issues, which we
> should just fix).

Given that there is no consensus that this is even a bug to be fixed, I am
not content to have multiarch blocked indefinitely on getting triplet fixes
propagated upstream; nor am I happy to have multiarch paths tied to
distro-specific implementation details. Are you going to do the heavy
lifting to get these changes accepted by GCC upstream? How long do you
think that will take, and at what point would you decide that you're not
getting anywhere and fall back to diverging from upstream for multiarch?

I've rather firmly committed to getting initial multiarch support landed in
the 11.04 Ubuntu release. If we can (collectively) get our decision made on
the path selection *now*, that's achievable - and we can be rid of ia32-libs
from Debian (unstable) and Ubuntu within a year, and we can bring armhf up
as the first multiarch-from-the-start port in Debian. If we instead let
ourselves get drawn into open-ended discussions trying to persuade GCC
upstream that we're right about triplets and they're wrong, it may be years
longer before we see any multiarch implementation at all. Remember that we
were discussing multiarch before sarge was released - I don't want to still
be discussing it after wheezy is out. :(

Thanks,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org

Wookey 02-21-2011 12:52 AM

Bug#594179: dpkg armhf patch acceptance status?
 
+++ Guillem Jover [2011-02-18 11:13 +0100]:

Guillem makes some good points about how GNU triplets should (and once
did) represent ABIs, and that if they still did, dpkg (and everything
else) could use them as the definitive ABI-indicator. He's quite
right.

_Something_ has to stand as nomenclature for the ABI. It could equally
well be GNU triplets or Multiarch Tuples. I'm quite happy for either
to be used. Multiarch directories could use triplets under those
circumstances and there would be no need for the new tuples.

But it is also the case that a given toolchain can produce binaries of
different ABIs, so there is logic in the GCC argument that the triplet
denotes a toolchain, not (necessarily) an ABI. (e.g arm-linux-gnueabi
toolchain can produce armel and armhf binaries (and in fact there are
more incompatible ABIs it could spit out, but we're not trying to make
distros out of those).

It makes sense to me that, these days, a triplet represents a
toolchain, and an MAtuple an ABI. If there was a 1-1 mapping between
these things then triplets would suffice, but there isn't.

So it seems to me, that we either have to persuade GCC to go back to
one triplet per ABI or we need to adopt the proposed Tuples.

It is logically neater in many ways to go back to one-triplet-per-ABI,
but like Steve, I'm not optimistic about this being ultimately acceptable
to upstream or that it could be done in a sensible period of time.
Starting again with a properly defined list and some rules about new
identifiers lets us get it right this time, and arrange for it to stay
right in the future.

So what happens if we do this? Let's look at some examples mentioned
so far:

> * I think it's important to be able to consistently support co-installing
> different default cross-compilers for different ABIs which would
> otherwise share the same triplet. And while we could avoid the problem
> for multiarch libraries co-installations by using the multiarch tuples,
> we would not be able to avoid it for the cross-compilers.

This is true, but in fact the arm-linux-gnueabi and arm-linux-gnuhf
toolchains would be the same apart from their libgcc and default options
(spec file). A multilib build of this toolchain, supporting both
options, would work for targetting both ports, but we do still lack a
good way of switching spec files to change the default target (which
would be useful for all sorts of reasons).

> * The assumption that each GNU triplet denotes a different ABI is so
> entrenched in the GNU build system, that we have things like the
> following all over the place to properly support cross-building:
>
> ,---
> ifeq ($(DEB_BUILD_GNU_TYPE), $(DEB_HOST_GNU_TYPE))
> confflags += --build $(DEB_HOST_GNU_TYPE)
> else
> confflags += --build $(DEB_BUILD_GNU_TYPE) --host $(DEB_HOST_GNU_TYPE)
> endif
> `---
>
> This would not hold true any longer if we didn't use a unique triplet
> per arch, and would thus break in quite annoying ways.

Yes, this is potentially a pain in the bum. We need to select a
toolchain (which could be 'arm-linux-gnueabi' for both arm port
targets, but with different default options and a different default
linker and include paths). I haven't yet worked out how those will be
selected, as currently they are built-in to the cross-compiler and
referring to it by name sets them implicictly.

debian/rules has access to the -a<DEB_HOST_ARCH> which allows the
necessary discrimiation, and automatically supplies the
/etc/dpkg-cross/cross-config.<DEB_HOST_ARCH> autoconf values, which
may be sufficient, along with config.guess.

There are a lot of instances of -L /usr/<DEB_HOST_GNU_TYPE>/lib which
need fixing for multiarch anyway, so even if we have unique GNU
triplets, work will be needed in lots of packages to make
cross-building work again. Anything which already relies on default
compiler paths should work fine.

If using non-unique toolchain triplets, we need a robust mechanism for
host ABI selection within the multilibs available.


> * Ignoring the bi-directional 1:1 mapping would be a PITA, as it implies
> not being able to automatically bootstrap dpkg on a new port or
> dpkg package builds on foreign distros. The former is annoying but not
> done frequently, the latter is going to be a problem for each non-dpkg
> based distribution, as they'll have to bootstrap dpkg on each arch
> where dpkg is built anew. It ends up being a matter of off-loading the
> knowledge of the arch and build system from the dpkg/gcc combo to the
> porters/maintainers, which seems rather unappealing.

I don't think I understand this point. Who are these people that
build dpkg on non-dpkg distro, and why will they have harder
bootstraping?

> > > The toolchain has the same triplet for binary incompatible producing
> > > objects, which seems like a bug/misfeature to me, and that's one of
> > > the assumptions dpkg-dev has relied on, in the same way as multiarch
> > > when deciding to use the triplet as a unique token for the installation
> > > directories.
> >
> > You describe this as a bug/misfeature, that might be true but I don't
> > think we can challenge this usage of triplets in the upstream
> > toolchain, and multiarch is moving to having its own tuples instead now
> > (http://wiki.debian.org/Multiarch/Tuples).
>
> Oh, but I think I just did. :) Also given the above I don't think it
> makes sense to invent a new set of tuples, the triplets should work
> just fine. In addition those tuples end up relying partly on the
> definition of the ABI the default gcc has for a given target,

Well, we need to be able to specify the ABI (and thus paths/gcc options) to
use. That has traditionally been done with the triplet causing gcc
defaults to be used, but it can just as well be done in an
upstreamable and cross-distro way using LSB-certified MAtuples. Either
could work - which is easier, or otherwise provides overall benefit?

> > > This also causes issue with not being able to have installed two
> > > cross-toolchains for say armel and armhf as they share triplet,
> > > although you can use the armel toolchain with few options to build for
> > > armhf, but then you'd need to specify those as part of the CC variable.

On in the spec file.

> > I wonder whether it would be a good idea to use multiarch tuples
> > internally; dpkg would still have to map to/from GNU triplets, but it
> > would force implementors to think about their ABI. I am not sure how
> > we can ensure that we've mapped to the right tuple though. Neither am
> > I sure that the multiarch tuples are frozen already, so it might be too
> > early for that either.
>
> Given the above conclusion, that the GNU triplets must be already unique
> per arch and cannot arbitrarily change ABI or it'd break stuff, I don't
> really see the point in switching the internal representation, because I
> don't really see the point in the multiarch tuples, when the GNU triplets
> seems to do the job just fine (modulo the armhf and i386 issues, which we
> should just fix).

This pretty-much boils down the issue. dpkg needs to use a unique and
reliable internal representation. Can that carry on being/go back to
being triplets, or do we have to use the MAtuples?

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110221015244.GM22368@dream.aleph1.co.uk">http://lists.debian.org/20110221015244.GM22368@dream.aleph1.co.uk

Wookey 02-21-2011 01:22 AM

Bug#594179: dpkg armhf patch acceptance status?
 
+++ Steve Langasek [2011-02-18 17:36 -0800]:
> > * The remaining problem at least for multiarch is the use of more
> > specialized cpu names for the i386 triplets, i486-linux-gnu on Debian,
> > which might change depending on the base instruction set to support,
> > for example i686-linux-gnu on Ubuntu.
>
> > Due to #611741, it already crossed my mind (but removed it from the
> > list of solutions before sending the reply as at the time it seemed
> > slighly preposterous just to fix the divergence with Ubuntu) that we
> > should switch back our i386 arch to use i386 as the base cpu name for
> > the triplet (i386-linux-gnu) in the same way we use it in other
> > arches, like on arm where we use arm-linux-gnu instead of something
> > like armv4tel-linux-gnueabi, for example. (I mentioned this already to
> > Steve and Raphal on some private conversation.)
>
> The challenge, as Matthias points out, is that these triplets are already so
> entrenched and there is so much software that handles x86 specially - even
> if incorrectly! - that it's prohibitive to switch back to i386-linux-gnu as
> our triplet because of all the re-porting work we'll have to do to get
> properly functioning packages on x86.

I know little of x86 world, but are we sure this is more work than the
fixing-up we'll need to do in many packages to properly deal with
multiple available ABIs within a toolchain, and making the new
MAtuples be used correctly everywhere? (especially when
cross-building, for both matters).

> The underlying intent of the tuples proposal is that we stop pretending that
> GNU triplets provide a valid global one-to-one mapping to ABIs, because we
> know from several concrete, high profile examples that this is not true.
>
> You have proposed a patch that Matthias has said "looks ok", but if he adds
> it to the Debian/Ubuntu gcc, that only solves the problem for Debian and
> Ubuntu. Multiarch is intended as a cross-vendor standard, fixing the FHS's
> inadequate lib<qual> directories and providing binary interoperability for
> anyone choosing to implement it. How can we achieve that if we're using
> strings in our paths that are dependent on distro-specific patches? Do we
> expect to tell other implementors they have to use our GCC?
>
> The multiarch tuple proposal would externalize this problem by establishing
> a central, standard, one-to-one mapping between ABIs and strings *for this
> precise purpose*.

I agree that this establishment of a generic FHS multiarch library
path standard is the right thing (TM), and that it's no good if it
can't be done upstream . But equally Guillem is right that
ABI-sepecific triplets could do that job too (and with less change to
existing practice overall).

> Given that there is no consensus that this is even a bug to be fixed, I am
> not content to have multiarch blocked indefinitely on getting triplet fixes
> propagated upstream; nor am I happy to have multiarch paths tied to
> distro-specific implementation details. Are you going to do the heavy
> lifting to get these changes accepted by GCC upstream? How long do you
> think that will take, and at what point would you decide that you're not
> getting anywhere and fall back to diverging from upstream for multiarch?

This is the nub of the practical problem.

> I've rather firmly committed to getting initial multiarch support landed in
> the 11.04 Ubuntu release. If we can (collectively) get our decision made on
> the path selection *now*, that's achievable - and we can be rid of ia32-libs
> from Debian (unstable) and Ubuntu within a year, and we can bring armhf up
> as the first multiarch-from-the-start port in Debian. If we instead let
> ourselves get drawn into open-ended discussions trying to persuade GCC
> upstream that we're right about triplets and they're wrong, it may be years
> longer before we see any multiarch implementation at all. Remember that we
> were discussing multiarch before sarge was released - I don't want to still
> be discussing it after wheezy is out. :(

I guess I missed the first 3-odd years of this debate. I'm assuming we've
already tried reasonably hard persuading the GCC people on this point?

Unless the feature of being able to separately specify toolchains
(triplets) and ABIs (tuples) is important (and it may be for
biarch/triarch arches, but I'm not yet convinced), then given the
extreme similarity of tuples and triplets, it does seem like
ABI-specific triplets would be a 'better' solution to this problem
than inventing a new set; and in general Debian likes to go for the
technically-best solutions even if it takes longer. (Multiarch in
general is an example of this).

However, equally, it's taken a really long time to get this far, and
if we made a reasonable effort to get GCC people to agree with
guilem's view of the world, but failed, then I'm happy that the
expediency of the new tuples is justified.

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110221022243.GN22368@dream.aleph1.co.uk">http://lists.debian.org/20110221022243.GN22368@dream.aleph1.co.uk

Guillem Jover 02-21-2011 05:32 AM

Bug#594179: dpkg armhf patch acceptance status?
 
[ Sorry for entangling the armhf bug with the i386 stuff, subsequent
replies should probably remove debian-arm and the bug report from
them. ]

On Fri, 2011-02-18 at 13:30:19 +0100, Matthias Klose wrote:
> On 18.02.2011 11:13, Guillem Jover wrote:
> >[ CCing Matthias, as I'd like your opinion on my proposed solution
> > involving some Debian gcc changes. ]
>
> The armhf patch for gcc looks ok, however I would like to see this
> better addressed in Linaro and/or upstream.

Ok, I'll run it through upstream to see what they say. I just wanted
to check if you'd be fine including it regardless of upstream's stance
on it, to at least be able to decide on the armhf triplet issue, the
multiarch paths decision would be unrelated to this then.

> >> Yes but x86 goes to the other extreme, with separate triplets for
> >> compatible ABIs ({i386,i486,i586,i686}-linux-gnu); the fact is the
> >> triplet is not a good ABI specifier.
> >
> >Sure, but as mentioned above, I'm now convinced the correct solution is
> >to switch back to i386-linux-gnu.
>
> No, this doesn't work. the triplet currently is set to
> i586-linux-gnu. Switching back to to i386 would loose the sync
> primitives and a not working gomp. There is too much relying on >=
> i586 in many configure's of other packages.

Oh, I think this actually strengthens my point!

The gcc-4.4:i386 package seems to be compiled on Debian to target i586
as the base instruction set, as can be seen in its debian/rules2:388,
which implies changing the triplet would not affect this (barring the
small change I'm attaching a patch for, untested), and thus the sync
primitive and gomp would be fine (at least from my understanding of
the gcc build system). And just to make this perfectly clear, I'm
not proposing to downgrade the actual instruction set base line.

So while the base instruction set was changed to i586 in gcc 4.4.0-1~exp1
it did not switch the GNU triplet to i586-linux-gnu to match:

$ dpkg --print-architecture
i386
$ gcc-4.4 -dumpmachine
i486-linux-gnu
$ gcc-4.5 -dumpmachine
i486-linux-gnu
$ gcc-4.6 -dumpmachine
i486-linux-gnu
$ grep ^i386 /usr/share/dpkg/cputable|tr -s ' '|cut -f2
i486

So I don't see how any debian/rules could be currently relying on
the i586-linux-gnu triplet (as long as they set them correctly through
the dpkg-architecture variables, coming from gcc, and not from
config.guess).

Also having to transition all our packaging to the new triplet every
time i386 changes the base line does not seem right, and it's
something we don't do on any of the other architectures. I'd even say
it's just wrong, packages should either use the default compiler base
instruction set, set their required one independenly of it, do
multiple builds and use hwcaps for libraries or do run time detection.

Given the above we'd need to either switch to i586-linux-gnu or
i386-linux-gnu, it seems to me both will imply the same amount of
changes? And thus going for the latter seems the correct solution,
it matches with the other architectures, can be used as the multiarch
paths and can reduce some divergence with Ubuntu, all of them a clear
win! :)

> Not only for the good, as the switch in Ubuntu to i686 did show,
> because many configure files assume sse with i686-linux-gnu.

I'd say any such assumption in those packages is buggy, per above.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110221063219.GA22285@gaara.hadrons.org">http://lists.debian.org/20110221063219.GA22285@gaara.hadrons.org

Steve Langasek 02-21-2011 06:31 AM

Bug#594179: dpkg armhf patch acceptance status?
 
On Mon, Feb 21, 2011 at 02:22:44AM +0000, Wookey wrote:
> > The challenge, as Matthias points out, is that these triplets are already so
> > entrenched and there is so much software that handles x86 specially - even
> > if incorrectly! - that it's prohibitive to switch back to i386-linux-gnu as
> > our triplet because of all the re-porting work we'll have to do to get
> > properly functioning packages on x86.

> I know little of x86 world, but are we sure this is more work than the
> fixing-up we'll need to do in many packages to properly deal with
> multiple available ABIs within a toolchain, and making the new
> MAtuples be used correctly everywhere? (especially when
> cross-building, for both matters).

"multiple available ABIs within a toolchain" is not a problem that has to be
solved on x86 today, so yes, it is more work. On x86 we do have at least
one tuple identifier for each ABI - we just have the problem that we have
/too many/ of them available on x86. But while multilib toolchains are
definitely of interest on x86 (-m32/-m64), each of these ABIs also have a
distinct tuple associated with them (x86_64-linux-gnu, i<n>86-linux-gnu).

> > You have proposed a patch that Matthias has said "looks ok", but if he adds
> > it to the Debian/Ubuntu gcc, that only solves the problem for Debian and
> > Ubuntu. Multiarch is intended as a cross-vendor standard, fixing the FHS's
> > inadequate lib<qual> directories and providing binary interoperability for
> > anyone choosing to implement it. How can we achieve that if we're using
> > strings in our paths that are dependent on distro-specific patches? Do we
> > expect to tell other implementors they have to use our GCC?

> > The multiarch tuple proposal would externalize this problem by establishing
> > a central, standard, one-to-one mapping between ABIs and strings *for this
> > precise purpose*.

> I agree that this establishment of a generic FHS multiarch library
> path standard is the right thing (TM), and that it's no good if it
> can't be done upstream . But equally Guillem is right that
> ABI-sepecific triplets could do that job too (and with less change to
> existing practice overall).

Multiarch is a significant departure from existing practice one way or the
other. I think the question of whether we maintain our ABI string list as a
delta against GNU triplets or as an entirely separate standard is really
quite a minor detail in comparison.

> I guess I missed the first 3-odd years of this debate. I'm assuming we've
> already tried reasonably hard persuading the GCC people on this point?

In the x86 case, it's not something that GCC people can fix. The
overloading of GNU triplets on x86_32 is now a well-entrenched practice
throughout the community, and neither they nor we can fix by fiat the
consequences of changing our triplet - only by a lot of hard work.

In the armhf case there was an upstream mailing list discussion, from which
the only emerging consensus (to the extent we can say there was one) was
that the official GNU triplet should stay the same. This mailing list
discussion informed the BoFs we had at DebConf about multiarch tuples, and I
think it qualifies as "tried reasonably hard". It's hard enough for my
purposes, because I'm not out to try to dictate policy to the GCC
maintainers, so when they say that's not what triplets mean, I greatly
prefer finding a path of lesser resistance.

I thought we had that with the multiarch tuples proposal, but maybe not. :/

> Unless the feature of being able to separately specify toolchains
> (triplets) and ABIs (tuples) is important (and it may be for
> biarch/triarch arches, but I'm not yet convinced), then given the
> extreme similarity of tuples and triplets, it does seem like
> ABI-specific triplets would be a 'better' solution to this problem
> than inventing a new set; and in general Debian likes to go for the
> technically-best solutions even if it takes longer. (Multiarch in
> general is an example of this).

I don't see either of these to be technically better or worse. The fact is
that we are going to *have* to document multiple points where our directory
strings do not follow naturally from the existing array of GNU triplets; and
that documentation needs to be maintained in a readily consumable fashion.
Given those constraints, I don't think that using a modified set of GNU
triplets is any better than creating new strings from scratch.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org

Guillem Jover 02-21-2011 09:15 AM

Bug#594179: dpkg armhf patch acceptance status?
 
On Mon, 2011-02-21 at 07:32:19 +0100, Guillem Jover wrote:
> The gcc-4.4:i386 package seems to be compiled on Debian to target i586
> as the base instruction set, as can be seen in its debian/rules2:388,
> which implies changing the triplet would not affect this (barring the
> small change I'm attaching a patch for, untested), and thus the sync
> primitive and gomp would be fine (at least from my understanding of
> the gcc build system). And just to make this perfectly clear, I'm
> not proposing to downgrade the actual instruction set base line.

And then I forgot to attach the patch. Here it is.

thanks,
guillem


All times are GMT. The time now is 11:32 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.