FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Pacman Development

 
 
LinkBack Thread Tools
 
Old 01-21-2011, 03:04 PM
Allan McRae
 
Default makepkg: add soprovides support

On 22/01/11 01:02, Florian Pritz wrote:

On 21.01.2011 15:49, Allan McRae wrote:

On 22/01/11 00:25, Florian Pritz wrote:

On 19.01.2011 19:54, Dan McGee wrote:

<snip>



* introducing dependencies on readelf and objdump. At least elsewhere
in the strip code, we use `file` to locate shared libraries. And you
can get the SONAME bit out of objdump -p, which would at least limit
this to one tool.


readelf and objdump are both in binutils, but I've noticed that objdump
doesn't support the file format generated by tcc although readelf does
and it's a valid ELF file afaict.

eu-objdump (elfutils) can't output that much, but it understands tcc's
format and outputs pretty nice architecture information. Sadly I have no
idea if that's portable.

Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's
fine since both are in core and you only need them for makepkg anyway.


We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin...
Whatever is used here needs to be portable to all those. I do not
believe elfutils is particularly portable...


I'd use objdump (binutils), but I don't know if it's compatible with
binaries not generated by gcc (at least tcc doesn't work) and I can't
really assume all users use gcc either.

There is a "portability" patch [1] for elfutils. Not sure if that is
what we need though.


That portability patch seems to be mainly to add compatibility to older
Linux systems, not for non-Linux systems.


Allan
 
Old 01-21-2011, 03:05 PM
Yaro Kasear
 
Default makepkg: add soprovides support

On Friday, January 21, 2011 08:49:48 am Allan McRae wrote:
> On 22/01/11 00:25, Florian Pritz wrote:
> > On 19.01.2011 19:54, Dan McGee wrote:
> <snip>
>
> >> * introducing dependencies on readelf and objdump. At least elsewhere
> >> in the strip code, we use `file` to locate shared libraries. And you
> >> can get the SONAME bit out of objdump -p, which would at least limit
> >> this to one tool.
> >
> > readelf and objdump are both in binutils, but I've noticed that objdump
> > doesn't support the file format generated by tcc although readelf does
> > and it's a valid ELF file afaict.
> >
> > eu-objdump (elfutils) can't output that much, but it understands tcc's
> > format and outputs pretty nice architecture information. Sadly I have no
> > idea if that's portable.
> >
> > Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's
> > fine since both are in core and you only need them for makepkg anyway.
>
> We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin...
> Whatever is used here needs to be portable to all those. I do not
> believe elfutils is particularly portable...
>
> Allan

Stop me if I am wrong, but don't all those platforms use ELF by default?
 
Old 01-21-2011, 03:17 PM
Allan McRae
 
Default makepkg: add soprovides support

On 22/01/11 02:05, Yaro Kasear wrote:

On Friday, January 21, 2011 08:49:48 am Allan McRae wrote:

On 22/01/11 00:25, Florian Pritz wrote:

On 19.01.2011 19:54, Dan McGee wrote:

<snip>


* introducing dependencies on readelf and objdump. At least elsewhere
in the strip code, we use `file` to locate shared libraries. And you
can get the SONAME bit out of objdump -p, which would at least limit
this to one tool.


readelf and objdump are both in binutils, but I've noticed that objdump
doesn't support the file format generated by tcc although readelf does
and it's a valid ELF file afaict.

eu-objdump (elfutils) can't output that much, but it understands tcc's
format and outputs pretty nice architecture information. Sadly I have no
idea if that's portable.

Now I'm using readelf (binutils) and eu-objdump (elfutils). I think it's
fine since both are in core and you only need them for makepkg anyway.


We have pacman/makepkg users on Linux, OSX, BSD, Hurd, Cygwin...
Whatever is used here needs to be portable to all those. I do not
believe elfutils is particularly portable...

Allan


Stop me if I am wrong, but don't all those platforms use ELF by default?



Yes, and... That does not make elfutils portable or fix the issue with
objdump and tcc generated files.


Allan
 
Old 01-21-2011, 03:54 PM
Thomas Bächler
 
Default makepkg: add soprovides support

Am 21.01.2011 17:01, schrieb Allan McRae:
> That reply is just wrong... i686 is not a restricted flavour of i386.
> It is the other way around. I can not run i686 optimised software on an
> i386 system. Just ask all the Via C3 owners who do not have that "nopl"
> instruction and the joys they had with glibc-2.12.

That is not the point. We should reflect what the dynamic linker tries
to do whenever we want to load a binary:

1) If we install a i686 version of libfoo.so.2 onto a i386 system, the
linker will still try to load that version (and fail with SIGILL).
2) If we install a x86_64 version and a i686 version of libfoo.so.2 into
the system, and launch a 32 bit binary, the linker will determine that
it cannot load the the x86_64 version, but the i686 version.

So, in order to "do it right", we only need to reflect the distinction
the linker does in 2). Not even the linker makes the distinction in 1),
so why should we?

And now think about use cases: The package architecture is already in
PKGINFO. So the only distinction we need to make is between 32 bit and
64 bit binaries - the only case where you are able to run more than one
architecture in the same system is when executing 32 bit binaries on a
64 bit platform with a backward compatibility mode for 32 bit. (AFAIK,
this is not only the case for x86, but for some other platforms, too).

The only info we should include into the soprovides/depends is whether
we have a 32 bit or a 64 bit binary. There shouldn't be an explicit
statement about architecture.

> So there is a real problem that you can not get the correct value out of
> the library on i686 systems. We could use CARCH, but that does not work
> for multilib stuff, which was the entire point of including it in the
> first place...

I think my previous comments should make this irrelevant, unless I am
completely wrong.

>>> and if we do keep
>>> it, it has *nothing* to do with a version in the normal ordering
>>> sense- it would belong as part of the provision name.
>>
>> I had that before and Allan didn't like it.
>>
>> http://mailman.archlinux.org/pipermail/pacman-dev/2010-February/010420.html
>>
>>
>
> I did not like provides=(libfoo.so) magically turning into
> "libfoo.so-i686" in the .PKGINFO file. The details of the package
> should reflect what is in the PKGBUILD.

So, to summarize:

1) Dan does not want a new PKGINFO field to be introduced for this.
2) Allan does not want any auto-generated meta-info in the provides and
depends in PKGINFO.
3) Dan does not want non-versioning information in the version field in
provides/depends in PKGINFO.

Where does that leave us? Essentially, we cannot get this feature
implemented, as all those restrictions combined leave us with no
options. Clearly, you don't see any options either, as you only provide
destructive criticism, no suggestions on how you think it should be done.
 
Old 01-21-2011, 04:35 PM
Dan McGee
 
Default makepkg: add soprovides support

Wow, this degenerated into a flamewar quickly. Can we all calm down
just a tad? If we were truly at an impasse, Allan and I wouldn't even
be looking at these patches. The biggest reason I'm putting up a stop
sign and red flags is the lack of explanation at the questions I've
posed, and stating I've only offered destructive criticism is not the
truth- I just don't understand this whole thing.

On Fri, Jan 21, 2011 at 10:54 AM, Thomas Bächler <thomas@archlinux.org> wrote:
> Am 21.01.2011 17:01, schrieb Allan McRae:
>> That reply is just wrong... *i686 is not a restricted flavour of i386.
>> It is the other way around. *I can not run i686 optimised software on an
>> i386 system. *Just ask all the Via C3 owners who do not have that "nopl"
>> instruction and the joys they had with glibc-2.12.
>
> That is not the point. We should reflect what the dynamic linker tries
> to do whenever we want to load a binary:
>
> 1) If we install a i686 version of libfoo.so.2 onto a i386 system, the
> linker will still try to load that version (and fail with SIGILL).
> 2) If we install a x86_64 version and a i686 version of libfoo.so.2 into
> the system, and launch a 32 bit binary, the linker will determine that
> it cannot load the the x86_64 version, but the i686 version.
>
> So, in order to "do it right", we only need to reflect the distinction
> the linker does in 2). Not even the linker makes the distinction in 1),
> so why should we?
>
> And now think about use cases: The package architecture is already in
> PKGINFO. So the only distinction we need to make is between 32 bit and
> 64 bit binaries - the only case where you are able to run more than one
> architecture in the same system is when executing 32 bit binaries on a
> 64 bit platform with a backward compatibility mode for 32 bit. (AFAIK,
> this is not only the case for x86, but for some other platforms, too).
>
> The only info we should include into the soprovides/depends is whether
> we have a 32 bit or a 64 bit binary. There shouldn't be an explicit
> statement about architecture.

Taking a step back here, if we want to not lock ourself into ELF-only
(OSX does not use it), it sounds like using objdump due to it's use of
the BFD backend which understands multiple file formats. From objdump
-a, although the output is rather arcane, we should always be able to
get the file format of the library, which I believe is the most
important to matching what the linker does.

Sidenote: I saw this when running objdump --info on my x86_64 box:
plugin
(header little endian, data little endian)
BFD: BFD (GNU Binutils) 2.21.0.20101217 assertion fail
/build/src/binutils/bfd/plugin.c:453
objdump: plugin: Bad value


>> So there is a real problem that you can not get the correct value out of
>> the library on i686 systems. *We could use CARCH, but that does not work
>> for multilib stuff, which was the entire point of including it in the
>> first place...
>
> I think my previous comments should make this irrelevant, unless I am
> completely wrong.
>
>>>> and if we do keep
>>>> it, it has *nothing* to do with a version in the normal ordering
>>>> sense- it would belong as part of the provision name.
>>>
>>> I had that before and Allan didn't like it.
>>>
>>> http://mailman.archlinux.org/pipermail/pacman-dev/2010-February/010420.html
>>>
>>>
>>
>> I did not like provides=(libfoo.so) magically turning into
>> "libfoo.so-i686" in the .PKGINFO file. *The details of the package
>> should reflect what is in the PKGBUILD.

See, this is what I still don't understand. Is this libprovides, or is
this "well we'll hijack your provides array sometimes depending on
what you put there".

It *still* hasn't been explained how this thing works in either a
commit message or docs. Do I include soprovides=() in my PKGBUILD? Do
I need to insert something in provides=() to get a library to show up
in the final package? Is it all completely automatic- all libraries in
the package get an automatic provide entry generated? This is why I am
so damn frustrated with these patches.

>
> So, to summarize:
>
> 1) Dan does not want a new PKGINFO field to be introduced for this.
Where did I say that- if I did, I'm not remembering.

> 2) Allan does not want any auto-generated meta-info in the provides and
> depends in PKGINFO.
No, he stated he doesn't want *magic changes* to data the user
specified in the PKGBUILD. I think he and I would be more comfortable
if this was less magic.

> 3) Dan does not want non-versioning information in the version field in
> provides/depends in PKGINFO.
Sure.

> Where does that leave us? Essentially, we cannot get this feature
> implemented, as all those restrictions combined leave us with no
> options. Clearly, you don't see any options either, as you only provide
> destructive criticism, no suggestions on how you think it should be done.
 
Old 01-21-2011, 04:45 PM
Florian Pritz
 
Default makepkg: add soprovides support

On 21.01.2011 18:35, Dan McGee wrote:
> It *still* hasn't been explained how this thing works in either a
> commit message or docs. Do I include soprovides=() in my PKGBUILD? Do
> I need to insert something in provides=() to get a library to show up
> in the final package? Is it all completely automatic- all libraries in
> the package get an automatic provide entry generated? This is why I am
> so damn frustrated with these patches.

We had all 3 versions and the current patch works as follows:
You add something like "provides=(foo bar libfoo.so)" to the PKGBUILD.
The function searches all provides in the whole pkgdir and filters that
list using your provides array. It then removes your entries
(versionless) and adds those it found (with version).

--
Florian Pritz -- {flo,bluewind}@server-speed.net
 
Old 01-21-2011, 05:09 PM
Thomas Bächler
 
Default makepkg: add soprovides support

Am 21.01.2011 18:35, schrieb Dan McGee:
> Wow, this degenerated into a flamewar quickly.

I deleted all flames before submitting this post. That said, I was a bit
pissed about that last part of Allan's post.

> and stating I've only offered destructive criticism is not the
> truth- I just don't understand this whole thing.

That statement was directed only at the part of Allan's reply that I
quoted right above the statement. It was not in any way meant for you,
or to any other part of Allan's post.

> On Fri, Jan 21, 2011 at 10:54 AM, Thomas Bächler <thomas@archlinux.org> wrote:
>> Am 21.01.2011 17:01, schrieb Allan McRae:
>>> That reply is just wrong... i686 is not a restricted flavour of i386.
>>> It is the other way around. I can not run i686 optimised software on an
>>> i386 system. Just ask all the Via C3 owners who do not have that "nopl"
>>> instruction and the joys they had with glibc-2.12.
>>
>> That is not the point. We should reflect what the dynamic linker tries
>> to do whenever we want to load a binary:
>>
>> 1) If we install a i686 version of libfoo.so.2 onto a i386 system, the
>> linker will still try to load that version (and fail with SIGILL).
>> 2) If we install a x86_64 version and a i686 version of libfoo.so.2 into
>> the system, and launch a 32 bit binary, the linker will determine that
>> it cannot load the the x86_64 version, but the i686 version.
>>
>> So, in order to "do it right", we only need to reflect the distinction
>> the linker does in 2). Not even the linker makes the distinction in 1),
>> so why should we?
>>
>> And now think about use cases: The package architecture is already in
>> PKGINFO. So the only distinction we need to make is between 32 bit and
>> 64 bit binaries - the only case where you are able to run more than one
>> architecture in the same system is when executing 32 bit binaries on a
>> 64 bit platform with a backward compatibility mode for 32 bit. (AFAIK,
>> this is not only the case for x86, but for some other platforms, too).
>>
>> The only info we should include into the soprovides/depends is whether
>> we have a 32 bit or a 64 bit binary. There shouldn't be an explicit
>> statement about architecture.
>
> Taking a step back here, if we want to not lock ourself into ELF-only
> (OSX does not use it), it sounds like using objdump due to it's use of
> the BFD backend which understands multiple file formats. From objdump
> -a, although the output is rather arcane, we should always be able to
> get the file format of the library, which I believe is the most
> important to matching what the linker does.

I do not want to lock anything explicitly to ELF. Considering that OSX
also supports 32 and 64 bit libraries on the same system (and ones that
contain both), it's a bad idea. The question remains, how portable can
we even be here? Do we maybe need support for different binary formats,
implemented in different ways?

>>> I did not like provides=(libfoo.so) magically turning into
>>> "libfoo.so-i686" in the .PKGINFO file. The details of the package
>>> should reflect what is in the PKGBUILD.
>
> See, this is what I still don't understand. Is this libprovides, or is
> this "well we'll hijack your provides array sometimes depending on
> what you put there".
>
> It *still* hasn't been explained how this thing works in either a
> commit message or docs. Do I include soprovides=() in my PKGBUILD? Do
> I need to insert something in provides=() to get a library to show up
> in the final package? Is it all completely automatic- all libraries in
> the package get an automatic provide entry generated? This is why I am
> so damn frustrated with these patches.

The original idea was this: makepkg automatically adds provides for all
libraries with a proper SONAME. These are added as provides to PKGINFO
in addition to the user-specified provides.

Alternatively, the PKGBUILD has to specify which libraries to add. So we
would write
soprovides=(libfoo.so)
and this would turn into a provider for PKGINFO that would contain some
information about architecture and SONAME. I don't like this option
because the list of libraries is maintained by a human and not the computer.

TBH, reading the current patch again, I am unsure what it actually does
now. I think it is using the first option.

>> So, to summarize:
>>
>> 1) Dan does not want a new PKGINFO field to be introduced for this.
> Where did I say that- if I did, I'm not remembering.

Now that you ask, I am not 100% sure you even did. I thought I
remembered something from a few months back. I apologize for not
confirming this.

But back to the point: Would you want this information separate? Adding
a new field to PKGINFO here would essentially duplicate the whole
provides/depends handling, and add new pacman code. When adding this
auto-generated info to PKGINFO's provides and depends, there are no
changes in pacman.

>> 2) Allan does not want any auto-generated meta-info in the provides and
>> depends in PKGINFO.
> No, he stated he doesn't want *magic changes* to data the user
> specified in the PKGBUILD. I think he and I would be more comfortable
> if this was less magic.

And what is "magic"? It is simply automatic. Furthermore, as far as I
can see, there are no changes to the user data, but only additions to it.

>> 3) Dan does not want non-versioning information in the version field in
>> provides/depends in PKGINFO.
> Sure.


So, let's summarize again what we want (because it seems you don't
understand that):

1) We want to have information in the PKGINFO that tells the user and
pacman which libraries are included in the package, and what their
SONAME versions are.

2) We also want (optional?) information in the PKGINFO about which
libraries in which versions are needed to run this package.

3) We want this information automatically generated, not manually
maintained.

The least invasive approach in our opinion was to add this information
to PKGINFO's provides and depends, as it would work without any changes
to pacman. This means that the provides and depends are extended by
auto-generated ones. Nothing is magically modified or removed.

Now, as everything we suggested is so weird to you: How would you want
such a feature implemented?
 
Old 01-21-2011, 05:10 PM
Thomas Bächler
 
Default makepkg: add soprovides support

Am 21.01.2011 19:09, schrieb Thomas Bächler:
> And what is "magic"? It is simply automatic. Furthermore, as far as I
> can see, there are no changes to the user data, but only additions to it.

Heh, wrong, sorry. I discussed so many variants of this with Florian,
that I have lost track of what we do now.
 
Old 01-21-2011, 11:20 PM
Allan McRae
 
Default makepkg: add soprovides support

On 22/01/11 02:54, Thomas Bächler wrote:

Where does that leave us? Essentially, we cannot get this feature
implemented, as all those restrictions combined leave us with no
options. Clearly, you don't see any options either, as you only provide
destructive criticism, no suggestions on how you think it should be done.


I only pointed out the things that I see are wrong because they are what
needed fixed. Also it was 2am when I replied in between dealing with a
sick child so my time for pleasantries was limited. I did not think
everybodies self esteem was so low that I needed to praise the correct
parts too.



And how can we do this portably and subject to the restrictions given...
Well, it has been established that using objdump or elfutils has
issues so lets just exclude them from the start. How else can we get
information about a file? file... That tells you whether a binary is
32 or 64 bit, and is already required by makepkg.



So:

provides=(libfoo.so)

Results in

%PROVIDES%
libfoo.so=6-32

where pkgver is the library soname and pkgrel indicates 32/64 bit. If
the "file" output does not include 32-bit or 64-bit, then just do not
set the pkgrel value.


That solves all comments made so far apart from the documentation needed
on what this does...


Allan
 
Old 01-22-2011, 08:31 AM
Florian Pritz
 
Default makepkg: add soprovides support

On 22.01.2011 01:20, Allan McRae wrote:
> And how can we do this portably and subject to the restrictions given...
> Well, it has been established that using objdump or elfutils has
> issues so lets just exclude them from the start. How else can we get
> information about a file? file... That tells you whether a binary is
> 32 or 64 bit, and is already required by makepkg.

file doesn't tell you the library soname so I don't know what ld will
use for linking. So far readelf seems to be the best choice because I
can also use it to generate the dependencies (right now it seems like we
are too focused on the provides). Your example below can be achieved
using readelf.
Also file's output doesn't have to be the same on different platforms.

> provides=(libfoo.so)
>
> Results in
>
> %PROVIDES%
> libfoo.so=6-32

Are you okay with that format Dan?

--
Florian Pritz -- {flo,bluewind}@server-speed.net
 

Thread Tools




All times are GMT. The time now is 07:24 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org