Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Development (http://www.linux-archive.org/fedora-development/)
-   -   Multilib help? (http://www.linux-archive.org/fedora-development/314854-multilib-help.html)

Bill Nottingham 01-25-2010 06:08 PM

Multilib help?
 
Richard W.M. Jones (rjones@redhat.com) said:
> Is there better documentation on how multilib works and how to resolve
> multilib problems than
> http://fedoraproject.org/wiki/PackagingDrafts/MultilibTricks ? I
> understand that it uses various heuristics before it corrupts files,
> but those don't seem to be documented.
>
> I just spent half a day tracking down a problem which turned out to be
> because multilib had corrupted a file, and I wish to solve this now
> once and for all.

'corrupted a file'? I'm really not sure what you mean here.

In any case, excessive verbiage follows:

...

Fedora, on certain architectures, supports multible binary ABIs.
This includes support for i386 on x86_64, and ppc64 on ppc.

Fedora supports these ABIs through a process called 'multi-arch'
or 'multilib', wherein the corresponding RPM packages for the secondary
arch are included along with their primary arch counterparts. These
packages are built via the same process as the primary arch packages,
and are identical to the packages that would be shipped in a
release where that arch would be the primary arch.

The primary function of multilib support in Fedora is to provide a
runtime environment for applications to use. To satisfy this, packages
that provide runtime libraries are included for the secondary arch.

The secondary function of multilib support in Fedora is to provide
a development environment. To satisfy this, packages that provide
development headers are included for the secondary arch. However,
it should be noted that it is preferred to use a full system of
the appropriate architecture to do development where possible; this
can be done by using the 'mock' package to create an appropriate
chroot.

Multilib packages are intentionally not provided for system daemons
(for example, to run alternate-wordsize versions of Apache or
PostgreSQL.) If you need to run a non-default wordsize of a system
daemon, it is recommended that you install the system with a release
that supports that architecture as the primary architecture.

Not all packages produced by a particular source RPM are included as
multilib packages; only subpackages that implement runtime libraries
or development environments are included.

Whether or not multilib packages are installed by default is controlled by
the 'multilib_policy' configuration file parameter for yum. The default
value for Fedora of 'best' installs only packages for the primary
architecture. If this is changed to 'all' packages for all compatible
architectures will be installed by default. Note that this can only be
changed post-install; anaconda uses the default of only installing packages
for the primary arch.

You can still at any time install packages for a particular arch by passing
the architecture directly to yum. For example, running 'yum install
glibc.i686', or putting 'glibc.i686' in a kickstart %packages stanza, will
install the 32-bit x86 version of glibc.

When installed in parallel, multilib RPM packages may contain files that
live in common paths. When this happens, RPM uses the following algorithm:

- If the file in both packages is identical, installation is allowed
and the file is written
- If the file in both packages is an ELF binary, the file used is the
file in the package for the primary architecture
- If the file in both packages is not an ELF binary a RPM conflict is
raised

If such conflicts are raised with packages in the Fedora tree, it can
be considered a packaging bug that needs to be fixed.

...

Bill
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

"Richard W.M. Jones" 01-25-2010 06:54 PM

Multilib help?
 
On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
> - If the file in both packages is identical, installation is allowed
> and the file is written
> - If the file in both packages is an ELF binary, the file used is the
> file in the package for the primary architecture
> - If the file in both packages is not an ELF binary a RPM conflict is
> raised

Are you sure about the third rule?

I think the problem arises because /usr/bin/libguestfs-supermin-helper
is different in libguestfs.i686 and libguestfs.x86_64, but this file
is a shell script, not an ELF binary.

These are the files in libguestfs.i686 and libguestfs.x86_64 that have
the same names but different content:

/usr/bin/hivexget (ELF binary)
/usr/bin/hivexml (ELF binary)
/usr/bin/libguestfs-test-tool (ELF binary)
/usr/libexec/libguestfs-test-tool-helper (ELF binary)
/usr/bin/libguestfs-supermin-helper (shell script)

Anyway, I will move the shell script and see if that helps.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Panu Matilainen 01-26-2010 09:13 AM

Multilib help?
 
On Mon, 25 Jan 2010, Richard W.M. Jones wrote:

> On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
>> - If the file in both packages is identical, installation is allowed
>> and the file is written
>> - If the file in both packages is an ELF binary, the file used is the
>> file in the package for the primary architecture
>> - If the file in both packages is not an ELF binary a RPM conflict is
>> raised
>
> Are you sure about the third rule?

Yes.

> I think the problem arises because /usr/bin/libguestfs-supermin-helper
> is different in libguestfs.i686 and libguestfs.x86_64, but this file
> is a shell script, not an ELF binary.
>
> These are the files in libguestfs.i686 and libguestfs.x86_64 that have
> the same names but different content:
>
> /usr/bin/hivexget (ELF binary)
> /usr/bin/hivexml (ELF binary)
> /usr/bin/libguestfs-test-tool (ELF binary)
> /usr/libexec/libguestfs-test-tool-helper (ELF binary)
> /usr/bin/libguestfs-supermin-helper (shell script)
>
> Anyway, I will move the shell script and see if that helps.

Wont help, the multilib file "shadowing" logic is NOT based on paths.
Looking at libguestfs.spec, here's the problem:

%global _use_internal_dependency_generator 0

The external dependency generator doesn't create the file "coloring" that
the multilib magic needs to operate. So you'll get conflicts on all paths
where content differs.

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Panu Matilainen 01-26-2010 09:35 AM

Multilib help?
 
On Tue, 26 Jan 2010, Panu Matilainen wrote:

> On Mon, 25 Jan 2010, Richard W.M. Jones wrote:
>
>> On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
>>> - If the file in both packages is identical, installation is allowed
>>> and the file is written
>>> - If the file in both packages is an ELF binary, the file used is the
>>> file in the package for the primary architecture
>>> - If the file in both packages is not an ELF binary a RPM conflict is
>>> raised
>>
>> Are you sure about the third rule?
>
> Yes.
>
>> I think the problem arises because /usr/bin/libguestfs-supermin-helper
>> is different in libguestfs.i686 and libguestfs.x86_64, but this file
>> is a shell script, not an ELF binary.
>>
>> These are the files in libguestfs.i686 and libguestfs.x86_64 that have
>> the same names but different content:
>>
>> /usr/bin/hivexget (ELF binary)
>> /usr/bin/hivexml (ELF binary)
>> /usr/bin/libguestfs-test-tool (ELF binary)
>> /usr/libexec/libguestfs-test-tool-helper (ELF binary)
>> /usr/bin/libguestfs-supermin-helper (shell script)
>>
>> Anyway, I will move the shell script and see if that helps.
>
> Wont help, the multilib file "shadowing" logic is NOT based on paths.
> Looking at libguestfs.spec, here's the problem:
>
> %global _use_internal_dependency_generator 0
>
> The external dependency generator doesn't create the file "coloring" that
> the multilib magic needs to operate. So you'll get conflicts on all paths
> where content differs.

...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with
internal dependency generator, and it gives:

Preparing... ###########################################
[100%]
file /usr/bin/libguestfs-supermin-helper conflicts between
attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and
libguestfs-1:1.0.75-1.fc12.x86_64

...which is the expected behavior for a non-elf file (script in this case
but doesn't matter). If both arch's of libguestfs have been installed,
--replacefiles or --force has been used to force the install despite
of a conflicting file.

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

"Richard W.M. Jones" 01-26-2010 10:18 AM

Multilib help?
 
On Tue, Jan 26, 2010 at 12:35:19PM +0200, Panu Matilainen wrote:
> ...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with
> internal dependency generator, and it gives:
>
> Preparing... ###########################################
> [100%]
> file /usr/bin/libguestfs-supermin-helper conflicts between
> attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and
> libguestfs-1:1.0.75-1.fc12.x86_64

The problem seems to be when you install them in separate RPM
transactions, then remove one (or possibly upgrade one). I haven't
worked out a reliable reproducer, but I do know it can happen on
machines where nothing has been forced.

Anyway, I rewrote the script so that it's the same on all
architectures now.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Panu Matilainen 01-26-2010 03:43 PM

Multilib help?
 
On Tue, 26 Jan 2010, Richard W.M. Jones wrote:

> On Tue, Jan 26, 2010 at 12:35:19PM +0200, Panu Matilainen wrote:
>> ...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with
>> internal dependency generator, and it gives:
>>
>> Preparing... ###########################################
>> [100%]
>> file /usr/bin/libguestfs-supermin-helper conflicts between
>> attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and
>> libguestfs-1:1.0.75-1.fc12.x86_64
>
> The problem seems to be when you install them in separate RPM
> transactions, then remove one (or possibly upgrade one). I haven't
> worked out a reliable reproducer, but I do know it can happen on
> machines where nothing has been forced.

It shouldn't be possible to install packages with conflicting files,
separate transactions or not, without some sort of forcing. If it is, then
its a rather grave bug somewhere in rpm file conflict logic and needs
finding and fixing. A reproducer would be very much appreciated.

Note that rpm < 4.6.0 did behave differently here: packages with
conflicting files were allowed to be installed in the same transaction but
not if installed separately, leading to strange situations. So if there's
a chance you've tested on RHEL/Centos at some point, that could explain it
as it'd then be just a matter of which order the packages got installed
in.

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

"Richard W.M. Jones" 01-26-2010 04:03 PM

Multilib help?
 
On Tue, Jan 26, 2010 at 06:43:35PM +0200, Panu Matilainen wrote:
> Note that rpm < 4.6.0 did behave differently here: packages with
> conflicting files were allowed to be installed in the same transaction but
> not if installed separately, leading to strange situations. So if there's
> a chance you've tested on RHEL/Centos at some point, that could explain it
> as it'd then be just a matter of which order the packages got installed
> in.

That could well explain why we've only seen this reported on EPEL 5.
I thought it was a coincidence, but looks like it could be the older
RPM version there.

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Panu Matilainen 01-26-2010 04:47 PM

Multilib help?
 
On Tue, 26 Jan 2010, Richard W.M. Jones wrote:

> On Tue, Jan 26, 2010 at 06:43:35PM +0200, Panu Matilainen wrote:
>> Note that rpm < 4.6.0 did behave differently here: packages with
>> conflicting files were allowed to be installed in the same transaction but
>> not if installed separately, leading to strange situations. So if there's
>> a chance you've tested on RHEL/Centos at some point, that could explain it
>> as it'd then be just a matter of which order the packages got installed
>> in.
>
> That could well explain why we've only seen this reported on EPEL 5.
> I thought it was a coincidence, but looks like it could be the older
> RPM version there.

Okay, if it has only been seen on EPEL then no need to look further. IIRC
it got fixed in RHEL 5 too, 5.3 or thereabouts.

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Till Maas 01-26-2010 08:34 PM

Multilib help?
 
On Tue, Jan 26, 2010 at 12:13:33PM +0200, Panu Matilainen wrote:

> %global _use_internal_dependency_generator 0
>
> The external dependency generator doesn't create the file "coloring" that
> the multilib magic needs to operate. So you'll get conflicts on all paths
> where content differs.

How can one both get the file "coloring" and add own dependencies with a
script to an rpm? Iirc the internal dependency generator needs to be
disabled to be able to specify a special script to add dependencies.

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Panu Matilainen 01-28-2010 09:07 PM

Multilib help?
 
On Tue, 26 Jan 2010, Till Maas wrote:

> On Tue, Jan 26, 2010 at 12:13:33PM +0200, Panu Matilainen wrote:
>
>> %global _use_internal_dependency_generator 0
>>
>> The external dependency generator doesn't create the file "coloring" that
>> the multilib magic needs to operate. So you'll get conflicts on all paths
>> where content differs.
>
> How can one both get the file "coloring" and add own dependencies with a
> script to an rpm? Iirc the internal dependency generator needs to be
> disabled to be able to specify a special script to add dependencies.

The sorry state of things right now is that there's no generic way that
works in every situation with the internal dependency generator, but for
some cases its possible:

1) All the require/provide helper scripts the internal dep generator uses
can be overridden and customized, such as the __perl_provides/requires
scripts.
2) Custom requires etc can be added in a scripted manner with
Requires: %(some-custom-script)
but this is rather limited as the script would run at early spec
parsing state, where one typically wants to generate dependencies from
what was just compiled. Which is possible (but cumbersome) for the
things that use helper scriptlets, ie case 1)

FWIW, I'm working on it. The internal dependency generator is a serious
bottleneck to what packagers would want/need to do, and the pile of ugly
hacks in it is getting a bit much.

The first step is probably to move all the actual dependency generation
work out of librpmbuild into helpers that can be overridden and
customized, so that 1) becomes a general (even if cumbersome) solution.
The main offender is elf dependency extraction which needs to be lifted
into separate helper, which should also open new possibilities for dealing
with non-native elf files (ie cross-builds).

It needs a whole lot more than that of course but you gotta start
somewhere...

- Panu -
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel


All times are GMT. The time now is 11:32 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.