Fun with libtool and cross-builds
 

Libtool is intended to make library linking 'just work' whatever the
details of your build mechanisms are.

However in Debian/Ubuntu cross-building it seems to go out of its way
to make it not-work. The problem is papered-over when using i386
machines, or amd64 machines and a new-enough (or old enough) version
of binutils, but this seems to me to be a very poor solution as it has
to be put into the linker for every cross-combination people might
choose to build between.

Making libtool actually do the right thing would be much better. In
fact it would be correct, and we like correctness.

This is not a new problem. In a bit of research I found a post
suggesting it was actually broken in 2002 (I've not yet looked into this).

(a long list of previous posts on the issue from OE, Debian, Ubuntu,
astlinux and buildroot people is at the end of this mail).

Now, before I get too carried away I want to check that I'm not just
misunderstanding something. It seems faintly crazy that this very
straightforward cross-comiling problem is not already solved.

And I have to admit that I have always done my best to avoid
understanding the details of libtool (like most people). However I
think the time has come to sort this out properly, if in fact it
hasn't been already.

The problem
-----------

We are crossbuilding Debian packages, that build libraries, which link to
other libraries. There are lots of those, but I'm using unixodbc and
libprce3 here as they seem fairly typical examples.

The libraries to link against are installed into a directory,
historically /usr/<triplet>/lib, but moving to
/usr/lib/<host-multiarchtuple> using multiarch mechanisms. They could also
be in sysroot locations if using sysroot mechanisms). The point is that they are _not_ in
/usr/lib (or /usr/lib/<build-multiarchtuple> in multiarch world).

libtool gets most of this process right (or at least not wrong), until
the library is installed into the package location typically
(debian/tmp/usr/lib) using libtool's 'install' mode, which then calls
libtool with mode 'relink'.

At this points it calls the linker and adds -L/usr/lib on the front -
thereby adding this path in front of the default cross-compiler path.

If called without this -L, or called with -L/usr/<triplet>/lib (i.e
the correct path, wherever that is), then everything is fine because
the libraries for the correct (host) architecture are found. So there
are 2 ways to get this right and one way to get it wrong, which is the
one libtool picks.

The workarounds which mean this has not yet been fixed are that the
linker used to just issue a warning and ignore libraries
which it didn't understand:
/usr/lib/libgcc_s.so: file not recognized: File format not recognized

And more recently it has been taught to understand specific
architectures enough to decide they are the wrong ones:
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for
-lpthread
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for
-lpthread
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc

However on amd64 machine with armel toolchain and
binutils-arm-linux-gnueabi_2.20.51.20100908-0ubuntu2cross1.52 the
skipping fails for static libs:
/usr/lib/libc.a: could not read symbols: File format not recognized
and the build fails.

on i386, or later versions of binutils, these are skipped the same as
the .so s.

However as stated above I don't think relying on this linker
behaviour is any real sort of solution - we should get libtool to
either leave it up to the toolchain, or pass the correct explicit
path.

Here is the example unixodbc case that illustrates the problem:
(Built with: xdeb -a armel --apt-source unixodbc, which runs:
dpkg-buildpackage -rfakeroot -d -us -uc -aarmel -b -tc )

make[4]: Entering directory /home/chroot-user/build/unixodbc-2.2.14p2/samples'
test -z "/usr/lib" || mkdir -p -- "/home/chroot-user/build/unixodbc-2.2.14p2/debian/tmp/usr/lib"
/bin/bash ../libtool --mode=install /usr/bin/install -c 'libboundparam.la' '/home/chroot-user/build/unixodbc-2.2.14p2/debian/tmp/usr/lib/libboundparam.la'
libtool: install: warning: relinking libboundparam.la'
libtool: install: (cd /home/chroot-user/build/unixodbc-2.2.14p2/samples; /bin/bash /home/chroot-user/build/unixodbc-2.2.14p2/libtool --tag CC --mode=relink arm-l$
libtool: relink: arm-linux-gnueabi-gcc -shared .libs/boundparam.o .libs/helper.o .libs/cboundtimestampparam.o -Wl,--whole-archive ../autotest/.libs/libgtrtstlc.$
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/lib/libc.a: could not read symbols: File format not recognized
collect2: ld returned 1 exit status
libtool: install: error: relink libboundparam.la' with the above command before installing it
make[4]: *** [install-libLTLIBRARIES] Error 1

I note that when doing this libtool --config shows that it does have
an idea of where the right place is:
maverick)wookey@kh:~/ubuntu/maverick/build/pcre3$ ./libtool --config | grep sys_lib
sys_lib_search_path_spec="/usr/lib/gcc/arm-linux-gnueabi/4.4.5 /usr/arm-linux-gnueabi/lib"
sys_lib_dlsearch_path_spec="/usr/arm-linux-gnueabi/lib /lib /usr/lib /usr/local/lib"

I haven't worked out exactly when and where libtool applies those,
but as it seems to get the compiling and linking part right they may
be being used (or the compiler defaults are just working as they
should). It's only the install/relink phase that is bust (so far as I
can tell).

pcre3 provides a simpler and quicker-to-build example of the same
problem.

Here are various previous postings on the subject:
Simon Richter in march last year, to libtool list:
http://lists.gnu.org/archive/html/libtool/2010-03/msg00013.html
(no response)

Loic Minier in October last year, to libtool list
http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/7626
(response: yes that seems to be a bug, no time to fix now, does sysroot
option fix it? 'Not for me' said lool)

Philip Prindeville from astlinux, July 2010, to libtool list
http://lists.gnu.org/archive/html/libtool/2010-07/msg00018.html
(response: send us some reproducible examples, and 'heres a ptch for
the sysroot-and-deleting-.la-files case')

Martin Panter from Openembedded in December 2010, to libtool-patches-gnu
http://osdir.com/ml/libtool-patches-gnu/2011-01/msg00025.html
which includes a patch to fix it.
(response: useful discussion of possible patches)

Opinions on the best way to make libtool do the right thing both now
and in the future are welcome. Ther are quite a few patches in the
above threads providing different solutions, and I haven't yet
determined which of them have been included where.


Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110209190247.GC22368@dream.aleph1.co.uk">http://lists.debian.org/20110209190247.GC22368@dream.aleph1.co.uk

Fun with libtool and cross-builds
 

Wookey <wookey@wookware.org> writes:

> At this points it calls the linker and adds -L/usr/lib on the front -
> thereby adding this path in front of the default cross-compiler path.

Please try to debug where the -L/usr/lib comes from, I don't believe
libtool is adding this by itself but instead it is told to add it by
some incorrect .la file or similar.

/Simon


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87ei7hx93u.fsf@latte.josefsson.org">http://lists.debian.org/87ei7hx93u.fsf@latte.josefsson.org

Fun with libtool and cross-builds
 

On 09/02/11 22:10, Simon Josefsson wrote:


Please try to debug where the -L/usr/lib comes from, I don't believe
libtool is adding this by itself but instead it is told to add it by
some incorrect .la file or similar.



This is from memory, as I also ran into the same problem. The problem
is, if my memory serves me right, in the following scenario:

Cross build library A and install it with DESTDIR (obviously) to
/tmp/otheroot

Try to cross build library B, that needs library A. Obviously, you're
going to give it -L /tmp/otheroot/usr/lib. The problem is that the .la
files in /tmp/otheroot/usr/lib point to /usr/lib as the place to find
the library. This only becomes a problem when a native version of
library A is installed on the real system, at which point the compiler
prefers that version, and then fails as it is of the wrong architecture.



In my projects I wound up "fixing" the la file with sed as part of the
build process, but that was an embedded project where:


The la files were not installed to the destination machine, and
The entire build was controlled by a single governing makefile.

If either one of these conditions is not present, my solution simply
wouldn't work. Another solution is to build A with
--prefix=/tmp/otheroot/usr, but I think we can all see how that might
break stuff in the library if the destination path is used inside the
code.



The only real solution I see for this is for libtool to have specific
support for linking against DESTDIR installed libraries (maybe make it
respect DESTDIR if it's defined during the build? That could be a
solution that is both easy to understand and simple to integrate)



Shachar
--
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Fun with libtool and cross-builds
 

]] Shachar Shemesh

| The only real solution I see for this is for libtool to have specific
| support for linking against DESTDIR installed libraries (maybe make it
| respect DESTDIR if it's defined during the build? That could be a
| solution that is both easy to understand and simple to integrate)

Isn't a much simpler solution just to get rid of .la files for platforms
where they're useless? They're not needed for dynamic linking and
Debian doesn't really support static linking well, and if you do need
static linking, you can just use pkg-config which more and more libs
support those days.

--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 874o8c8a60.fsf@qurzaw.varnish-software.com">http://lists.debian.org/874o8c8a60.fsf@qurzaw.varnish-software.com

Fun with libtool and cross-builds
 

[This was also posted to debian-devel but I got the linaro addesss
wrong: http://lists.debian.org/debian-devel/2011/02/msg00196.html]

Libtool is intended to make library linking 'just work' whatever the
details of your build mechanisms are.

However in Debian/Ubuntu cross-building it seems to go out of its way
to make it not-work. The problem is papered-over when using i386
machines, or amd64 machines and a new-enough (or old enough) version
of binutils, but this seems to me to be a very poor solution as it has
to be put into the linker for every cross-combination people might
choose to build between.

Making libtool actually do the right thing would be much better. In
fact it would be correct, and we like correctness.

This is not a new problem. In a bit of research I found a post
suggesting it was actually broken in 2002 (I've not yet looked into this).

(a long list of previous posts on the issue from OE, Debian, Ubuntu,
astlinux and buildroot people is at the end of this mail).

Now, before I get too carried away I want to check that I'm not just
misunderstanding something. It seems faintly crazy that this very
straightforward cross-comiling problem is not already solved.

And I have to admit that I have always done my best to avoid
understanding the details of libtool (like most people). However I
think the time has come to sort this out properly, if in fact it
hasn't been already.

The problem
-----------

We are crossbuilding Debian packages, that build libraries, which link to
other libraries. There are lots of those, but I'm using unixodbc and
libprce3 here as they seem fairly typical examples.

The libraries to link against are installed into a directory,
historically /usr/<triplet>/lib, but moving to
/usr/lib/<host-multiarchtuple> using multiarch mechanisms. They could also
be in sysroot locations if using sysroot mechanisms). The point is that they are _not_ in
/usr/lib (or /usr/lib/<build-multiarchtuple> in multiarch world).

libtool gets most of this process right (or at least not wrong), until
the library is installed into the package location typically
(debian/tmp/usr/lib) using libtool's 'install' mode, which then calls
libtool with mode 'relink'.

At this points it calls the linker and adds -L/usr/lib on the front -
thereby adding this path in front of the default cross-compiler path.

If called without this -L, or called with -L/usr/<triplet>/lib (i.e
the correct path, wherever that is), then everything is fine because
the libraries for the correct (host) architecture are found. So there
are 2 ways to get this right and one way to get it wrong, which is the
one libtool picks.

The workarounds which mean this has not yet been fixed are that the
linker used to just issue a warning and ignore libraries
which it didn't understand:
/usr/lib/libgcc_s.so: file not recognized: File format not recognized

And more recently it has been taught to understand specific
architectures enough to decide they are the wrong ones:
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for
-lpthread
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for
-lpthread
/usr/arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc

However on amd64 machine with armel toolchain and
binutils-arm-linux-gnueabi_2.20.51.20100908-0ubuntu2cross1.52 the
skipping fails for static libs:
/usr/lib/libc.a: could not read symbols: File format not recognized
and the build fails.

on i386, or later versions of binutils, these are skipped the same as
the .so s.

However as stated above I don't think relying on this linker
behaviour is any real sort of solution - we should get libtool to
either leave it up to the toolchain, or pass the correct explicit
path.

Here is the example unixodbc case that illustrates the problem:
(Built with: xdeb -a armel --apt-source unixodbc, which runs:
dpkg-buildpackage -rfakeroot -d -us -uc -aarmel -b -tc )

make[4]: Entering directory /home/chroot-user/build/unixodbc-2.2.14p2/samples'
test -z "/usr/lib" || mkdir -p -- "/home/chroot-user/build/unixodbc-2.2.14p2/debian/tmp/usr/lib"
/bin/bash ../libtool --mode=install /usr/bin/install -c 'libboundparam.la' '/home/chroot-user/build/unixodbc-2.2.14p2/debian/tmp/usr/lib/libboundparam.la'
libtool: install: warning: relinking libboundparam.la'
libtool: install: (cd /home/chroot-user/build/unixodbc-2.2.14p2/samples; /bin/bash /home/chroot-user/build/unixodbc-2.2.14p2/libtool --tag CC --mode=relink arm-l$
libtool: relink: arm-linux-gnueabi-gcc -shared .libs/boundparam.o .libs/helper.o .libs/cboundtimestampparam.o -Wl,--whole-archive ../autotest/.libs/libgtrtstlc.$
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/lib/gcc/arm-linux-gnueabi/4.5.1/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/lib/libc.a: could not read symbols: File format not recognized
collect2: ld returned 1 exit status
libtool: install: error: relink libboundparam.la' with the above command before installing it
make[4]: *** [install-libLTLIBRARIES] Error 1

I note that when doing this libtool --config shows that it does have
an idea of where the right place is:
maverick)wookey@kh:~/ubuntu/maverick/build/pcre3$ ./libtool --config | grep sys_lib
sys_lib_search_path_spec="/usr/lib/gcc/arm-linux-gnueabi/4.4.5 /usr/arm-linux-gnueabi/lib"
sys_lib_dlsearch_path_spec="/usr/arm-linux-gnueabi/lib /lib /usr/lib /usr/local/lib"

I haven't worked out exactly when and where libtool applies those,
but as it seems to get the compiling and linking part right they may
be being used (or the compiler defaults are just working as they
should). It's only the install/relink phase that is bust (so far as I
can tell).

pcre3 provides a simpler and quicker-to-build example of the same
problem.

Here are various previous postings on the subject:
Simon Richter in march last year, to libtool list:
http://lists.gnu.org/archive/html/libtool/2010-03/msg00013.html
(no response)

Loic Minier in October last year, to libtool list
http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/7626
(response: yes that seems to be a bug, no time to fix now, does sysroot
option fix it? 'Not for me' said lool)

Philip Prindeville from astlinux, July 2010, to libtool list
http://lists.gnu.org/archive/html/libtool/2010-07/msg00018.html
(response: send us some reproducible examples, and 'heres a ptch for
the sysroot-and-deleting-.la-files case')

Martin Panter from Openembedded in December 2010, to libtool-patches-gnu
http://osdir.com/ml/libtool-patches-gnu/2011-01/msg00025.html
which includes a patch to fix it.
(response: useful discussion of possible patches)

Opinions on the best way to make libtool do the right thing both now
and in the future are welcome. Ther are quite a few patches in the
above threads providing different solutions, and I haven't yet
determined which of them have been included where.


Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110210105954.GA20683@dream.aleph1.co.uk">http://lists.debian.org/20110210105954.GA20683@dream.aleph1.co.uk

Fun with libtool and cross-builds
 

On Thu, Feb 10, 2011, Wookey wrote:
> Loic Minier in October last year, to libtool list
> http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/7626
> (response: yes that seems to be a bug, no time to fix now, does sysroot
> option fix it? 'Not for me' said lool)

FTR, I just tried libtool.git, and it seems to still be affected by the
issue

./bootstrap
cd tests/depdemo
./configure --build=x86_64-linux-gnu --host=arm-linux-gnueabi --libdir=/usr/lib
make install DESTDIR=`pwd`/destdir

test -z "/usr/lib" || /bin/mkdir -p "/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib"
/bin/bash ../libtool --mode=install /usr/bin/install -c libl2.la '/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib'
libtool: install: warning: relinking `libl2.la'
libtool: install: (cd /home/lool/scratch/libtool/libtool-git/tests/depdemo/l2; /bin/bash /home/lool/scratch/libtool/libtool-git/tests/depdemo/libtool --tag CC --mode=relink arm-linux-gnueabi-gcc -g -O2 -no-undefined -o libl2.la -rpath /usr/lib l2.lo ../l1/libl1.la -inst-prefix-dir /home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir)
libtool: relink: arm-linux-gnueabi-gcc -shared -fPIC -DPIC .libs/l2.o -L/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib -L/usr/lib -ll1 -O2 -Wl,-soname -Wl,libl2.so.0 -o .libs/libl2.so.0.0.0
/usr/lib/gcc/arm-linux-gnueabi/4.5.2/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/lib/libc.a: could not read symbols: File format not recognized
collect2: ld returned 1 exit status
libtool: install: error: relink `libl2.la' with the above command before installing it


I'd recommend you work with test cases (not necessarily libtool's) as
various compilation mode might impose subtly different constraints on
our tools (toolchain, libtool and others). This will help us give
upstreams a recipe to reproduce and will also keep our tools working
for use cases we care about. Here, we can try changing libtool and see
if the above test passes -- it's not really an isolated test ATM
though.

--
Loïc Minier


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110210112652.GB2051@bee.dooz.org">http://lists.debian.org/20110210112652.GB2051@bee.dooz.org

Fun with libtool and cross-builds
 

* Simon Josefsson <simon@josefsson.org> schrieb:
> Wookey <wookey@wookware.org> writes:
>
> > At this points it calls the linker and adds -L/usr/lib on the front -
> > thereby adding this path in front of the default cross-compiler path.
>
> Please try to debug where the -L/usr/lib comes from, I don't believe
> libtool is adding this by itself but instead it is told to add it by
> some incorrect .la file or similar.

When I first encountered that problem (more precisely: misdesign)
about 10yrs ago, libtool indeed added it explicitly, *and* it
also was taken from the .la files.

Meanwhile the first bug seems to be away now, but .la files still
tend to be broken. As nobody really needs same (at least nobody
who's not completely lobotomized ;-o), simply remove them all.


cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/

phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110226171057.GA10616@nibiru.local">http://lists.debian.org/20110226171057.GA10616@nibiru.local

Fun with libtool and cross-builds
 

* Loïc Minier <loic.minier@linaro.org> schrieb:
> On Thu, Feb 10, 2011, Wookey wrote:
> > Loic Minier in October last year, to libtool list
> > http://thread.gmane.org/gmane.comp.gnu.libtool.bugs/7626
> > (response: yes that seems to be a bug, no time to fix now, does sysroot
> > option fix it? 'Not for me' said lool)
>
> FTR, I just tried libtool.git, and it seems to still be affected by the
> issue
>
> ./bootstrap
> cd tests/depdemo
> ./configure --build=x86_64-linux-gnu --host=arm-linux-gnueabi --libdir=/usr/lib
> make install DESTDIR=`pwd`/destdir
>
> test -z "/usr/lib" || /bin/mkdir -p "/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib"
^^^^^^^^^^
absolutely wrong.

> /bin/bash ../libtool --mode=install /usr/bin/install -c libl2.la '/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib'
> libtool: install: warning: relinking `libl2.la'
> libtool: install: (cd /home/lool/scratch/libtool/libtool-git/tests/depdemo/l2; /bin/bash /home/lool/scratch/libtool/libtool-git/tests/depdemo/libtool --tag CC --mode=relink arm-linux-gnueabi-gcc -g -O2 -no-undefined -o libl2.la -rpath /usr/lib l2.lo ../l1/libl1.la -inst-prefix-dir /home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir)
> libtool: relink: arm-linux-gnueabi-gcc -shared -fPIC -DPIC .libs/l2.o -L/home/lool/scratch/libtool/libtool-git/tests/depdemo/destdir/usr/lib -L/usr/lib -ll1 -O2 -Wl,-soname -Wl,libl2.so.0 -o .libs/libl2.so.0.0.0
> /usr/lib/gcc/arm-linux-gnueabi/4.5.2/../../../../arm-linux-gnueabi/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
> /usr/lib/libc.a: could not read symbols: File format not recognized
> collect2: ld returned 1 exit status
> libtool: install: error: relink `libl2.la' with the above command before installing it

Obvious, since it uses wrong library pathes ;-o


cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/

phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110227181848.GB10616@nibiru.local">http://lists.debian.org/20110227181848.GB10616@nibiru.local