Early adopters of symbol based dependencies needed
 

Hello,

since the upload of dpkg 1.14.8 to unstable, it's now possible for
library packages to generate "symbols" control files that will be used by
other packages to get more accurate (and less strict) dependencies.

As this is a far reaching change, I'd like some skilled maintainers to try
it out "for real" and help me flesh out some generic guidelines so that when
we start using it on more packages, we do it right from the first time.

The wiki page http://wiki.debian.org/UsingSymbolsFiles will be used to
collect various remarks and prepare some useful documentation that will
be later spread to maintainers through d-d-a (and maybe merged back in
some dpkg manual page).

Integrated documentation
------------------------
The existing documentation is integrated in various dpkg manual pages:
- dpkg-gensymbols(1)
- dpkg-shlibdeps(1)
- deb-symbols(5)

Debhelper support
-----------------
With debhelper 5.0.61, dh_makeshlibs will also call dpkg-gensymbols if it
finds debian/<package>.symbols (or debian/<package>.symbols.<arch>). So
for packages using debhelper, the only thing to do is to drop the right
symbol file(s) at the right place and add build-depends on dpkg-dev (>= 1.14.9)
and debhelper (>= 5.0.61).

Some pre-generated symbols files can be downloaded on
http://qa.debian.org/cgi-bin/mole/seedsymbols/

Beware, those files have been auto-generated and should be verified by the
maintainer (check that the version are correct, I just fixed a bug where
epoch were lost). Adapt the dependency field if needed, verify if
the dependency associated to some symbols have to be bumped (see below),
etc.

Warnings
--------
1/ The symbols files associate a minimum version to each symbol that an
application might be using. By default, that version is calculated as the
first version where the symbol appeared. This is the best approximation
that we can automatically generate. Please note however that the function
might have been extended in backwards-compatible ways which would still
deserve an increment of the minimal required version (for example when new
values are allowed in existing parameters which were previously not
accepted). To detect such problems one must have a good knowledge of the
API of the library.

It would be great to have an exhaustive list of similar cases that
maintainers should be watching for in upstream ChangeLogs.

2/ Watch out if adding support of symbols files break unofficial
architectures (like armel or kfreebsd-i386/amd64). Because the
pre-generated files only take into account the current list of official
architectures, so you might not be aware of some differences in the symbol
set. You can check the build log of your package on unofficial
architectures on http://experimental.debian.net/
(http://experimental.debian.net/build.php?pkg=glibc for example).

Feel free to make suggestions to avoid such breakage when possible. See
also discussions in http://bugs.debian.org/452022 on that topic.

Cheers,
--
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

Le mardi 20 novembre 2007 à 16:22 +0100, Raphael Hertzog a écrit :
> 2/ Watch out if adding support of symbols files break unofficial
> architectures (like armel or kfreebsd-i386/amd64). Because the
> pre-generated files only take into account the current list of official
> architectures, so you might not be aware of some differences in the symbol
> set. You can check the build log of your package on unofficial
> architectures on http://experimental.debian.net/
> (http://experimental.debian.net/build.php?pkg=glibc for example).
>
> Feel free to make suggestions to avoid such breakage when possible. See
> also discussions in http://bugs.debian.org/452022 on that topic.

A guaranteed way to trigger such issues is to export all functions built
in the library as symbols, which is unfortunately the default behavior.
This way the list will change if you change the optimization parameters
or on unofficial architectures.

If upstream doesn't support this, the simplest way to restrict the list
of functions is to use a version script ; this script can be written
manually or automatically generated by libtool with the -export-symbols
or -export-symbols-regex options.

--
.'`.
: :' : We are debian.org. Lower your prices, surrender your code.
`. `' We will add your hardware and software distinctiveness to
`- our own. Resistance is futile.

Early adopters of symbol based dependencies needed
 

So FWIW, I have aalib using a symbol file, it's what I used to test the
debhelper modifications. Haven't uploaded it yet because it will have to
build-depend on the recent dpkg bugfix. Also because dpkg-shlibdeps is
now smart enough to complain about some unnecessary linkages to things
like libm and libslang in libaa-bin, and I haven't sat down and figured
out if I should really remove those direct linkages (probably).

I also thought about using symbol files for the not very shared
libraries in the fbreader source package, but there's C++ symbol
mangling going on, so I think I can't.

> 1/ The symbols files associate a minimum version to each symbol that an
> application might be using. By default, that version is calculated as the
> first version where the symbol appeared.

Assuming the first version of the symbol is at least in oldstable. :-)

--
see shy jo

Early adopters of symbol based dependencies needed
 

Raphael Hertzog wrote:
> With debhelper 5.0.61, dh_makeshlibs will also call dpkg-gensymbols if it
> finds debian/<package>.symbols (or debian/<package>.symbols.<arch>). So
> for packages using debhelper, the only thing to do is to drop the right
> symbol file(s) at the right place and add build-depends on dpkg-dev (>= 1.14.9)
> and debhelper (>= 5.0.61).

I suppose you'd also want to stop passing -V to dh_makeshlibs,
since the symbol versioning should take care of setting the version.
-V without any parameters is evil anyway.

Although, what happens if a package is built against the library on
a buildd that doesn't have the new dpkg-dev yet? It would generate a
wrong unversioned dependency in this case.

--
see shy jo

Early adopters of symbol based dependencies needed
 

On Tue, 20 Nov 2007, Joey Hess wrote:
> I also thought about using symbol files for the not very shared
> libraries in the fbreader source package, but there's C++ symbol
> mangling going on, so I think I can't.

You can, but it generally means having to handle arch-specific symbols
files because symbol mangling vary from arch to arch (at least between
32bit and 64bit arch).

> > 1/ The symbols files associate a minimum version to each symbol that an
> > application might be using. By default, that version is calculated as the
> > first version where the symbol appeared.
>
> Assuming the first version of the symbol is at least in oldstable. :-)

In fact, I scanned stable, testing and unstable to create the initial
files. So, oldstable has not been considered. And now the files are kept
up-to-date with all unstable uploads (experimental ones are not
considered).

Cheers,
--
Raphaël Hertzog

Premier livre français sur Debian GNU/Linux :
http://www.ouaza.com/livre/admin-debian/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

On Tue, Nov 20, 2007 at 01:20:33PM -0500, Joey Hess wrote:
> Raphael Hertzog wrote:
> > With debhelper 5.0.61, dh_makeshlibs will also call dpkg-gensymbols if it
> > finds debian/<package>.symbols (or debian/<package>.symbols.<arch>). So
> > for packages using debhelper, the only thing to do is to drop the right
> > symbol file(s) at the right place and add build-depends on dpkg-dev (>= 1.14.9)
> > and debhelper (>= 5.0.61).
>
> I suppose you'd also want to stop passing -V to dh_makeshlibs,
> since the symbol versioning should take care of setting the version.
> -V without any parameters is evil anyway.
>
> Although, what happens if a package is built against the library on
> a buildd that doesn't have the new dpkg-dev yet? It would generate a
> wrong unversioned dependency in this case.

Do buildds have special rules for build dependencies on dpkg-dev ? O_o

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

Raphael Hertzog <hertzog@debian.org> writes:

> Hello,
>
> since the upload of dpkg 1.14.8 to unstable, it's now possible for
> library packages to generate "symbols" control files that will be used
> by other packages to get more accurate (and less strict) dependencies.
>
> As this is a far reaching change, I'd like some skilled maintainers to
> try it out "for real" and help me flesh out some generic guidelines so
> that when we start using it on more packages, we do it right from the
> first time.
>
> The wiki page http://wiki.debian.org/UsingSymbolsFiles will be used to
> collect various remarks and prepare some useful documentation that will
> be later spread to maintainers through d-d-a (and maybe merged back in
> some dpkg manual page).

Hi Raphael,

I'll give this a try for libremctl1. It seems like a good candidate for a
test of something that's still fairly experimental: there are currently no
reverse-depends in Debian, it's a small library, it's already using symbol
versioning, and I'm also upstream.

> Some pre-generated symbols files can be downloaded on
> http://qa.debian.org/cgi-bin/mole/seedsymbols/

This is awesome. Thank you very much for doing this. It makes it much
easier to start.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

On Tue, Nov 20, 2007 at 04:22:56PM +0100, Raphael Hertzog wrote:
> Some pre-generated symbols files can be downloaded on
> http://qa.debian.org/cgi-bin/mole/seedsymbols/
>
> Beware, those files have been auto-generated and should be verified by the
> maintainer (check that the version are correct, I just fixed a bug where
> epoch were lost). Adapt the dependency field if needed, verify if
> the dependency associated to some symbols have to be bumped (see below),
> etc.

Are the @Base in these files really necessary ? I mean, most packages
have no symbol versioning and thus use the "Base" version. Does it work
without it being explicitly put ? If not, don't you think it would make
sense ?

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

On Tue, Nov 20, 2007 at 10:05:41PM +0100, Mike Hommey wrote:
> On Tue, Nov 20, 2007 at 01:20:33PM -0500, Joey Hess wrote:
> > Raphael Hertzog wrote:
> > > With debhelper 5.0.61, dh_makeshlibs will also call dpkg-gensymbols if it
> > > finds debian/<package>.symbols (or debian/<package>.symbols.<arch>). So
> > > for packages using debhelper, the only thing to do is to drop the right
> > > symbol file(s) at the right place and add build-depends on dpkg-dev (>= 1.14.9)
> > > and debhelper (>= 5.0.61).
> >
> > I suppose you'd also want to stop passing -V to dh_makeshlibs,
> > since the symbol versioning should take care of setting the version.
> > -V without any parameters is evil anyway.
> >
> > Although, what happens if a package is built against the library on
> > a buildd that doesn't have the new dpkg-dev yet? It would generate a
> > wrong unversioned dependency in this case.
>
> Do buildds have special rules for build dependencies on dpkg-dev ? O_o

Oh now I understood. Well, I guess the -V should still be used...

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Early adopters of symbol based dependencies needed
 

Raphael Hertzog <hertzog@debian.org> writes:

> since the upload of dpkg 1.14.8 to unstable, it's now possible for
> library packages to generate "symbols" control files that will be used
> by other packages to get more accurate (and less strict) dependencies.

> As this is a far reaching change, I'd like some skilled maintainers to
> try it out "for real" and help me flesh out some generic guidelines so
> that when we start using it on more packages, we do it right from the
> first time.

A couple of early points of feedback, which I'll send here since I think
they'd benefit from broader discussion:

* The seed symbols files include the full Debian package version for when
the symbol was first available. For example, for libremctl1, all of
the symbols are marked as first available in 2.10-1.

This causes a problem for backports. A backport of 2.10-1 would
provide all the same symbols, but would have a version of
2.10-1~bpo40+1 or something similar. This would be less than the
version advertised by the symbols file, which means that the backport
wouldn't satisfy the regular package dependency even though it would
work. This will mean that symbols files will have to be adjusted when
building backported packages, unless I miss some subtlety.

I always use dependencies like >= 2.10 in shlibs files rather than the
more specific 2.10-1 because of this problem. I'm not sure if that's
the right general solution, but people who start from the seed files
should at least consider removing the Debian revision in cases like
this to make backporting easier.

* The new warnings from the dpkg-* tools warn about any binary Perl
module because all binary Perl modules use symbols from Perl itself but
traditionally aren't linked directly against libperl. (There was some
reason for this that I don't recall off-hand.) Should these warnings
just be ignored? Suppressed in some way? Should binary Perl modules
link against libperl? I haven't worked through the implications in my
head yet.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org