Hi,

Some packages require running with versions of libraries at least as
high as the versions used during the build. I think we should offer
packagers a system to generate dependencies close to this, but not as
strict. This would complement our current infrastructure nicely,
some of the rationale follows.


In Debian, we have the shlibs system and now the "symbols" system to
inject dependencies when ELF binaries depend on shared libs or symbols
in shared libs.

The shlibs system worked nicely and permits injecting high deps when
the ABI is augmented backwards-compatibly, but this is not enough in
all cases:
Consider libfoo which doesn't change API/ABI, but sees a bug fix,
shlibs aren't bumped as the lib is backwards-compatible. Package
foobar detects the bugged version of libfoo at build time, and
enables/disables a workaround. The shlibs/symbol systems will
neither prevent foobar to build the workaround and run with the new
libfoo, nor to disable the workaround and run with the old libfoo.

The symbols system currently bypasses shlibs and will make sure that
you get a version of a lib providing all ELF symbols that your binaries
need, but like the shlibs system it only sees the ELF level:
Consider libfoo2 which manipulates frozzles and doesn't change
API/ABI at the ELF level, but gains support for a new frozzle.
Package foobar2a needs the new frozzle, package foobar2b doesn't.
The shlibs or symbols systems will either a) inject dependencies on the
new version and hence give overly high dependencies for foobar2b or
b) inject deps on the old version and hence give too loose deps
(requiring the foobar2a to add a manual dependencies to get the new
frozzle).

In general, the shlibs and symbols systems inject dependencies which we
would like as low as possible for each package [1], but the amount of
information in the binaries is insufficient and encourages maintainers
to bump injected dependencies at high levels.

Similar problems exist outside of ELF binaries, such as across Python
packages.


An idea that came up is to use a per-dependent package information
provided by the maintainer such as the build-deps version [2]. It
would require a map from deps to build-deps and could typically be
combined with the existing systems to inject a dependency >=
max(shlibdeps version, bdeps version).

An example would be package foobar3 build-depending on libfoo3 (>=
2.10.2). When built against libfoo3 2.12.0-2 which provides a symbols
file, the resulting binaries need symbols from libfoo3 (>= 2.8.7) (as
the symbols file say), the new system injects a dep on libfoo3 >=
max(2.10.2, 2.8.7) == 2.10.2.
If libfoo3 only provides shlibs and no symbol file and the shlibs
claim a dep on libfoo3 (>= 2.10), the system would inject a dep on
libfoo3 >= max(2.10.2, 2.10) == 2.10.2.


What do you think of such a system? Please share your ideas and
comments!



Thanks to Raphaël Hertzog, Josselin Mouette, Julien Cristau, Aurélien
Jarno, and Julien Blache (and the others I forget) for their arguments
and comments on the topic and similar topics.


[1] to help transitions to testing, or to ease installation of packages
from $next-release without rebuilding, to ease upgrades

[2] it could be a new field, but you would want to use it in the
build-deps anyway; using b-deps is backwards-compatible and the most
likely version we would want to start with
--
Loïc Minier


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Sun, Dec 02, 2007 at 05:11:52PM +0100, Loïc Minier wrote:
>
> An example would be package foobar3 build-depending on libfoo3 (>=
> 2.10.2). When built against libfoo3 2.12.0-2 which provides a symbols
> file, the resulting binaries need symbols from libfoo3 (>= 2.8.7) (as
> the symbols file say), the new system injects a dep on libfoo3 >=
> max(2.10.2, 2.8.7) == 2.10.2.
> If libfoo3 only provides shlibs and no symbol file and the shlibs
> claim a dep on libfoo3 (>= 2.10), the system would inject a dep on
> libfoo3 >= max(2.10.2, 2.10) == 2.10.2.

So you would need to build-depend on both libfoo-dev and libfoo3?


Kurt


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Sun, Dec 02, 2007, Kurt Roeckx wrote:
> > An example would be package foobar3 build-depending on libfoo3 (>=
> > 2.10.2). When built against libfoo3 2.12.0-2 which provides a symbols
> > file, the resulting binaries need symbols from libfoo3 (>= 2.8.7) (as
> > the symbols file say), the new system injects a dep on libfoo3 >=
> > max(2.10.2, 2.8.7) == 2.10.2.
> > If libfoo3 only provides shlibs and no symbol file and the shlibs
> > claim a dep on libfoo3 (>= 2.10), the system would inject a dep on
> > libfoo3 >= max(2.10.2, 2.10) == 2.10.2.
>
> So you would need to build-depend on both libfoo-dev and libfoo3?

Sorry, I meant "foobar3 build-depending on libfoo3-*dev*" (which is why
I suggest we want a mapping to relate deps to build-deps); thanks for
pointing out this typo.

--
Loïc Minier


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Sun, Dec 02, 2007 at 05:11:52PM +0100, Loïc Minier wrote:
> What do you think of such a system? Please share your ideas and
> comments!

I think there is a problem using build dependencies for that purpose:
There are dozens of reasons why you want to build-depend on libfoo-dev
>= version that do NOT involve working around bugs in the library at
runtime. There are so many that it would just pollute a lot of
dependencies for a few interesting cases, and would sadly end up going
in the opposite way we are currently heading for with the addition of
dpkg-gensymbols.

Now, the thing is you can pretty much already do some kind of trick to
do what you want, with shlibs.local. The only problem with shlibs.local
is that is forces to use the shlibs in this file and only these, so if
the package's shlibs tell to depend on a newer version, it won't be
taken into account.

Mike


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Sun, 02 Dec 2007, Mike Hommey wrote:
> On Sun, Dec 02, 2007 at 05:11:52PM +0100, Loïc Minier wrote:
> > What do you think of such a system? Please share your ideas and
> > comments!
>
> I think there is a problem using build dependencies for that purpose:
> There are dozens of reasons why you want to build-depend on libfoo-dev
> >= version that do NOT involve working around bugs in the library at
> runtime. There are so many that it would just pollute a lot of
> dependencies for a few interesting cases, and would sadly end up going
> in the opposite way we are currently heading for with the addition of
> dpkg-gensymbols.
>
> Now, the thing is you can pretty much already do some kind of trick to
> do what you want, with shlibs.local. The only problem with shlibs.local
> is that is forces to use the shlibs in this file and only these, so if
> the package's shlibs tell to depend on a newer version, it won't be
> taken into account.

I also explained that you can manually add a dependency in debian/control.
dpkg-gencontrol will simplify the dependency as he want (by keeping the
stricter one).

However this requires a manual intervention of the maintainer while Loïc
hoped that using the build-dep it would require no change by the
maintainer.

Cheers,
--
Raphaël Hertzog

Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Sun, Dec 02, 2007, Mike Hommey wrote:
> I think there is a problem using build dependencies for that purpose:
> There are dozens of reasons why you want to build-depend on libfoo-dev
> >= version that do NOT involve working around bugs in the library at
> runtime. There are so many that it would just pollute a lot of
> dependencies for a few interesting cases, and would sadly end up going
> in the opposite way we are currently heading for with the addition of
> dpkg-gensymbols.

First, I see this solution as a complement to dpkg-gensymbols when
symbols are not at the proper granularity to inject deps.

Yes, in some cases build-deps are bumped for -dev features which are
not connected to the runtime lib -- shlibs are bumped very often for a
very small part of the ABI and symbols have a similar problem for ABI
which can't be expressed by matters of symbols.

Consider a library providing parsing for "sin()" and "cos()" via
"result_t parse(const char *)". It gains support for "tan()" in a new
version. If you want to make sure that anything checking at buildtime
for "tan()" gets it at runtime, you need to inject a very high version
via shlibs or symbols deps all the time, and all libs get a very high
dep -- even if they only use "sin()".

> Now, the thing is you can pretty much already do some kind of trick to
> do what you want, with shlibs.local. The only problem with shlibs.local
> is that is forces to use the shlibs in this file and only these, so if
> the package's shlibs tell to depend on a newer version, it won't be
> taken into account.

Of course, you can argue that the packages should maintain dependencies
on such ABI manually, but in the case of e.g. Gtk+, I don't expect to
require 1000 packages to maintain a proper dep on libgtk2.0-0 with
basically the same version as the build-dep.

Many upstreams already check for a particular version of a build-dep in
their configure.ac or whatever; this usually translates in a build-dep
with a similar version in debian/control. That's why I propose to use
this data for runtime as well as it often represents what upstream
needs at build time and runtime.
And even if the build-dep was bumped for some other reason, I think
it doesn't hurt much.

--
Loïc Minier


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Hi,

Le dimanche 02 décembre 2007 Ã* 17:11 +0100, Loïc Minier a écrit :
> An idea that came up is to use a per-dependent package information
> provided by the maintainer such as the build-deps version [2]. It
> would require a map from deps to build-deps and could typically be
> combined with the existing systems to inject a dependency >=
> max(shlibdeps version, bdeps version).

I like it, and I also think it would be the opportunity to go further in
the direction of dependency automation.

Currently, many upstreams use pkg-config to specify their build
dependencies, and it is, regardless of other issues, a good system that
we should promote towards upstream developers. We could build on
pkg-config and replace it, during the build, by a wrapper that dumps
some information in a build-specific place when the call is successful.

Let’s say that the upstream configure.ac specifies the following:
PKG_CHECK_MODULES (FOO, x11 >= 1.0.2
gtk+2.0 >= 2.8
gconf >= 2.0)

And the source package specifies:
Build-Depends: libx11-dev (>= 2:1.0), libgtk2.0-dev (>= 2.10), libgconf2-dev (>= 2.0)

Every successful call to pkg-config would fill in a file, let’s say
debian/pkgconfig.deps, that would in the end contain:

# pkgconfig_file required_version dev_package shared_package
version
x11 1.0.2 libx11-dev libx11-6 2:1.0.3-7
gtk+2.0 2.8 libgtk2.0-dev libgtk2.0-0 2.12.2-1
gconf 2.0 libgconf2-dev libgconf2-4 2.20.1-1
(The package version is needed because you need to extract epochs.)

In the end, shlibs generation would be able to generate the correct
dependency, based on the highest of the three versions:
1. the version required by upstream;
2. the version required by the build-deps;
3. the version generated by the symbols file.

Plus, in this specific case, it would make the build fail because the
Debian maintainer has forgotten to bump the libx11-dev build-dependency
to 2:1.0.2, which is deadly useful information.

--
.'`.
: :' : We are debian.org. Lower your prices, surrender your code.
`. `' We will add your hardware and software distinctiveness to
`- our own. Resistance is futile.

Le dimanche 02 décembre 2007 à 19:35 +0100, Mike Hommey a écrit :
> I think there is a problem using build dependencies for that purpose:
> There are dozens of reasons why you want to build-depend on libfoo-dev
> >= version that do NOT involve working around bugs in the library at
> runtime. There are so many that it would just pollute a lot of
> dependencies for a few interesting cases, and would sadly end up going
> in the opposite way we are currently heading for with the addition of
> dpkg-gensymbols.

I can think of very few cases where you want to bump the
build-dependency without bumping the runtime dependency as well, and I
cannot think of cases where doing that could actually break something.

--
.'`.
: :' : We are debian.org. Lower your prices, surrender your code.
`. `' We will add your hardware and software distinctiveness to
`- our own. Resistance is futile.

On Mon, Dec 3, 2007 at 17:33:41 +0100, Josselin Mouette wrote:

> Every successful call to pkg-config would fill in a file, let’s say
> debian/pkgconfig.deps, that would in the end contain:
>
> # pkgconfig_file required_version dev_package shared_package
> version
> x11 1.0.2 libx11-dev libx11-6 2:1.0.3-7
> gtk+2.0 2.8 libgtk2.0-dev libgtk2.0-0 2.12.2-1
> gconf 2.0 libgconf2-dev libgconf2-4 2.20.1-1
> (The package version is needed because you need to extract epochs.)
>
> In the end, shlibs generation would be able to generate the correct
> dependency, based on the highest of the three versions:
> 1. the version required by upstream;
> 2. the version required by the build-deps;
> 3. the version generated by the symbols file.
>
> Plus, in this specific case, it would make the build fail because the
> Debian maintainer has forgotten to bump the libx11-dev build-dependency
> to 2:1.0.2, which is deadly useful information.
>
Why would you depend on any particular version of libx11-6? The
build-dep on libx11-dev >= 1.0.2 I can understand, but the runtime
dependency has nothing to do with this.

Cheers,
Julien


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org