Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo Development (http://www.linux-archive.org/gentoo-development/)
-   -   What are || ( ) dependencies? (http://www.linux-archive.org/gentoo-development/466849-what-dependencies.html)

Ciaran McCreesh 12-17-2010 02:25 PM

What are || ( ) dependencies?
 
How should a dependency like || ( a b c ) be interpreted?

Traditionally, it's been described as something like:

* if a matches an installed package, a
* otherwise, if b matches an installed package, b
* otherwise, if c matches an installed package, c
* otherwise, if a is installable, a
* otherwise, if b is installable, b
* otherwise, if c is installable, c
* otherwise, error

Things get messier when you've got || ( a >b-2.1 ) and b-2.0 is
installed and a is not. Should b be upgraded to 2.1, or should a be
selected? What about if you decide upon a early on, and then later on
something hard-depends upon b? What about if you've got || ( a[foo]
b ) and a[-foo] is installed?

As a result of things like this, Portage has had various different sets
of heuristics over time, and Paludis has had a different set. This is
causing problems. Specifically, consider a dependency like the
following, which is present in quite a few ebuilds:

|| (
<x11-libs/libX11-1.3.99.901[xcb]
>=x11-libs/libX11-1.3.99.901
)

Paludis currently interprets this as "I prefer <1.3.99.901, but will
also accept >=1.3.99.901". In particular, if <1.3.99.901[xcb] is
already installed, libX11 won't be upgraded. Some Portage versions also
do this, and others don't.

There's one easy fix, which solves this and every other possible
convoluted case (and some of those can be fairly horrible...): require
ebuild developers to always list 'best' things leftmost. So if you're
doing || ( >=a-2 <a-2 ) then you must put the >= dep first (even if the
>= version is masked -- that's guaranteed to work). If the dependency
is rewritten like this then all the ambiguity goes away:

|| (
>=x11-libs/libX11-1.3.99.901
<x11-libs/libX11-1.3.99.901[xcb]
)

The other option is that we mandate a particular selection algorithm
for || ( ) dependencies. This is a nuisance, for three reasons:

* different Portage versions have done different things

* it prevents the package mangler from doing something clever or
offering additional features

* every algorithm will do the wrong thing for certain combinations of
dependencies if not given any preference information

So would anyone be especially opposed to making "best leftmost" an
explicit requirement, enforced by repoman where possible (at least for
the >= / < case)?

--
Ciaran McCreesh

Sebastian Luther 12-17-2010 03:27 PM

What are || ( ) dependencies?
 
Am 17.12.2010 16:25, schrieb Ciaran McCreesh:
> So would anyone be especially opposed to making "best leftmost" an
> explicit requirement, enforced by repoman where possible (at least for
> the >= / < case)?
>

Why can't the PM handle >= / < cases itself?



Sebastian

Ciaran McCreesh 12-17-2010 03:37 PM

What are || ( ) dependencies?
 
On Fri, 17 Dec 2010 17:27:05 +0100
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> Am 17.12.2010 16:25, schrieb Ciaran McCreesh:
> > So would anyone be especially opposed to making "best leftmost" an
> > explicit requirement, enforced by repoman where possible (at least
> > for the >= / < case)?
>
> Why can't the PM handle >= / < cases itself?

Because things are almost never as simple as 'just' >= / <. You can add
in clever trickery to deal with very specific cases, but the second
someone throws things off by adding in a use dependency or a third
package, things get weird.

Consider a variation on the original case: || ( <a-2 >=a-2[x] ) where
the user has specified -x for a. What should happen then?

What about || ( <a-2[x] b >=a-2 ) ? Should that be rewritten in the same
way?

What about || ( <a-2[x] ( >=a-2 b ) ) ? Should the package mangler be
clever enough to figure that one out too? What if b isn't already
installed there?

Which is really the problem: clever heuristics get extremely
complicated very quickly, and they're never enough.

--
Ciaran McCreesh

"Paweł Hajdan, Jr." 12-17-2010 03:49 PM

What are || ( ) dependencies?
 
On 12/17/10 4:25 PM, Ciaran McCreesh wrote:
> As a result of things like this, Portage has had various different sets
> of heuristics over time, and Paludis has had a different set.

Generally it seems fine to have different heuristics (I'll comment on
the specific problem below).

> Paludis currently interprets this as "I prefer <1.3.99.901, but will
> also accept >=1.3.99.901". In particular, if <1.3.99.901[xcb] is
> already installed, libX11 won't be upgraded. Some Portage versions also
> do this, and others don't.

I don't understand why we can't upgrade libX11 in that case. Shouldn't
emerge -uDNa world (or its Paludis equivalent) "think" like this:

Okay, I have libX11 installed here, and a more recent version is
available. The more recent version satisfies this || () dependency, so
just update it.

> There's one easy fix, which solves this and every other possible
> convoluted case (and some of those can be fairly horrible...): require
> ebuild developers to always list 'best' things leftmost.

Sounds reasonable.

> The other option is that we mandate a particular selection algorithm
> for || ( ) dependencies.

Doesn't that somehow contradict the idea that || () lists equivalent
dependencies? Maybe we should fix the heuristics.

In this specific case, it seems reasonable to still upgrade libX11, right?

> So would anyone be especially opposed to making "best leftmost" an
> explicit requirement, enforced by repoman where possible (at least for
> the >= / < case)?

I don't think that >= / < case is enforceable by repoman (i.e. that we
always prefer the more recent version of a package).

However, saying that the preferred dependency should be listed first
sounds reasonable.

Paweł

Sebastian Luther 12-17-2010 03:56 PM

What are || ( ) dependencies?
 
Am 17.12.2010 17:37, schrieb Ciaran McCreesh:
> On Fri, 17 Dec 2010 17:27:05 +0100
> Sebastian Luther <SebastianLuther@gmx.de> wrote:
>> Am 17.12.2010 16:25, schrieb Ciaran McCreesh:
>>> So would anyone be especially opposed to making "best leftmost" an
>>> explicit requirement, enforced by repoman where possible (at least
>>> for the >= / < case)?
>>
>> Why can't the PM handle >= / < cases itself?
>
> Because things are almost never as simple as 'just' >= / <. You can add
> in clever trickery to deal with very specific cases, but the second
> someone throws things off by adding in a use dependency or a third
> package, things get weird.

I thought we were talking about the simplest case here, that is a list
of atoms for the same cat/pkg.

>
> Consider a variation on the original case: || ( <a-2 >=a-2[x] ) where
> the user has specified -x for a. What should happen then?
>
> What about || ( <a-2[x] b >=a-2 ) ? Should that be rewritten in the same
> way?
>
> What about || ( <a-2[x] ( >=a-2 b ) ) ? Should the package mangler be
> clever enough to figure that one out too? What if b isn't already
> installed there?

What would repoman enforce here?

>
> Which is really the problem: clever heuristics get extremely
> complicated very quickly, and they're never enough.
>

Agreed.



Sebastian

Ciaran McCreesh 12-17-2010 04:01 PM

What are || ( ) dependencies?
 
On Fri, 17 Dec 2010 17:49:22 +0100
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:
> > Paludis currently interprets this as "I prefer <1.3.99.901, but will
> > also accept >=1.3.99.901". In particular, if <1.3.99.901[xcb] is
> > already installed, libX11 won't be upgraded. Some Portage versions
> > also do this, and others don't.
>
> I don't understand why we can't upgrade libX11 in that case. Shouldn't
> emerge -uDNa world (or its Paludis equivalent) "think" like this:
>
> Okay, I have libX11 installed here, and a more recent version is
> available. The more recent version satisfies this || () dependency, so
> just update it.

That's not really how the Paludis resolver thinks. Basically, when it
encounters a || ( ) dependency, it selects one of the children based
upon a scoring algorithm. At the time it encounters the || ( )
dependency, it doesn't know for sure that it's allowed to upgrade
libX11, since a later ebuild might hard-dep upon the lower version.

Now, I *could* make it treat the very specific case of || ( <a-1[x]
>=a-1 ) as being a single dep spec like a[either <1[x] or >=1], but
that doesn't really help as soon as someone does this:

|| ( <a-1[x] ( >=a-1 b ) )

which looks a lot like something someone would do.

> In this specific case, it seems reasonable to still upgrade libX11,
> right?

In this very specific case, rewriting || ( a[stuff] a[otherstuff] ) to
be like a[stuff || otherstuff] would solve the problem. But even
slightly altering the dependencies would break this. So the question is
to what degree a package mangler is required to be clever.

> > So would anyone be especially opposed to making "best leftmost" an
> > explicit requirement, enforced by repoman where possible (at least
> > for the >= / < case)?
>
> I don't think that >= / < case is enforceable by repoman (i.e. that we
> always prefer the more recent version of a package).

It's at least detectable...

--
Ciaran McCreesh

Ciaran McCreesh 12-17-2010 04:09 PM

What are || ( ) dependencies?
 
On Fri, 17 Dec 2010 17:56:21 +0100
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> >> Why can't the PM handle >= / < cases itself?
> >
> > Because things are almost never as simple as 'just' >= / <. You can
> > add in clever trickery to deal with very specific cases, but the
> > second someone throws things off by adding in a use dependency or a
> > third package, things get weird.
>
> I thought we were talking about the simplest case here, that is a list
> of atoms for the same cat/pkg.

Here's the problem: if the package mangler gets this right (which
afaik both Portage and Paludis do):

|| ( <a-1 >=a-1 )

you might naively expect it to get these right too:

|| ( <a-1 ( >=a-1 b ) )
|| ( <a-1 >a-1[x] ) # where a[-x] is the current user configuration

which it probably won't, and even if it does, it will get other minor
variations on these themes wrong.

> > What about || ( <a-2[x] ( >=a-2 b ) ) ? Should the package mangler
> > be clever enough to figure that one out too? What if b isn't already
> > installed there?
>
> What would repoman enforce here?

For that case, it probably couldn't detect it. But getting repoman to
yell at people for the simple cases at least would probably help people
to learn that they should always go best-leftest.

Part of the problem here is that developers don't seem to know about
the whole leftmost thing, and the heuristics we've all put in mean they
can quite often get away with not knowing it.

It also doesn't help that it's not really documented anywhere. It's not
in the devmanual, it's not in PMS (and it's hard to put it there,
assuming we're not requiring a particular selection algorithm), and the
stuff in ebuild(5) for || dependencies is just plain wrong.

--
Ciaran McCreesh

Patrick Lauer 12-17-2010 04:27 PM

What are || ( ) dependencies?
 
On 12/17/10 18:09, Ciaran McCreesh wrote:
> On Fri, 17 Dec 2010 17:56:21 +0100
> Sebastian Luther <SebastianLuther@gmx.de> wrote:
>>>> Why can't the PM handle >= / < cases itself?
>>>
>>> Because things are almost never as simple as 'just' >= / <. You can
>>> add in clever trickery to deal with very specific cases, but the
>>> second someone throws things off by adding in a use dependency or a
>>> third package, things get weird.
>>
>> I thought we were talking about the simplest case here, that is a list
>> of atoms for the same cat/pkg.
>
> Here's the problem: if the package mangler gets this right (which
> afaik both Portage and Paludis do):
>
> || ( <a-1 >=a-1 )
>
> you might naively expect it to get these right too:
>
> || ( <a-1 ( >=a-1 b ) )
> || ( <a-1 >a-1[x] ) # where a[-x] is the current user configuration
>
> which it probably won't, and even if it does, it will get other minor
> variations on these themes wrong.

There are multiple valid solutions. The default heuristic for "right"
seems to be "highest version", but other solutions fulfill the
dependency specification too.

> Part of the problem here is that developers don't seem to know about
> the whole leftmost thing, and the heuristics we've all put in mean they
> can quite often get away with not knowing it.

I'm not sure if I want that enforced, but a suggestion that
left-to-right order may be used by the package manager sounds like a
reasonable idea.

> It also doesn't help that it's not really documented anywhere. It's not
> in the devmanual, it's not in PMS (and it's hard to put it there,
> assuming we're not requiring a particular selection algorithm), and the
> stuff in ebuild(5) for || dependencies is just plain wrong.
>
Patches Welcome? ;)

Donnie Berkholz 12-18-2010 01:13 AM

What are || ( ) dependencies?
 
On 15:25 Fri 17 Dec , Ciaran McCreesh wrote:
> How should a dependency like || ( a b c ) be interpreted?
>
> Traditionally, it's been described as something like:
>
> * if a matches an installed package, a
> * otherwise, if b matches an installed package, b
> * otherwise, if c matches an installed package, c
> * otherwise, if a is installable, a
> * otherwise, if b is installable, b
> * otherwise, if c is installable, c
> * otherwise, error
>
> Things get messier when you've got || ( a >b-2.1 ) and b-2.0 is
> installed and a is not. Should b be upgraded to 2.1, or should a be
> selected?

It depends ... see later.

> What about if you decide upon a early on, and then later on something
> hard-depends upon b?

Then you're collapsing the graph too early. =)
(speaking as an utter novice)

> What about if you've got || ( a[foo] b ) and a[-foo] is installed?

See later again..

> As a result of things like this, Portage has had various different sets
> of heuristics over time, and Paludis has had a different set. This is
> causing problems. Specifically, consider a dependency like the
> following, which is present in quite a few ebuilds:
>
> || (
> <x11-libs/libX11-1.3.99.901[xcb]
> >=x11-libs/libX11-1.3.99.901
> )
>
> Paludis currently interprets this as "I prefer <1.3.99.901, but will
> also accept >=1.3.99.901". In particular, if <1.3.99.901[xcb] is
> already installed, libX11 won't be upgraded. Some Portage versions also
> do this, and others don't.

Why is this a problem that needs to be resolved at the specification
level rather than a difference between implementations? If a package
manager is making strange choices,

> There's one easy fix, which solves this and every other possible
> convoluted case (and some of those can be fairly horrible...): require
> ebuild developers to always list 'best' things leftmost. So if you're
> doing || ( >=a-2 <a-2 ) then you must put the >= dep first (even if the
> >= version is masked -- that's guaranteed to work). If the dependency
> is rewritten like this then all the ambiguity goes away:

I'd thought people already knew that this was typical behavior of an ||
group (as per the simple example in ebuild(5)), but you've said
differently later in this thread. I certainly wouldn't mind documenting
that left is best in cases where none are installed, since this has been
expected behavior to those of us who do know.

However, that doesn't resolve the case where a package is installed but
is either too old or has a mismatched USE flag to the dep. It's not
clear to me how this proposal would deal with the system-dependent
components.

I think whether a deep upgrade is requested (or otherwise directly
targeting the dependency for possible upgrading) should impact the
choice between >= and <. If no upgrade is desired, allow the < dep to be
fulfilled; otherwise attempt an upgrade to the newest version and see if
it matches.

For the mismatched case, I'd attempt to flip the USE unless that would
produce a conflict with another package; in that case, pick the leftmost
other fulfiller.

> || (
> >=x11-libs/libX11-1.3.99.901
> <x11-libs/libX11-1.3.99.901[xcb]
> )
>
> The other option is that we mandate a particular selection algorithm
> for || ( ) dependencies. This is a nuisance, for three reasons:
>
> * different Portage versions have done different things
>
> * it prevents the package mangler from doing something clever or
> offering additional features
>
> * every algorithm will do the wrong thing for certain combinations of
> dependencies if not given any preference information

I think this last point is one of the strongest ones. Humans may need to
decide what's best for any specific package's dependencies.

> So would anyone be especially opposed to making "best leftmost" an
> explicit requirement, enforced by repoman where possible (at least for
> the >= / < case)?

Not in the entirely ambiguous case, but things are trickier when one of
the packages is installed, even in nonmatching version/USE.

--
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

Ciaran McCreesh 12-18-2010 02:21 PM

What are || ( ) dependencies?
 
On Fri, 17 Dec 2010 20:13:55 -0600
Donnie Berkholz <dberkholz@gentoo.org> wrote:
> > What about if you decide upon a early on, and then later on
> > something hard-depends upon b?
>
> Then you're collapsing the graph too early. =)
> (speaking as an utter novice)

Yeah, but unfortunately, there's no way to figure out when too early
is. What if it's one of a's dependencies that hard-depends upon b?
Until you've decided upon something, you don't know what dependencies
are going to be pulled in, so you're left having to make possibly
incorrect decisions and then try to undo them later on if possible.

> Why is this a problem that needs to be resolved at the specification
> level rather than a difference between implementations? If a package
> manager is making strange choices,

The problem's how you define strange choices. If dependencies aren't
listed best-leftmost, every package manager makes strange choices for
some combinations. Either this can be fixed by getting developers to
always write things best-leftmost, or it can be fixed by mandating
specific behaviour for all package managers for || ( ) deps. I'd much
rather we did the former.

> I'd thought people already knew that this was typical behavior of an
> || group (as per the simple example in ebuild(5)), but you've said
> differently later in this thread. I certainly wouldn't mind
> documenting that left is best in cases where none are installed,
> since this has been expected behavior to those of us who do know.

Well, we're running across a fair number of cases along the lines of
the libX11 one.... https://bugs.gentoo.org/show_bug.cgi?id=348518 is
what prompted the email -- it turns out vlc is by no means the only
package doing this, though, which gives me two options for Paludis: add
in a heuristic that gets that very specific case right (and update PMS
requiring package manglers to do the same), or get people to list their
deps the other way around.

--
Ciaran McCreesh


All times are GMT. The time now is 07:52 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.