FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 09-19-2012, 04:22 AM
Ben de Groot
 
Default example conversion of gentoo-x86 current deps to unified dependencies

On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:
> On Sun, Sep 16, 2012 at 03:39:22PM +0800, Ben de Groot wrote:
>> Thanks. I have given it a quick overview for the qt herd. I still
>> don't see what using DEPENDENCIES adds to what we do now with separate
>> *DEPEND variables. I see no convincing reason to change what we do.
>
> Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194
> eclasses; grand total of 31554 sources of metadata content (I say
> metadata since vapier has eblits in use which are just phase
> functions- I'm not scanning those intentionally).
>
> Doing some simple scans of the tree, here's some stats; note these
> stats are duplicated in the glep (they're nice selling points, thus
> might as well):
>
> 1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
> I'm aware of have been for literally, what it sounds like- depends
> that are both DEPEND and RDEPEND.
>
> 2) scanning for assignments of RDEPEND=.*|${?DEPEND}? gets a hit on
> 5343 unique sources. Searching for the inverse gets a hit on 10008
> unique sources. Meaning that right there, ~48.6% of the tree is
> duplicating deps between the two forms. This puts us to 16083 unique
> sources in the tree that would benefit in some form (~51%).
>
> 3) What's interesting about that 51% is the eapi groupings; in EAPI4,
> the autosetting of RDEPEND="${RDEPEND:-${DEPEND}}" was discontinued.
> Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.
>
> 4) Again, keep in mind that the grep's in use above are single line
> matches- I'm definitely missing some ebuilds, and for complex
> dependencies that are appended/set by the eclass, likely missing a lot
> of that too.
>
> So... basically, people are already doing this manually with their own
> intermediate vars.

And this works fine, so it doesn't warrant a cosmetic change.

> Just a rough but mildly entertaining stat there's basically 8.38MB
> worth of normalized, literal fricking dependency; using a crappy algo
> (rather than a human doing the deps who can do it better), 37.4%
> (3.1MB) of that is removed via going to dependencies. It goes
> without saying that this would be a helluva lot less torturous on a
> proper PM implementation that parses the tree once, and renders- let
> alone repoman being able to avoid repeat scans of things it already
> has examined.
>
> Mind you, portage doesn't do that, but this would be good incentive to
> do a proper tree.

PM internals are not really relevant to us developers. Just make it work... ;-)

The current *DEPEND system works. So again, I don't see a convincing
reason for change.

>> As I've said before on IRC, we need a good costs/benefits overview.
>> Right now I only see costs (migrating ebuilds and eclasses) and no
>> benefits.
>
> Offhand... I wouldn't be pushing for this if I didn't think this would
> be a boon over all- both for devs, and PMs.
>
> I think the actual 'cost' probably got lost in the noise of the
> various flames; what I'm proposing is basically zero cost for devs, it
> shoves the work to the PM, leaving the option for devs to do a more
> fine grained form if they can.

As soon as you expose this to devs (ebuild/eclass developers and
maintainers), you can no longer talk about zero cost.

We either have the enormous cost of migrating the whole tree to the
new system — which I understand you are not advocating because of the
cost; or we have the (lower, but not zero) cost of maintaining two
systems in the tree, which developers will have to know and understand
in order to be able to deal with them correctly. The latter will also
make eclasses needlessly more complex.

So in my opinion it is folly either way, because the costs are high,
and the benefits only cosmetic.

> I'm starting a seperate thread w/ a glep for this; I think you should
> take a look at the exact details of the specification including how
> DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short
> version: if existent, they're automatically folded into DEPENDENCIES
> in my proposal); the end result is basically zero pain transition,
> while enabling us to start getting gains.

Which gains? The glep only talks about cosmetics...

--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
 
Old 09-19-2012, 10:59 AM
Duncan
 
Default example conversion of gentoo-x86 current deps to unified dependencies

Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted:

> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:

>> So... basically, people are already doing this manually with their own
>> intermediate vars.
>
> And this works fine, so it doesn't warrant a cosmetic change.

@ferringb:

yngwin has a point that I've not seen addressed.

What /is/ wrong with the whole CDEPEND intermediate var idea? It seems
to work and /I/ don't know of any problems with it (and it would appear,
neither does yngwin), yet you talk about it as if there's something wrong
with it.

And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the
same problems (or lack thereof)?

FWIW I personally like the whole single-var idea, and CERTAINLY
appreciate the various statistical cache savings, etc. If we were
starting from scratch now, I'd definitely favor the single var approach.
But the combined developer mental cost of having to learn the new method
and then maintain a working understanding of both over some longer period
is nothing to sneeze at, and I'm not entirely convinced that it's worth
that cost, even assuming a doubling of the number of dependency types
with a lot of commonality between them, and the added benefit a single
deps var would have in that case.

And the case for a single deps var isn't being helped by the implication
that there's something wrong with both the intermediate var and copying
var methods, without ever saying what that "wrong" might be, in the face
of the experience of many that those existing methods "just work". So if
there's something wrong with them, let's get it out there where people
can see it. And if there isn't, please eliminate the noise of that
implication from the argument.

Thanks. =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 09-19-2012, 01:09 PM
Michael Orlitzky
 
Default example conversion of gentoo-x86 current deps to unified dependencies

On 09/19/2012 06:59 AM, Duncan wrote:
> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted:
>
>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:
>
>>> So... basically, people are already doing this manually with their own
>>> intermediate vars.
>>
>> And this works fine, so it doesn't warrant a cosmetic change.
>
> @ferringb:
>
> yngwin has a point that I've not seen addressed.
>
> What /is/ wrong with the whole CDEPEND intermediate var idea? It seems
> to work and /I/ don't know of any problems with it (and it would appear,
> neither does yngwin), yet you talk about it as if there's something wrong
> with it.
>
> And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the
> same problems (or lack thereof)?

The problem appears as we introduce more DEPEND variables (which is what
prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, CDEPEND, and
DDEPEND, and there's only some (i.e. not total) sharing going on then
the COMMON_DEPEND pattern starts to fall apart. You potentially need,

AB_DEPEND
AC_DEPEND
AD_DEPEND
BC_DEPEND
BD_DEPEND
CD_DEPEND
ABC_DEPEND
ABD_DEPEND
ACD_DEPEND
BCD_DEPEND
ABCD_DEPEND (COMMON_DEPEND)

This obviously gets worse as more DEPEND vars are introduced.
 
Old 09-19-2012, 01:16 PM
Ian Stakenvicius
 
Default example conversion of gentoo-x86 current deps to unified dependencies

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> On 09/19/2012 06:59 AM, Duncan wrote:
>> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
>> excerpted:
>>
>>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com>
>>> wrote:
>>
>>>> So... basically, people are already doing this manually with
>>>> their own intermediate vars.
>>>
>>> And this works fine, so it doesn't warrant a cosmetic change.
>>
>> @ferringb:
>>
>> yngwin has a point that I've not seen addressed.
>>
>> What /is/ wrong with the whole CDEPEND intermediate var idea? It
>> seems to work and /I/ don't know of any problems with it (and it
>> would appear, neither does yngwin), yet you talk about it as if
>> there's something wrong with it.
>>
>> And while we're at it, do DEPEND="$RDEPEND ..." style solutions
>> have the same problems (or lack thereof)?
>
> The problem appears as we introduce more DEPEND variables (which is
> what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> sharing going on then the COMMON_DEPEND pattern starts to fall
> apart. You potentially need,
>
> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> (COMMON_DEPEND)
>
> This obviously gets worse as more DEPEND vars are introduced.
>

Well not really, no -- the additional *DEPENDs that are being proposed
(or at least mentioned) for new EAPI will either remove atoms from
COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
COMMON_DEPEND or other intermediate variable won't really be necessary
for them.

Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBZxZIACgkQ2ugaI38ACPDp4wD/atjvaOsi/ntDMB1Dj7lSAVmW
45qKz6+OO+H/+6eFeVIA/Rz0s7FiG6d2frboHXpYrDBzM1FZcU85AqZti34tR8+h
=E78Z
-----END PGP SIGNATURE-----
 
Old 09-30-2012, 10:15 PM
Brian Harring
 
Default example conversion of gentoo-x86 current deps to unified dependencies

Pardon the belated response; responding to emails that are quick where
possible, but lagging on -dev. Missed this one however...

On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> > On 09/19/2012 06:59 AM, Duncan wrote:
> >> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
> >> excerpted:
> >>
> >>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com>
> >>> wrote:
> >>
> >>>> So... basically, people are already doing this manually with
> >>>> their own intermediate vars.
> >>>
> >>> And this works fine, so it doesn't warrant a cosmetic change.
> >>
> >> @ferringb:
> >>
> >> yngwin has a point that I've not seen addressed.
> >>
> >> What /is/ wrong with the whole CDEPEND intermediate var idea? It
> >> seems to work and /I/ don't know of any problems with it (and it
> >> would appear, neither does yngwin), yet you talk about it as if
> >> there's something wrong with it.
> >>
> >> And while we're at it, do DEPEND="$RDEPEND ..." style solutions
> >> have the same problems (or lack thereof)?
> >
> > The problem appears as we introduce more DEPEND variables (which is
> > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> > CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> > sharing going on then the COMMON_DEPEND pattern starts to fall
> > apart. You potentially need,
> >
> > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
> > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> > (COMMON_DEPEND)
> >
> > This obviously gets worse as more DEPEND vars are introduced.
> >
>
> Well not really, no -- the additional *DEPENDs that are being proposed
> (or at least mentioned) for new EAPI will either remove atoms from
> COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
> COMMON_DEPEND or other intermediate variable won't really be necessary
> for them.

It depends on the dep type actually, and how we're viewing those deps-
if they must be complete or not.

Consider test depends for example. Either we can specify it as "in
addition to depend"- which sucks, because it's entirely possible for a
DEPEND target to not be required for TDEPEND.

While that's a corner case, it's actually easy to address; just
require TDEPEND to be comprehensive for the test phase.

Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part
of RDEPEND.

Moving on to a real world example... PDEPEND. Currently a full graph
resolution requires pulling RDEPEND and PDEPEND, collapsing them, and
ensuring they're fullfilled for anything that is already installed
(ie, PDEPEND is required after the transaction is completed).

In our current separated var setup, this as said, this requires the PM
to track/handle it separately. However, were PDEPEND to be
complete/full- that would mean the PM could just render for depost
and know "this is what is necessary outside of the transactional block
of building/installing it".

So RDEPEND and PDEPEND actually share a *shitton*- they're the same
contents in 97% of the tree. Meaning dep:run,post? for 97% of the
tree, w/ 3% needing to have an addition depost? section.

Offhand, comprehensive deps make things easier for devs- it gives them
the ability to be crystal-freaking-clear as to what's needed at each
stage; via them doing so, it means the resolver has a greater space to
dig itself out of fucked up situations if necessary.

Additionally, it actually makes life easier for PM authors. If we had
TDEPEND (test), this is how things would go; note this is written for
just an install, not a replacement (replace is similar, just noisier
deflecting from my point):

1) ensure DEPEND is satisfied
2) run phases setup -> compile
3) ensure TDEPEND is satisfied; Implicitly, DEPEND in the process (if
there is a cycle induced by TDEPEND + DEPEND being required, we have
no way out of it).
4) run install phase
5) ensure RDEPEND is satisfied. No longer care about TDEPEND/DEPEND.
6) run pkg_preinst, merge, pkg_postinst.
7) a transaction is opened up, that must be completed before the PM
exits; before that exit, RDEPEND must still be satisfied, as must
PDEPEND.

With what I'm suggesting.
1) ensure dep:build is satisfied
2) run phases setup -> compile
3) Ensure dep:test is satisfied. If cycle breaking is necessary,
anything in dep:build but not dep:test can be pulled.
4) install phase
5) ensure dep:run is satisfied. Again, if cycle breaking is
necessary, punt whatever isn't in dep:run as necessary to break that
cycle.
6) pkg_preinst, merge, pkg_postinst.
7) transactional block; ensure depost is satisfied before exiting
said transaction.

That's for install; for the PM considering a pre-installed pkg, it
converts from;
1) ensure RDEPEND and PDEPEND are satisfied

to
1) ensure depost is satisfied.

Tbh, I'm not sure I'm communicating these gains as well as I could be;
at first glance, I'm pretty sure people will say "what's the
difference?" because the gains/difference there are subtle.

From a PM level, I can tell you this shift to complete deps per stage
*does* make things easier via simplifying the rules at each step
(namely, render for this form, that's what's required for this point
on), while giving full control to devs to be dead on accurate in their
deps specified (which the PM can then exploit for better cycle
breaking).

The point I'm trying to make here is that each dep phase
should be authorative; in doing so, you start getting
a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND
isn't completely, but mostly a subset of RDEPEND as RDEPEND is a
likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).

So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
the ebuild. Or you could just use a syntax form that allows you to
directly inline that up front, rather than having to muck around w/
intermediate vars.


> Besides, this isn't actually a -problem- as there's nothing which
> really requires one to use such helpers; ebuild writers just, well,
> can.

Getting to this point; yes, people could hack around it manually.
Pretty sure that hasn't been in doubt.

The question is what do we *gain* from making them do so. No one here
has stated what we gain from COMMON_DEPENDS in comparison to
DEPENDENCIES (where they can just inline it).

And as I'm trying to lay out, where dependencies *should* be going
towards, this issue will get worse if we're doing it out of band from
DEPENDENCIES.

~harring
 
Old 10-01-2012, 12:23 AM
Duncan
 
Default example conversion of gentoo-x86 current deps to unified dependencies

Brian Harring posted on Sun, 30 Sep 2012 15:15:18 -0700 as excerpted:

> The point I'm trying to make here is that each dep phase should be
> authorative; in doing so, you start getting a lot of potential subsets
> (DEPEND is a subset of TDEPEND, TDEPEND isn't completely, but mostly a
> subset of RDEPEND as RDEPEND is a likely a superset of DEPEND; PDEPEND
> is a superset of RDEPEND).
>
> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in the
> ebuild. Or you could just use a syntax form that allows you to directly
> inline that up front, rather than having to muck around w/ intermediate
> vars.

Thanks /very/ much! You said you weren't sure you were being clear, but
this is the first time I've /really/ understood what must surely be the
root, at any reasonable level at all.

Let me see if I've got it right:

Yes, in some ways all we're dealing with here is "optics", but the
_problem_ is that with the proposed proliferation in detailed depend-
types, what is now a simple CDEPEND and thus conceptually easy to handle,
breaks into 10/20/50/whatever-large-number different shards, and what's
conceptually easy to handle /now/ becomes many many times more difficult
to handle, both conceptually for package maintainers and practically for
iterative resolution in the PMs, due to the interplay of all the
resulting *CDEPENDs.

The proposed solution to that explosion in conceptual complexity not only
changes the "optics", but by making most of those detail-depend-types
absolute/authoritative, allows both package managers (the programs,
machine) and package maintainers (the humans) to consider each depend-
type separately, thus decreasing both conceptual complexity to a once
again manageable level for package maintainers (humans), and practical
complexity for package managers (machine), increasing efficiency,
reducing resolution time and probably eventually memory/installed-db/
cache size as well.


Of course now I better understand Ciaran's argument for labels as well,
since it would extend the absolute/authoritative principle even further,
into the actual deps specification method in ebuilds/eclasses, thereby
reducing conceptual context load even further via more explicitly
absolute deps at the local level.

But like you, in practice I don't see that going anywhere in gentoo, in
the near/short-intermediate future, primarily due to political realities,
but practically, also due to the conceptual leap it'd require from devs
(as Ciaran himself points out in response to your statistical analysis of
exherbo's repo, former gentoo devs simply don't tend to take advantage of
this aspect of labels in exherbo either; the conceptual leap is in
practice simply too much). Thus, while academically, his label approach
is arguably better in terms of efficiency of absolutes, in practice,
there's little or no difference between how it's used, and how your
filtering approach will be used. Further, given the conceptual distance
between labels and gentoo's current approach, with filters falling in
between and political reality, the pragmatic filters approach at least
has /some/ chance of passing the dev-debate stage and being approved,
implemented and actually available for use in something like a reasonable
timeframe (say EAPI-6, in a year's time, a bit more for actual in-tree
use, given the historic EAPI-a-year processing). But exherbo style
labels support altho academically better, given political reality, in all
likelihood would take at least 2-3 years to pass and be usable in-tree.
And even then, its practical use as demonstrated in exherbo wouldn't take
advantage of the differences for another couple years after that, at
least.

Given that, having use of the useful pragmatic approach in a year's time
or so seems best, as opposed to an arguably (as you've pointed out, no
practical demonstration of it in exherbo yet, at least that you've been
able to find) more academically ideal approach in three, without any real
benefit over the pragmatic for five or more.

>> Besides, this isn't actually a -problem- as there's nothing which
>> really requires one to use such helpers; ebuild writers just, well,
>> can.
>
> Getting to this point; yes, people could hack around it manually. Pretty
> sure that hasn't been in doubt.

That's clearer now. Yes, people could continue to hack around it via
CDEPENDS, etc. But the number of common vars (or alternative
RDEPEND="$DEPEND ..." assignments) and the resulting conceptual load on
the human maintainer is set to increase exponentially as the number of
depend-types increases linearly. At some point it's just no longer
practically maintainable, and the whole process breaks down.

What the single dependencies variable aims to do in both the filtering
and labels forms is prevent that ultimate conceptual overload and
resulting process breakdown by allowing direct compound assignments, thus
eliminating the intermediate assignments and their exponential
proliferation.

Thanks again, Brian. Much clearer now, indeed, at least for me, and
presumably for others who previously had the same problem I was having.

=:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 10-02-2012, 05:47 PM
Ian Stakenvicius
 
Default example conversion of gentoo-x86 current deps to unified dependencies

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 30/09/12 06:15 PM, Brian Harring wrote:
>
> Pardon the belated response; responding to emails that are quick
> where possible, but lagging on -dev. Missed this one however...
>

No worries, there's a lot going on..



>>>> yngwin has a point that I've not seen addressed.
>>>>
>>>> What /is/ wrong with the whole CDEPEND intermediate var idea?
>>>>
>>>
>>> The problem appears as we introduce more DEPEND variables
>>> (which is what prompted the proposal, IIRC). If we have
>>> ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
>>> (i.e. not total) sharing going on then the COMMON_DEPEND
>>> pattern starts to fall apart. You potentially need,
>>>
>>> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
>>> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
>>> (COMMON_DEPEND)
>>>
>>> This obviously gets worse as more DEPEND vars are introduced.
>>>
>>
>> Well not really, no -- the additional *DEPENDs that are being
>> proposed (or at least mentioned) for new EAPI will either remove
>> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
>> tersely that a COMMON_DEPEND or other intermediate variable won't
>> really be necessary for them.
>
> It depends on the dep type actually, and how we're viewing those
> deps- if they must be complete or not. [ .. Snip! .. ]
>
> The point I'm trying to make here is that each dep phase should be
> authorative; in doing so, you start getting a lot of potential
> subsets (DEPEND is a subset of TDEPEND, TDEPEND isn't completely,
> but mostly a subset of RDEPEND as RDEPEND is a likely a superset of
> DEPEND; PDEPEND is a superset of RDEPEND).
>
> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
> the ebuild. Or you could just use a syntax form that allows you
> to directly inline that up front, rather than having to muck around
> w/ intermediate vars.
>

... I think what you've just described here might be where the primary
difference in thinking is for most of us, between moving to
DEPENDENCIES and keeping the current *DEPENDs -- to me, from an ebuild
writer's perspective, the *DEPENDS -shouldn't- be authoritative. IE,
instead of thinking of PDEPEND as a superset of RDEPEND I consider
they are two separate sets, which should not intersect, and are
unioned together to form the full set of runtime dependencies. IE:
FULL_RUNTIME_DEPEND="$RDEPEND $PDEPEND" somewhere inside portage, if
portage actually needed it this way.

And I see this as how many of the other proposed new *DEPENDs would
work too, ie, they are a refined subset of the bigger sets and should
not intersect with the 'parent' *DEPEND that was used instead on older
EAPIs.

So if this were to change, it might make sense (as Duncan i think
pointed out in his response to this message), to a debate on whether
or not ebuilds must specify an authoritative list for each dep phase.
(I haven't read through PMS but I'm going to assume that it doesn't
specify this anywhere yet--and if it does, i'm sure Ciaran or someone
will quote it in response
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBrKKsACgkQ2ugaI38ACPBA9wD/a+XVf2g9s6dcpW1513aXQpYV
tK94QdLkax0Hs6tKFqcA/0+v6oFON2Bd3hedv9DHg7N42NNvtTKK6sOw18OpL0aO
=frmC
-----END PGP SIGNATURE-----
 
Old 10-03-2012, 04:00 AM
Ben de Groot
 
Default example conversion of gentoo-x86 current deps to unified dependencies

> On 30/09/12 06:15 PM, Brian Harring wrote:
>
>>>>> yngwin has a point that I've not seen addressed.
>>>>>
>>>>> What /is/ wrong with the whole CDEPEND intermediate var idea?
>>>>>
>>>>
>>>> The problem appears as we introduce more DEPEND variables
>>>> (which is what prompted the proposal, IIRC). If we have
>>>> ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
>>>> (i.e. not total) sharing going on then the COMMON_DEPEND
>>>> pattern starts to fall apart. You potentially need,
>>>>
>>>> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
>>>> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
>>>> (COMMON_DEPEND)
>>>>
>>>> This obviously gets worse as more DEPEND vars are introduced.
>>>>
>>>
>>> Well not really, no -- the additional *DEPENDs that are being
>>> proposed (or at least mentioned) for new EAPI will either remove
>>> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
>>> tersely that a COMMON_DEPEND or other intermediate variable won't
>>> really be necessary for them.

Another thing I wanted to point out is that those "potential" extra
variables are not needed in practice. We already have 98% of the tree
(if I got the previously mentioned stats right) that does fine with
just one or two ({R,}DEPEND). The majority of that other 2% needs just
one more variable. There may be corner cases where more vars would be
needed, but those will never be more than a few ebuilds.

It's just not worth it to completely change the way we do things (or
use two systems in parallel) just for a few ebuilds that would
significantly benefit.

If we were a new distro and designing our ebuild format from scratch,
then yes, I would say your proposal has merit. But we aren't. We have
hundreds of people and tens of thousands of ebuilds using *DEPEND just
fine. There are no big problems, only corner-cases. We're not talking
about incremental improvements either (such as was the case e.g. with
use deps).

Let's just keep things simple, and refrain from "fixing" what isn't broken.
--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
 
Old 10-07-2012, 02:09 PM
"Steven J. Long"
 
Default example conversion of gentoo-x86 current deps to unified dependencies

On Sun, Sep 30, 2012 at 03:15:18PM -0700, Brian Harring wrote:
> On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote:
> > On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> > > The problem appears as we introduce more DEPEND variables (which is
> > > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> > > CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> > > sharing going on then the COMMON_DEPEND pattern starts to fall
> > > apart. You potentially need,
> > >
> > > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
> > > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> > > (COMMON_DEPEND)
> > >
> > > This obviously gets worse as more DEPEND vars are introduced.
> > >
> >
> > Well not really, no -- the additional *DEPENDs that are being proposed
> > (or at least mentioned) for new EAPI will either remove atoms from
> > COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
> > COMMON_DEPEND or other intermediate variable won't really be necessary
> > for them.
>
> It depends on the dep type actually, and how we're viewing those deps-
> if they must be complete or not.
>
> Consider test depends for example.

I'd like to see some concrete discussion of link-depends/lib-depends, since
they represent a much more fundamental type of dependency, and would imo
remove atoms from a lot of other places.

Discussing test dependencies first, seems like putting the cart before the
horse.

> The point I'm trying to make here is that each dep phase
> should be authorative; in doing so, you start getting
> a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND
> isn't completely, but mostly a subset of RDEPEND as RDEPEND is a
> likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).
>
And all of them are a superset of LDEPEND (which is why I say it's more
fundamental, and class it as an omission from the original conversion
from ports to portage. The fact that link-dependencies are required to be
installed before build can begin, does not make them BUILD_DEPENDS, of
which DEPEND is the equivalent in Gentoo.)

In terms of what has been mentioned before, the set of dependencies
required at a particular point, LDEPEND are required at all of them.

As for the authoritative set at each point, you've already shown that the
package mangler has to build that currently, so there's no implementation
cost in continuing to do so.

> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
> the ebuild. Or you could just use a syntax form that allows you to
> directly inline that up front, rather than having to muck around w/
> intermediate vars.
>
Or you could address the gaping hole in the spec, and allow people to simply
specify the most fundamental dependency, without having to hack it in.

Then TDEPEND would just be what was required in addition during TEST phases,
a large amount of those common dependency variables would not be needed, and
similarly for C/HOST_DEPENDS (aka the new DEPEND under the awfully-specified
HDEPEND EAPI) since by definition a library dependency lives in the same
ROOT as the installed package.

>
> > Besides, this isn't actually a -problem- as there's nothing which
> > really requires one to use such helpers; ebuild writers just, well,
> > can.
>
> Getting to this point; yes, people could hack around it manually.
> Pretty sure that hasn't been in doubt.
>
> The question is what do we *gain* from making them do so. No one here
> has stated what we gain from COMMON_DEPENDS in comparison to
> DEPENDENCIES (where they can just inline it).
>
LDEPEND enables much better tracking of linkage, which is pretty fundamental
to building and maintaining a distribution. It's also something that
programmers specify upfront in configure and makefiles. As shown above,
it eliminates a lot of common dependencies, and at least some HDEPEND (if
that variable name is used accurately.)

Additionally, separating out the variable (which yes, can also be done with
a label, but it does need to be done) means linkage can be tracked
automatically based on the sort of data that every package supplies, every
other distribution provides, and every developer has to think about when
writing an ebuild; instead of on yet another syntax obfus^W complication that
was designed for java and python plugins.

Doing it in a separate variable as opposed to labelling it, means that
variable can be treated differently, should that be desired in a new EAPI.
Such a change in how LDEPEND were specified, would not impact on any of the
rest of the single variable format (which is more complex, and more
interdependent, at least in its scripted form) and would not need to be
verified against all the other types of specification in the rest of that
"integrated" variable. So a separate variable is more flexible.

It's also a *lot* more transparent.

For instance, if we'd had that before, do you agree we'd have been scanning
binaries to check for linkage outside that set and system? (After all, we
scan for a lot of other stuff already; scanelf is a gentoo success story.)

How would usage of revdep-rebuild been affected? What about implementation
of preserved-libs?

> And as I'm trying to lay out, where dependencies *should* be going
> towards, this issue will get worse if we're doing it out of band from
> DEPENDENCIES.
>
You've given a convincing argument for the package mangler using a unified
variable internally, and if the PM authors all agree, using that in the
cache format. (ie the extra parsing complexity and maintenance is worth it
for the difference in IO across the cache.)

But you've also shown exactly how all that information can be inferred from
the existing ebuild format (and once inferred would actually be simpler than
the whole variable specified in the ebuild.) As such, it appears more like
something that should be in the back-end, without requiring such a shift in
how people write ebuilds.

Your additional arguments as to why this gets more complex later down the
line are predicated on ignoring LDEPEND.

If that requires a bit more runtime at cache-generation, we're already paying
that price. Saving a small amount of time there, where most users never see
it, doesn't seem worth the mental effort, and maintenance cost, required
to change the format. "Just go ahead and use it in the cache," seems like
the best solution.

Regards,
steveL.
 

Thread Tools




All times are GMT. The time now is 07:54 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org