FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 10-28-2009, 04:11 AM
Brian Harring
 
Default adding a modification timestamp to the installed pkgs database (vdb)

On Tue, Oct 27, 2009 at 11:32:30AM -0700, Zac Medico wrote:
> Brian Harring wrote:
> > The proposal is pretty simple; if code modifies the vdb in any
> > fashion, it needs to update the mtime on a file named
> > '.modification_time' in the root of the vdb.
>
> I'd to prefer using the mtime of the /var/db/pkg directory itself,
> since existence of a '.modification_time' file isn't going to prove
> that an programs that don't recognize that file haven't made any
> modifications.

Grumble. Works for me.


> We can also use the mtimes of category subdirectories, in order to
> indicate whether a modification has occurred in any given category.

Pkgcore already relies on that for old style virtuals cache. The
pisser there is that modifications w/in a node don't result in a
category level mtime- it certainly would be nice to have it formalized
in some fashion so that cache regeneration could just work on the
areas it needs to.

~brian
 
Old 01-12-2010, 09:12 AM
Ciaran McCreesh
 
Default adding a modification timestamp to the installed pkgs database (vdb)

On Mon, 11 Jan 2010 15:35:51 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> I'm a bit surprised by the low amount of discussions this topic has
> generated.

There's no discussion because Brian refuses to address any comments on
the proposal and just says "we should do it anyway, and if you want it
done properly instead, do it yourself".

--
Ciaran McCreesh
 
Old 01-12-2010, 10:12 PM
Brian Harring
 
Default adding a modification timestamp to the installed pkgs database (vdb)

On Tue, Jan 12, 2010 at 10:12:52AM +0000, Ciaran McCreesh wrote:
> On Mon, 11 Jan 2010 15:35:51 -0700
> Denis Dupeyron <calchan@gentoo.org> wrote:
> > I'm a bit surprised by the low amount of discussions this topic has
> > generated.
>
> There's no discussion because Brian refuses to address any comments on
> the proposal and just says "we should do it anyway, and if you want it
> done properly instead, do it yourself".

This is a bit of bullshit, per the norm. There is plenty of
discussion- the problem is you don't like the direction it's gone.
You want a whole new vdb- I don't oppose that. However I'm not
interested in trying to standardize a new vdb format into PMS, at
least not yet.

Your argument can basically be summed up as "don't do the minimal
tweak, do the whole new vdb with defined caches that all can share".

Which is a fine notion, but not at all what I'm interested in- I want
smething w/in the next 6 months PMs can start deploying for user
benefit. I've literally seen discussions about vdb2 for >5 years now-
pinning performance bets on a white elephant is not in my interest.
Further, a vdb2 specified only in spce is useless w/out actual long
term usage of it for performance/reliability testing.


The daft thing about this is that you're ignoring one core transition
issue w/ vdb2- if someone did create a vdb2, they still would need a
synchronization mechanism (one quite similar to what I'm proposing).
Already stated that in the bug-

http://bugs.gentoo.org/show_bug.cgi?id=290428#c11

Repeating the relevant snippet for your benefit, consider the
following chain of events-

1) portage/pkgcore support the PMS defined vdb2 while paludis doesn't
2) portage/pkgcore are invoked modifying the livefs; vdb1, vdb2 is
updated.
3) paludis is invoked. vdb1 is updated, vdb2 is not
4) portage and pkgcore now cannot rely upon vdb2, since vdb1 now
contains extra modifications due to paludis not supporting vdb2.


To address #4, you need a way to detect that vdb1 was modified... then
portage/pkgcore can synch vdb2 up w/ the changes in vdb1 (namely via
basically rebuilding vdb2 from vdb1).

The only real alternative to the issue above is to that of dropping
vdb1- when you jump to vdb2, you drop vdb1. This means all existing
tools have to know of both locations, have to support at least read on
both locations- if a PM (again, paludis for the example) didn't
support VDB2, under the alternative approach paludis wouldn't be
usable since there would not be a vdb for it to inspect.

In my opinion that is not a viable alternative- but you're free to
propose whatever the hell you want.


Summarizing; the synchronization primitive is needed for any future
vdb2; once all vdb modifiers (paludis being the sole hold out per the
norm) support updating the timestamp (and a couple of months have
gone by), you get the following benefits-

1) PMs can rebuild their vdb cache validation to use that timestamp,
simplifying their code and reducing redundancy in their validation
2) caches that are currently too expensive to deploy due to the cost
of having to validate their way through the vdb can rely on
effectively a boolean- is the timestamp newer then what they know of.
This makes said caches access far faster, making them a net win
(instead of a net loss)- portages vdb metadata cache comes to mind for
this one for example.
3) interested parties can start developing their own vdb2, including
letting users play with it while avoiding forced lock in (this is good
on multiple accounts- a vdb2 existing only in a spec is worthless in
performance promises compared to a vdb2 implemented and tested).


Summing it up; what ciaran wants is reliant on what I'm proposing,
what I'm proposing provides tangible benefits now, not years down the
line as his proposal would entail.

There's the lovely executive summary.
~harring
 
Old 01-17-2010, 07:59 AM
Ciaran McCreesh
 
Default adding a modification timestamp to the installed pkgs database (vdb)

2010/1/12 Brian Harring <ferringb@gmail.com>:
>> There's no discussion because Brian refuses to address any comments
>> on the proposal and just says "we should do it anyway, and if you
>> want it done properly instead, do it yourself".
>
> This is a bit of bullshit, per the norm. *There is plenty of
> discussion- the problem is you don't like the direction it's gone.
> You want a whole new vdb- I don't oppose that. *However I'm not
> interested in trying to standardize a new vdb format into PMS, at
> least not yet.

No, I want a decent cache proposal that lets package managers know
what's changed, not one that sometimes (but not always) might let
package managers know when some things have changed, but not what's
changed and not what they can still assume.

> Your argument can basically be summed up as "don't do the minimal
> tweak, do the whole new vdb with defined caches that all can share".

No, I want the well defined caches that all can share.

> The daft thing about this is that you're ignoring one core transition
> issue w/ vdb2- if someone did create a vdb2, they still would need a
> synchronization mechanism (one quite similar to what I'm proposing).

If you replace VDB, you need a well defined cache mechanism. So let's
do that bit now.

> 1) portage/pkgcore support the PMS defined vdb2 while paludis doesn't
> 2) portage/pkgcore are invoked modifying the livefs; vdb1, vdb2 is
> updated.
> 3) paludis is invoked. *vdb1 is updated, vdb2 is not
> 4) portage and pkgcore now cannot rely upon vdb2, since vdb1 now
> contains extra modifications due to paludis not supporting vdb2.

No, we'd not do it that way. If we're ditching VDB, the only sane way
to do it is to ditch it with an rm -fr when creating the new layout.
Keeping two sets of data around is going to lead to breakage no matter
how well we do things.

> Summarizing; the synchronization primitive is needed for any future
> vdb2

No. A *proper* cache validation mechanism is needed. What you're
suggesting isn't enough to use for anything at all.

> Summing it up; what ciaran wants is reliant on what I'm proposing,

No, what I want in the long term is reliant upon implementing a decent
cache setup in the short term.

--
Ciaran McCreesh
 
Old 01-17-2010, 08:24 AM
Tobias Klausmann
 
Default adding a modification timestamp to the installed pkgs database (vdb)

Hi!

On Sun, 17 Jan 2010, Ciaran McCreesh wrote:
> > 1) portage/pkgcore support the PMS defined vdb2 while paludis doesn't
> > 2) portage/pkgcore are invoked modifying the livefs; vdb1, vdb2 is
> > updated.
> > 3) paludis is invoked. *vdb1 is updated, vdb2 is not
> > 4) portage and pkgcore now cannot rely upon vdb2, since vdb1 now
> > contains extra modifications due to paludis not supporting vdb2.
>
> No, we'd not do it that way. If we're ditching VDB, the only sane way
> to do it is to ditch it with an rm -fr when creating the new layout.
> Keeping two sets of data around is going to lead to breakage no matter
> how well we do things.

Please also provide a downgrade path, i.e. a way to go back from
the new DB version to the current one should it be necessary (if
there is no such path, Murphy will see to it that the new format
breaks in interesting[0] ways).

Just my 0.0139304869 Euros (at current exchange rate),
Tobias

[0] Chinese-style interesting

--
printk("whoops, seeking 0
");
linux-2.6.6/drivers/block/swim3.c
 
Old 01-17-2010, 09:48 AM
Christian Faulhammer
 
Default adding a modification timestamp to the installed pkgs database (vdb)

Hi,

Ciaran McCreesh <ciaran.mccreesh@googlemail.com>:
> That probably wouldn't be possible. One of the reasons we want to
> ditch VDB is to allow multiple slots of the same cat/pkg-ver to be
> installed in parallel (which is in turn necessary to allow some of the
> more hideous dynamic slot abuses that people are after). VDB doesn't
> support that, so you probably won't be able to go back once you've
> started using new features.
>
> *shrug* all of this is years off, anyway. It's at least EAPI 5
> territory. We can work all this out later if EAPI 4 ever happens.

As much as you love to have the new and shiny VDB2, it is far off.
Prototyping and drafting implementations would be great to have some
base where we can discuss on (in a civil manner). So having this
timestamp would be a good way to prepare a sane migration path. In the
end we have to care for our users and EAPI was just provided to have a
clear way.

V-Li

--
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>
 
Old 01-18-2010, 02:42 PM
Brian Harring
 
Default adding a modification timestamp to the installed pkgs database (vdb)

On Sun, Jan 17, 2010 at 11:09:07AM +0000, Ciaran McCreesh wrote:
> 2010/1/17 Christian Faulhammer <fauli@gentoo.org>:
> > Ciaran McCreesh <ciaran.mccreesh@googlemail.com>:
> > *As much as you love to have the new and shiny VDB2, it is far off.
> > Prototyping and drafting implementations would be great to have some
> > base where we can discuss on (in a civil manner). *So having this
> > timestamp would be a good way to prepare a sane migration path.
>
> No, it wouldn't. Brian's proposal a) would be of no use whatsoever for
> VDB2 migration, and b) would not be used by VDB2. Having a *decent*
> cache validation mechanism is a good idea; having a half-baked one is
> a waste of time.

Propose something, or shut up frankly.

If all you're going to contribute is "it's half baked" claims, you're
wasting folks time. You've had a couple of months of time to
counterpropose something- back it up with a proposal or be silent
please.

As is, quite a few folk see how experimental vdb2/vdb1 synchronization
can be done w/ this timestamp- your claims thus far that it won't work
seem to boil down to "but not everyone will update the timestamp".

Which gets right back to why I'm elevating this to the council to
*force* PMS to include this, thus force the holdout (paludis) to
update the timestamp thus invalidating your cyclical claim.

Either way, you find issues w/ the proposal you're more then free to
propose something else- hell, I'll even listen if it's sane.

What I won't do is sit around and listen to you whinge about the sky
falling or that I/others are being idiots via not going
the route *you* want and standardizing caches across all the managers-
as I said, you want that functionality *you* propose it.

About the only thing paludis shares w/ portage/pkgcore is a potential
installed-pkgs-cache of pkg names; this isn't incredibly useful
frankly (it's nice for cold cache searches but that's it). The cache
usage between portage/pkgcore vs paludis differs a fair bit, as such
trying to define an LCD vdb cache is pointless. Further it's not what
I'm after and you've already opposed adding caches to vdb1 w/in the
ticket- you want something beyond this, then go nuts.


Either way, that's pretty much the bar I'm sitting for continuing
discussion of this w/ you- either it's going to be productive w/
specific claims (no more of this vague handwaving bullshit) and moving
towards accomplishing something or I'm just going to continue
ignoring your disruptive behaviour, instead getting majority PMS
consensus and then pushing it up to the council bypassing your
shenanigans.

It's not how things should be done, but it's about the only way to get
something done when you dig in and go cyclical. Wish it weren't that
way, but I've more interest in progress then playing games w/
folk looking to be poisonous.

Seriously, if you can't even be bothered to spell out your claims in
full or layout a counter proposal, instead spending your time
screaming "nyah nyah it won't work!" as you did for prefix, I'm not
having it.

There are better uses of folks time frankly, and users deserve
functionality over daft pissing matches.

Be productive and constructive, or be ignored pretty much.
~harring
 

Thread Tools




All times are GMT. The time now is 12:25 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org