FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 11-23-2009, 05:49 PM
Denis Dupeyron
 
Default mtime preservation

I'm going to try and sum up the situation of the thread started in
[1]. Feel free to correct me or add to the summary in replies.

There seems to be two main ideas. I have removed the authors' name in
quotes below in order to make sure we talk about the ideas and not who
proposed them.

1- All packages are treated equally. Some files have their mtime
preserved, some don't. We need to agree on what files have their mtime
preserved and at what phase the mtime is frozen. What we seemed to
have agreed on is that the wording in PMS needs to be unambiguous
enough that it doesn't defeat the purpose. Here's a first proposition:

"The package manager must preserve modification times of regular files.
This includes files being compressed before merging. Exceptions to
this are:

(Now we need to enumerate the exceptions

* files newly created by the package manager,
(This will cover splitdebug, for example. (And please don't tell me
that the wording is flawed because the PM could save a file's contents
in some buffer, then delete the file and create it newly. This would
be as unreasonable as the rot-13 example.) )

* binary object files being stripped of symbols.

(Anything else missing from above list?)"


2- Here's the second idea, shamelessly pasted (note that it says EAPI4
below instead of EAPI3 but this is irrelevant to the idea):

"Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
similar to docompress) in both src_install and pkg_preinst. Doing so
would instruct the package manager that it must preserve mtimes
(including subsecond, if supported on the filesystem) for a particular
set of paths, even if doing so means no stripping etc. All other mtimes
may be rewritten as the package manager sees fit, and from EAPI 4
onwards must be rewritten at merge time for anything predating the
start of the build."

In both cases nanosecond resolution may be required and is a problem
due to python. The following workaround can be used until the
nanosecond issue is fixed in python:

"Alternatively, we could simply make portage spawn the mv binary
whenever rename fails (it fails when the source and destination are
on different devices). Although it's relatively slow, it should
solve the problem."

My intention is to ask the council to vote on which method is
preferable in two weeks. I will also ask the council on whether we
still want mtime preservation for EAPI3 or if we now think it's better
to push it to EAPI4. Please discuss.

Denis.

[1] http://archives.gentoo.org/gentoo-council/msg_becc15a2882cc7e4f1079b2f9f4dcaad.xml
 
Old 11-23-2009, 07:39 PM
Zac Medico
 
Default mtime preservation

Denis Dupeyron wrote:
> 1- All packages are treated equally. Some files have their mtime
> preserved, some don't. We need to agree on what files have their mtime
> preserved and at what phase the mtime is frozen.

I'd vote for method 1.

> My intention is to ask the council to vote on which method is
> preferable in two weeks. I will also ask the council on whether we
> still want mtime preservation for EAPI3 or if we now think it's better
> to push it to EAPI4. Please discuss.

You can probably do method 1 retroactively for all EAPIs, since the
few existing packages which require mtime preservation are
presumably broken already anyway (any damage is already done), and
packages which don't require mtime preservation are not hurt by it.
--
Thanks,
Zac
 
Old 11-23-2009, 10:19 PM
Brian Harring
 
Default mtime preservation

On Mon, Nov 23, 2009 at 11:49:25AM -0700, Denis Dupeyron wrote:
> 2- Here's the second idea, shamelessly pasted (note that it says EAPI4
> below instead of EAPI3 but this is irrelevant to the idea):
>
> "Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
> similar to docompress) in both src_install and pkg_preinst. Doing so
> would instruct the package manager that it must preserve mtimes
> (including subsecond, if supported on the filesystem) for a particular
> set of paths, even if doing so means no stripping etc. All other mtimes
> may be rewritten as the package manager sees fit, and from EAPI 4
> onwards must be rewritten at merge time for anything predating the
> start of the build."

Someone mind explaining to me why we're making mtime preservation so
nasty? Having to enumerate every pathway that requires mtime
preservation is pretty arduous for the ebuild dev, meaning it's
unlikely they'll get it right, leading to bugs.

The thing I'm not understanding here is that pkgcore since day one has
preserved mtime- I've yet to see any complaints about that nor any
issues caused by it. Portage shifted over a year or two back, same
thing, haven't heard complaints.

The thing that's bugging me about this whole discussion is that we're
spending more time focusing on stupid things the manager could do to
(basically intentionally) screw up the mtime preservation.

I know it won't fly, but realistically stating "the package manager
must preserve mtime- if there are instances where it does not (due to
some feature, perhaps splitdebug or some form of compression) it is
the package managers responsibility to ensure this does not break the
ebuilds resultant merge/runtime invocation".

Via such wording an exemption is created and it's made clear it's the
managers responsibility to keep things playing nice... if the manager
can't do that, then the feature/functionality that is changing the
mtime (resulting in the pkg on disk failing) must be fixed or
disabled.

The nice thing about this wording is that it basically matches what
the case is now, and what has worked for quite a few years.


> In both cases nanosecond resolution may be required and is a problem
> due to python. The following workaround can be used until the
> nanosecond issue is fixed in python:

It'd be nice if someone enumerated merge scenarios where nanosecond
resolution is actually required. Seems like a white elephant to me,
especially in combination w/ the fact that the target fs may not
support nanosecond.

Basically, if it's required the manager support nanosecond resolution
for merging, ebuilds must be able to rely on that- end result, any
merging to FS's that do not support nanosecond (this includes the
intermediate ${D}) are no longer supported. Unless I'm missing
something, iso-9660 doesn't look to support nanosecond resolution.
Beyond that, does cramfs/squashfs? If not, taking this to the
logical conclusion the livecds aren't technically supported as a merge
environment.

Essentially If we're going to require the manager support NS
resolution w/out requiring all FS's involved support NS level
resolution, in reality ebuilds can't rely on NS level resolution.

Thus drop the requirement since we gain nothing from it- if it's not a
gurantee ebuilds can rely on then they should be writing to the lowest
common denominator- second level resolution. That may sound heinous,
but it's also being realistic.


> "Alternatively, we could simply make portage spawn the mv binary
> whenever rename fails (it fails when the source and destination are
> on different devices). Although it's relatively slow, it should
> solve the problem."

Yeah... no. Slowing down the main manager for a thereotical edge case
doesn't seem particularly useful to me

~harring
 
Old 11-24-2009, 01:26 PM
Duncan
 
Default mtime preservation

Brian Harring posted on Mon, 23 Nov 2009 15:19:00 -0800 as excerpted:

>> "Alternatively, we could simply make portage spawn the mv binary
>> whenever rename fails (it fails when the source and destination are on
>> different devices). Although it's relatively slow, it should solve the
>> problem."
>
> Yeah... no. Slowing down the main manager for a thereotical edge case
> doesn't seem particularly useful to me

I say go for second resolution now, basically using the #1 proposal, and
apply it to all eapis for the reasons zac outlined, but with a fixed
effective date say 90 or 180 days out from the council resolution to give
PMs time to come into compliance. To try for nanosecond resolution now
is simply letting the complex perfect be the enemy of the simple good,
when what we have at present is the no-good.

Then make a donsmtimes like the #2 proposal, that individual ebuilds can
call for specific paths if it's really necessary. It's more work for the
ebuilder, but that's in relative measure to the slowdown it'll cause on
two out of three PMs including our core PM, so a bit of discouragement
from using it could be considered a good thing, tho it would be there, if
required.

If at some point in the future ns resolution becomes a much bigger issue
and many ebuilds are using the donsmtimes call, there's nothing stopping
us from looking at further tightening up the requirements for the general
case, probably as part of an eapi, then.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 11-24-2009, 09:21 PM
Ciaran McCreesh
 
Default mtime preservation

On Mon, 23 Nov 2009 15:19:00 -0800
Brian Harring <ferringb@gmail.com> wrote:
> Someone mind explaining to me why we're making mtime preservation so
> nasty? Having to enumerate every pathway that requires mtime
> preservation is pretty arduous for the ebuild dev, meaning it's
> unlikely they'll get it right, leading to bugs.

It's not in the least bit nasty. It's requiring people to be explicit
about special requirements.

> The thing I'm not understanding here is that pkgcore since day one
> has preserved mtime- I've yet to see any complaints about that nor
> any issues caused by it. Portage shifted over a year or two back,
> same thing, haven't heard complaints.

You can't have been listening very hard, then. The complaint is that it
results in files with dumb mtimes being merged to /.

> I know it won't fly, but realistically stating "the package manager
> must preserve mtime- if there are instances where it does not (due to
> some feature, perhaps splitdebug or some form of compression) it is
> the package managers responsibility to ensure this does not break the
> ebuilds resultant merge/runtime invocation".
>
> Via such wording an exemption is created and it's made clear it's the
> managers responsibility to keep things playing nice... if the manager
> can't do that, then the feature/functionality that is changing the
> mtime (resulting in the pkg on disk failing) must be fixed or
> disabled.
>
> The nice thing about this wording is that it basically matches what
> the case is now, and what has worked for quite a few years.

Wording such as that just isn't suitable for a specification. It
requires implementers to guess what future ebuilds are going to
rely upon (and ebuilds regularly do rely upon weird quirks), and makes
it impossible to produce a package manager that can be shown to be
compliant.

> > In both cases nanosecond resolution may be required and is a problem
> > due to python. The following workaround can be used until the
> > nanosecond issue is fixed in python:
>
> It'd be nice if someone enumerated merge scenarios where nanosecond
> resolution is actually required. Seems like a white elephant to me,
> especially in combination w/ the fact that the target fs may not
> support nanosecond.

POSIX considers several of the non-nanosecond resolution calls to be
deprecated. Going "la la la I can't hear you!" because Python happens
to have utterly screwed this up is just going to lead to problems when
programs start using sub-second validity checks -- 'make' already does,
and it's given rise to various build-directory-related issues.

> Basically, if it's required the manager support nanosecond resolution
> for merging, ebuilds must be able to rely on that- end result, any
> merging to FS's that do not support nanosecond (this includes the
> intermediate ${D}) are no longer supported. Unless I'm missing
> something, iso-9660 doesn't look to support nanosecond resolution.
> Beyond that, does cramfs/squashfs? If not, taking this to the
> logical conclusion the livecds aren't technically supported as a
> merge environment.

No, we're after a requirement that the package manager must not screw up
nanosecond-resolution timestamps, and must not partially preserve and
partially not preserve them.

> > "Alternatively, we could simply make portage spawn the mv binary
> > whenever rename fails (it fails when the source and destination are
> > on different devices). Although it's relatively slow, it should
> > solve the problem."
>
> Yeah... no. Slowing down the main manager for a thereotical edge
> case doesn't seem particularly useful to me

...or you could just fix Python.

--
Ciaran McCreesh
 
Old 11-25-2009, 08:13 PM
Denis Dupeyron
 
Default mtime preservation

A quick note to tell you that I have tried to look for examples of
breakage due to how mtime preservation is currently implemented in
portage. Unfortunately I didn't find anything, maybe because I'm not
knowledgeable enough or because I can't afford to spend any more time
on this. If any of you can provide an example then please do so.

In case nobody could show any sign of such breakage I would have to
add to the list of two propositions to vote on:
3- Do nothing.

We can always point to code that shows the implementation is
suboptimal but we have to understand that some of us who are less
proactive at fixing issues may want to procrastinate until breakage
actually happens.

Denis.
 
Old 11-25-2009, 08:27 PM
Ciaran McCreesh
 
Default mtime preservation

On Wed, 25 Nov 2009 14:13:58 -0700
Denis Dupeyron <calchan@gentoo.org> wrote:
> A quick note to tell you that I have tried to look for examples of
> breakage due to how mtime preservation is currently implemented in
> portage. Unfortunately I didn't find anything, maybe because I'm not
> knowledgeable enough or because I can't afford to spend any more time
> on this. If any of you can provide an example then please do so.

Portage uses two ways of merging a file: os.rename() and the manual
way.

os.rename() correctly preserves mtimes.

Python's os.utime call, which is what Portage uses to preserve mtimes
for files that it installs the hard way, and which is not a wrapper for
the Unix utime call, uses an IEEE 754 double to handle timestamps. But
a double only gives sixteen accurate decimal digits, and you've got ten
to the left of the decimal point, leaving only six reliably preserved,
with the remaining three digits being corrupted.

Thus, packages can end up being installed with some files having
accurately-preserved mtimes, and some having corrupted mtimes, which
will lead to mtime comparisons giving incorrect results.

--
Ciaran McCreesh
 
Old 11-25-2009, 08:52 PM
Duncan
 
Default mtime preservation

Ciaran McCreesh posted on Wed, 25 Nov 2009 21:27:18 +0000 as excerpted:

> Portage uses two ways of merging a file: os.rename() and the manual way.
>
> os.rename() correctly preserves mtimes.
>
> Python's os.utime call, which is what Portage uses to preserve mtimes
> for files that it installs the hard way, and which is not a wrapper for
> the Unix utime call, uses an IEEE 754 double to handle timestamps. But a
> double only gives sixteen accurate decimal digits, and you've got ten to
> the left of the decimal point, leaving only six reliably preserved, with
> the remaining three digits being corrupted.
>
> Thus, packages can end up being installed with some files having
> accurately-preserved mtimes, and some having corrupted mtimes, which
> will lead to mtime comparisons giving incorrect results.

That's a great explanation (thanks, I now know the details to the degree
I'd be interested), but what was asked for was examples of breakage, aka
actual bugs. I seem to remember either you or someone else mentioning
that there had been a couple issues already, and I'll take the claim at
face value, but actual bug numbers would be nice. =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
 
Old 11-25-2009, 09:13 PM
Ciaran McCreesh
 
Default mtime preservation

On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:
> That's a great explanation (thanks, I now know the details to the
> degree I'd be interested), but what was asked for was examples of
> breakage, aka actual bugs.

Why? You can easily look and see that it's broken. Examples will merely
be dismissed as one-off cases that can be worked around, or as relying
upon a string of coincidences that will "obviously" never really
happen, right up until they do, at which point they'll be dismissed
with a WORKSFORME. What you have is a proof that it's broken, which is
far better than an example.

--
Ciaran McCreesh
 
Old 11-25-2009, 09:59 PM
Ulrich Mueller
 
Default mtime preservation

>>>>> On Wed, 25 Nov 2009, Ciaran McCreesh wrote:

> On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
> Duncan <1i5t5.duncan@cox.net> wrote:
>> That's a great explanation (thanks, I now know the details to the
>> degree I'd be interested), but what was asked for was examples of
>> breakage, aka actual bugs.

> Why? You can easily look and see that it's broken.

Only for a suitable definition of "broken".

> Examples will merely be dismissed as one-off cases that can be
> worked around, or as relying upon a string of coincidences that will
> "obviously" never really happen, right up until they do, at which
> point they'll be dismissed with a WORKSFORME.

Real examples would be issues like bugs 83877 [1] or 263387 [2].
Nothing that could be easily dismissed or worked around. Both issues
are fixed with Portage since a long time.

I don't know of any example where non-preservation of nanosecond
timestamps would cause problems.

> What you have is a proof that it's broken, which is far better than
> an example.

So we have a proven theorem, but unfortunately the cases where it is
applicable form an empty set. ;-)

Ulrich

[1] <http://bugs.gentoo.org/83877#c36>
[2] <http://bugs.gentoo.org/263387>
 

Thread Tools




All times are GMT. The time now is 01:35 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org