FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 10-26-2010, 07:28 AM
Raphael Hertzog
 
Default Pre-approval request for dpkg sync() changes for squeeze

On Tue, 26 Oct 2010, Andreas Barth wrote:
> * Phillip Susi (psusi@cfl.rr.com) [101025 17:00]:
> > On 10/24/2010 1:20 PM, Andreas Barth wrote:
> > > I quite often run dpkg installs / upgrades in pbuilder ramdisks. If
> > > the sync could be reduced to only that ramdisk, everything would be
> > > fine.
> >
> > That's exactly the environment where you would disable syncing entirely
> > since you don't care if the pbuilder chroot becomes corrupted if the
> > system crashes; you will just rebuild it from the tarball anyhow.
>
> So, what is the appropriate configuration setting for dpkg, and where
> are the bug reports for our usual tools to auto-set it correct? If
> there is none, this argument is moot.

This request made to the release team is precisely about allowing us to
introduce a new dpkg option that can be used to disable syncing despite
being this late in the release cycle.

Until this request is accepted it's difficult to file bug reports asking
people to use the option. I expect at least d-i would want to use this
option by default.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer ◈ [Flattr=20693]

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101026072807.GE29921@rivendell.home.ouaza.com">h ttp://lists.debian.org/20101026072807.GE29921@rivendell.home.ouaza.com
 
Old 11-05-2010, 09:18 PM
Philipp Kern
 
Default Pre-approval request for dpkg sync() changes for squeeze

Hi,

On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote:
> I'd like to apply two patches for the next dpkg 1.15.8.6 upload. And
> would like to run them through you to avoid having to possibly revert
> them from dpkg's sid git branch.
>
> Those are related to the fsync()/sync() changes in dpkg from some time
> ago, the patches would:
>
> 1) Switch back from sync() to fsync() before rename() (while keeping
> the sync() code around for the benefit of other distributions
> that might not want to switch just yet). So to avoid unrelated
> I/O when there's background work being done for example. This
> hack also only works on Linux where sync() is synchronous, so
> it would unify that code path for all dpkg supported platforms.
>
> Bug: #588339
>
> <http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=87740373>

I do have to say that I'm not overly happy about this change this late in the
freeze. I suppose this resembles what we currently have in Lenny? What are
the implications for ext4 and btrfs? Why was it changed in the first place?

> 2) Add a new --force-unsafe-io to disable those fsync() for people
> using certain file systems where the only option is to choose
> between reliability or acceptable performance. Or in cases where
> the data is cached and it might not be important to lose it.
>
> Bug: #584254
>
> <http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=0484cd7d>

So even when you don't do sync() anymore you still want to disable those
fsyncs()? Granted, it's a self-contained new feature. What's the use case
for squeeze? d-i I heared, but I'm not sure that they'd use it in this
cycle. And tmpfs users, or who?

Kind regards,
Philipp Kern
 
Old 11-06-2010, 06:46 AM
Guillem Jover
 
Default Pre-approval request for dpkg sync() changes for squeeze

On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote:
> On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote:
> > Those are related to the fsync()/sync() changes in dpkg from some time
> > ago, the patches would:
> >
> > 1) Switch back from sync() to fsync() before rename() (while keeping
> > the sync() code around for the benefit of other distributions
> > that might not want to switch just yet). So to avoid unrelated
> > I/O when there's background work being done for example. This
> > hack also only works on Linux where sync() is synchronous, so
> > it would unify that code path for all dpkg supported platforms.
> >
> > Bug: #588339
> >
> > <http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=87740373>
>
> I do have to say that I'm not overly happy about this change this late in the
> freeze.

My idea was to get those on the planned 1.15.9, but after the sudden
freeze and the initial reactions to the first unblock request I didn't
feel like asking for any more changes. Anyway, the blame would be
obviously on me in case these do not get accepted due to the delay.

> I suppose this resembles what we currently have in Lenny?

No, lenny's dpkg does not do sync() nor fsync() before renames on
extracted files from archives.

dpkg 1.15.6 started doing fsync() for extracted files.
dpkg 1.15.7.2 switched from fsync() to sync() on Linux.

> What are the implications for ext4 and btrfs?

They are going to be slower with this change.

> Why was it changed in the first place?

I wrote about this in <http://lwn.net/Articles/392599/>. Extremely short
summary: because fsync() was considered too slow on ext4 (#578635).

> > 2) Add a new --force-unsafe-io to disable those fsync() for people
> > using certain file systems where the only option is to choose
> > between reliability or acceptable performance. Or in cases where
> > the data is cached and it might not be important to lose it.
> >
> > Bug: #584254
> >
> > <http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=0484cd7d>
>
> So even when you don't do sync() anymore you still want to disable those
> fsyncs()?

Given the performance degradation the first patch will introduce on at
least ext4, users might want to disable fsync()s. Arguably they seem
to want to do that now even with sync(), as people are already using
hacks like eatmydata to disable them, the problem is that this will
disable *all* syncs, including the ones for the database among others.
If the file system gets corrupted you might still have a chance of
recovering, if the databases do, you most probably need either backups
or a reinstall.

> Granted, it's a self-contained new feature. What's the use case
> for squeeze? d-i I heared, but I'm not sure that they'd use it in this
> cycle. And tmpfs users, or who?

Other users are buildds using fresh unpacked snapshots for each build
session. ext4 users who would prefer faster dpkg runs rather than data
safety.


The problem with sync() is that even if it gives better performance
than fsync()s on ext4, it's just the wrong thing to use, sync()
affects all mount points, so if you happen to have for example a server
with lots of writes on say a /srv mount point, and the admin decides to
do a large install/upgrade, it will force all that data down to all
mount points, for each archive unpacked.

The target for this is ext4 users mainly, the syncs were included mostly
to protect them from data loss. They are also the ones suffering the
I/O performance degradation. I don't think it's fair to make users not
suffering such problems suffer unrelated issues like the ones imposed by
sync(). As such I think it should be them who have to decide between
performance or data loss.


Also I found it interesting that the ext4 developers were complaining
about userspace developers not doing the right thing regarding atomic
renames (which would include fsync()) during the huge zero-length
discussions, to then state that fsync() is obviously too slow and heavy
handed and thus should better not be used.

<https://bugzilla.kernel.org/show_bug.cgi?id=15910>

Ultimately I just think this is a problem with ext4. Regardless of that,
I'll try to come up with a better, portable alternative to improve the
performance for 1.16.x if possible.


Finally, while accepting patch 2) alone might make sense, accepting 1)
alone would not.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101106074601.GA32490@gaara.hadrons.org">http://lists.debian.org/20101106074601.GA32490@gaara.hadrons.org
 
Old 11-06-2010, 08:20 AM
Raphael Hertzog
 
Default Pre-approval request for dpkg sync() changes for squeeze

Hi,

On Sat, 06 Nov 2010, Guillem Jover wrote:
> Finally, while accepting patch 2) alone might make sense, accepting 1)
> alone would not.

BTW, I think we should go with patch 2 alone currently (i.e. just add
--force-unsafe-io).

Continuing to use sync() instead of fsync() is the best compromise we have
currently to have reasonable performance on Linux with all filesystems.

I don't think that introducing a big performance degradation on ext4 users
is a good idea at this point of the release cycle even if they have a
work-around with --force-unsafe-io.

I would like however to see --force-unsafe-io so that we can have the best
performance when we want it (in d-i, in temporary build chroots, etc.).

Cheers,
--
Raphaël Hertzog ◈ Debian Developer ◈ [Flattr=20693]

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101106092029.GE1478@rivendell.home.ouaza.com">ht tp://lists.debian.org/20101106092029.GE1478@rivendell.home.ouaza.com
 
Old 11-06-2010, 08:41 AM
Sven Joachim
 
Default Pre-approval request for dpkg sync() changes for squeeze

On 2010-11-06 08:46 +0100, Guillem Jover wrote:

> On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote:
>> On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote:
>
>> What are the implications for ext4 and btrfs?
>
> They are going to be slower with this change.

Are you sure this is the case for btrfs? Mike Hommey has described the
sync() implementation as really horrible[0] (unfortunately the webserver on
glandium.org is currently down, but the Google cache[1] has a copy).

For ext4, mounting with the nodelalloc option helps a lot, although this
option allegedly slows down ext4 in the general case.

Regards,
Sven


0. http://glandium.org/blog/?p=1169
1. http://webcache.googleusercontent.com/search?q=cache:zFsVyqYwxAAJ:glandium.org/blog/%3Fp%3D1169+mike+hommey+dpkg+sync&cd=1&ct=clnk


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87r5eywzz9.fsf@turtle.gmx.de">http://lists.debian.org/87r5eywzz9.fsf@turtle.gmx.de
 
Old 11-06-2010, 06:13 PM
Guillem Jover
 
Default Pre-approval request for dpkg sync() changes for squeeze

On Sat, 2010-11-06 at 10:41:46 +0100, Sven Joachim wrote:
> On 2010-11-06 08:46 +0100, Guillem Jover wrote:
> > On Fri, 2010-11-05 at 23:18:47 +0100, Philipp Kern wrote:
> > > On Fri, Oct 22, 2010 at 11:35:54AM +0200, Guillem Jover wrote:
> > > What are the implications for ext4 and btrfs?
> >
> > They are going to be slower with this change.
>
> Are you sure this is the case for btrfs? Mike Hommey has described the
> sync() implementation as really horrible[0] (unfortunately the webserver on
> glandium.org is currently down, but the Google cache[1] has a copy).

Buh, yeah, sorry, I meant that to apply only to ext4.

I didn't remember Mike's blog post about btrfs, but Modestas pointed out
in [2] it's twice as worse using sync().

[2] <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254>

> For ext4, mounting with the nodelalloc option helps a lot, although this
> option allegedly slows down ext4 in the general case.

Ah right, that was pointed out in a thread recently. This should also
fix any zero-length problems on ext4 too AFAIUI.

> 0. http://glandium.org/blog/?p=1169
> 1. http://webcache.googleusercontent.com/search?q=cache:zFsVyqYwxAAJ:glandium.org/blog/%3Fp%3D1169+mike+hommey+dpkg+sync&cd=1&ct=clnk

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101106191317.GA19493@gaara.hadrons.org">http://lists.debian.org/20101106191317.GA19493@gaara.hadrons.org
 
Old 11-06-2010, 08:18 PM
Guillem Jover
 
Default Pre-approval request for dpkg sync() changes for squeeze

Hi!

On Sat, 2010-11-06 at 10:20:29 +0100, Raphael Hertzog wrote:
> On Sat, 06 Nov 2010, Guillem Jover wrote:
> > Finally, while accepting patch 2) alone might make sense, accepting 1)
> > alone would not.
>
> BTW, I think we should go with patch 2 alone currently (i.e. just add
> --force-unsafe-io).
>
> Continuing to use sync() instead of fsync() is the best compromise we have
> currently to have reasonable performance on Linux with all filesystems.

I disagree, as that does not seem to be the case currently anyway.

> I don't think that introducing a big performance degradation on ext4 users
> is a good idea at this point of the release cycle even if they have a
> work-around with --force-unsafe-io.

Installing/upgrading software is not something you usually do that often
on a stable system. And I'm planning to apply the switch to fsync()
for 1.16.0 so that will almost immediately affect testing/sid once the
freeze is over anyway.

> I would like however to see --force-unsafe-io so that we can have the best
> performance when we want it (in d-i, in temporary build chroots, etc.).

Well, I think it's completely unfair, that due to "design flaws" (some
might want to call that conformance to the spec) in a file system,
everyone has to suffer for it. We have to remember all this is mostly
for ext4 benefit after all. I'm always eager to do the right thing, in
this case using fsync() if it fixes real problems, but when the correct
and simple solution [0] which is also what was initially proposed and
expected (just look at man mount, auto_da_alloc, among others) by those
same who designed such system is impracticable on *that* file system,
then that's just not right.

[0] Not to take into account the delayed rename() gymnastics we have
had to do in dpkg, and possible future additional complexity,
aio/fsync() in threads or subprocesses/etc?.

The worst that could happen on other file systems w/o the sync()/fsync()
before rename()s for extracted files was that the dpkg database might
get slightly out of sync relative to what was installed on disk, but
that's at most confusing, nothing compared to getting zero-length files
all over the place.

Switching to sync() was a mistake, a workaround that seemingly improved
things, but a workaround non the less with pernicious side effects.
Obviously --force-unsafe-io is also a workaround, one I'd rather not
have, but at least it's portable and has some utility outside ext4.

The zero-length problem should affect only new recently installed
systems with ext4 anyway. Those users might be ok with dpkg doing safe
I/O, but they will not be ok with say user data which can get lost as
easily, something I think is worse than system files. For that matter
using mostly any software (except few, probably database related) will
be dangerous, you just have to check around for the few projects using
fsync() at all! Not even rpm does for the extracted files.

Using nodelalloc as Sven pointed out, is IMO the only sane option for
those users if they value their data (also data=ordered). And something
that should probably be mentioned in the release notes anyway regardless
of any dpkg change. Or just recommend not using ext4 at all?

This and my other related mails might come across probably as quite
loaded, but the situation regarding ext4 does not leave much options
available, so I've been feeling between a rock and a hard place.

Anyway, I think I'm starting to repeat myself, and that I've exposed
my case, I'll leave the release team to deliberate, and in case they
decline I'll probably prepare dpkg backports to be placed somewhere.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101106211855.GB19493@gaara.hadrons.org">http://lists.debian.org/20101106211855.GB19493@gaara.hadrons.org
 
Old 11-06-2010, 08:33 PM
Jonathan Nieder
 
Default Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote:

> The worst that could happen on other file systems w/o the sync()/fsync()
> before rename()s for extracted files was that the dpkg database might
> get slightly out of sync relative to what was installed on disk, but
> that's at most confusing, nothing compared to getting zero-length files
> all over the place.
[...]
> The zero-length problem should affect only new recently installed
> systems with ext4 anyway.

To throw another variable in the works: doesn't ext4 fall in the same
category as "other file systems" today? See v2.6.30-rc1~416^2~15
(ext4: Automatically allocate delay allocated blocks on rename,
2009-02-23).


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101106213314.GA32653@burratino">http://lists.debian.org/20101106213314.GA32653@burratino
 
Old 11-07-2010, 04:12 AM
Guillem Jover
 
Default Pre-approval request for dpkg sync() changes for squeeze

On Sat, 2010-11-06 at 16:33:14 -0500, Jonathan Nieder wrote:
> Guillem Jover wrote:
> > The worst that could happen on other file systems w/o the sync()/fsync()
> > before rename()s for extracted files was that the dpkg database might
> > get slightly out of sync relative to what was installed on disk, but
> > that's at most confusing, nothing compared to getting zero-length files
> > all over the place.
> [...]
> > The zero-length problem should affect only new recently installed
> > systems with ext4 anyway.
>
> To throw another variable in the works: doesn't ext4 fall in the same
> category as "other file systems" today? See v2.6.30-rc1~416^2~15
> (ext4: Automatically allocate delay allocated blocks on rename,
> 2009-02-23).

Given <https://bugzilla.kernel.org/show_bug.cgi?id=15910> was filed
against 2.6.32-21-generic, no, I don't think so. Although it could have
been improved somehow with later versions, but then the bug report has
not been updated. Eric Sandeen's comment is relevant in that he states
applications cannot rely on not using fsync() for ext4.

regards,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101107051204.GA28983@gaara.hadrons.org">http://lists.debian.org/20101107051204.GA28983@gaara.hadrons.org
 
Old 11-07-2010, 04:54 AM
Jonathan Nieder
 
Default Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote:

> <https://bugzilla.kernel.org/show_bug.cgi?id=15910>

Ah, sorry for the noise.

I can see why one would be annoyed with Ted's proposed solutions
(e.g., don't crash), yes.

Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101107055447.GA32504@burratino">http://lists.debian.org/20101107055447.GA32504@burratino
 

Thread Tools




All times are GMT. The time now is 07:39 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org