FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 10-22-2010, 09:35 AM
Guillem Jover
 
Default Pre-approval request for dpkg sync() changes for squeeze

Hi!

I'd like to apply two patches for the next dpkg 1.15.8.6 upload. And
would like to run them through you to avoid having to possibly revert
them from dpkg's sid git branch.

Those are related to the fsync()/sync() changes in dpkg from some time
ago, the patches would:

1) Switch back from sync() to fsync() before rename() (while keeping
the sync() code around for the benefit of other distributions
that might not want to switch just yet). So to avoid unrelated
I/O when there's background work being done for example. This
hack also only works on Linux where sync() is synchronous, so
it would unify that code path for all dpkg supported platforms.

Bug: #588339

<http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=87740373>

2) Add a new --force-unsafe-io to disable those fsync() for people
using certain file systems where the only option is to choose
between reliability or acceptable performance. Or in cases where
the data is cached and it might not be important to lose it.

Bug: #584254

<http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=commitdiff;h=0484cd7d>

I'm planning to apply those for the 1.16.x branch anyway, and I think
would be important to have the same behaviour on the version that will
go into stable, more so if those problematic file systems are to
become more widely used.

I'll also be sending a summary of what's to come to debian-devel once
those patches land on any of the dpkg git branches.

thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101022093554.GA14617@gaara.hadrons.org">http://lists.debian.org/20101022093554.GA14617@gaara.hadrons.org
 
Old 10-22-2010, 04:10 PM
Jonathan Nieder
 
Default Pre-approval request for dpkg sync() changes for squeeze

Guillem Jover wrote:

> 1) Switch back from sync() to fsync() before rename() (while keeping
> the sync() code around for the benefit of other distributions
> that might not want to switch just yet). So to avoid unrelated
> I/O when there's background work being done for example. This
> hack also only works on Linux where sync() is synchronous, so
> it would unify that code path for all dpkg supported platforms.
>
> Bug: #588339

Will there be a NEWS.Debian.gz entry for this? I fear that the
performance degradation would be unacceptable to some people (using
ext4, for example).

> 2) Add a new --force-unsafe-io to disable those fsync() for people
> using certain file systems where the only option is to choose
> between reliability or acceptable performance.

dpkg uses the "rename trick" in all the cases this applies to, right?
If so, a certain degree of reliability is guaranteed in precisely
those file systems where fsync() is slow.

What risks remain with --force-unsafe-io on those filesystems?

Context: a good part of ext3's popularity comes from its combination
of speed and resiliance to crashing. To sacrifice one of the two is
sacrificing something dear to some people.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101022161043.GF9224@burratino">http://lists.debian.org/20101022161043.GF9224@burratino
 
Old 10-22-2010, 08:11 PM
Phillip Susi
 
Default Pre-approval request for dpkg sync() changes for squeeze

On 10/22/2010 5:35 AM, Guillem Jover wrote:
> 1) Switch back from sync() to fsync() before rename() (while keeping

Don't you WANT to use sync? If you fsync every file that is going to be
rather slow since it forces a disk write for every file, rather than
allowing writes between each file to be combined and batched. Ideally
you want to unpack all files from all packages being installed/upgraded,
then issue one big sync(), and finally all of the rename()s, with one
final sync(), don't you?

To throw out some numbers, it seems like it can easily take 60ms or more
for an fsync(), between writing the file data, writing the journal,
writing inode, writing the directory entry, then finally the journal
again. If you are replacing 1000 files then you are spending 60 seconds
just on the fsync calls. If you only use a single sync, then the whole
thing could easily write all of the file data, adjacent inodes, journal
commits, etc in 1-2 seconds with only 2-3 seeks.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4CC1F004.10907@cfl.rr.com">http://lists.debian.org/4CC1F004.10907@cfl.rr.com
 
Old 10-24-2010, 11:16 AM
Goswin von Brederlow
 
Default Pre-approval request for dpkg sync() changes for squeeze

Phillip Susi <psusi@cfl.rr.com> writes:

> On 10/22/2010 5:35 AM, Guillem Jover wrote:
>> 1) Switch back from sync() to fsync() before rename() (while keeping
>
> Don't you WANT to use sync? If you fsync every file that is going to be
> rather slow since it forces a disk write for every file, rather than
> allowing writes between each file to be combined and batched. Ideally
> you want to unpack all files from all packages being installed/upgraded,
> then issue one big sync(), and finally all of the rename()s, with one
> final sync(), don't you?
>
> To throw out some numbers, it seems like it can easily take 60ms or more
> for an fsync(), between writing the file data, writing the journal,
> writing inode, writing the directory entry, then finally the journal
> again. If you are replacing 1000 files then you are spending 60 seconds
> just on the fsync calls. If you only use a single sync, then the whole
> thing could easily write all of the file data, adjacent inodes, journal
> commits, etc in 1-2 seconds with only 2-3 seeks.

Or 5 minutes because sync() also needs to write out the 16GB cache data
to my usb 1.0 drive that is not involved with dpkg at all.

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 87eibfet8q.fsf@frosties.localdomain">http://lists.debian.org/87eibfet8q.fsf@frosties.localdomain
 
Old 10-24-2010, 04:57 PM
Phillip Susi
 
Default Pre-approval request for dpkg sync() changes for squeeze

On 10/24/2010 07:16 AM, Goswin von Brederlow wrote:

Or 5 minutes because sync() also needs to write out the 16GB cache data
to my usb 1.0 drive that is not involved with dpkg at all.


True, but that seems a bit of a contrived corner case. Most of the time
when people are upgrading, I'd wager that they don't have many gb of
dirty cache buffers already sitting in the cache. Though that does make
me wonder why there isn't a sync() variant that applies only to a
specific mount point. That would at least keep other unrelated mounts
out of the picture.


If you are going to use fsync() to make sure you don't flush any
unrelated data, then maybe you should at least use aio and issue all of
the fsync calls overlapped at once, so you can hopefully have the writes
combined into fewer, larger writes, with fewer seeks between them.


Then again, you could probably do better by using O_DIRECT when writing
the file in the first place and avoid having to latter flush at all.



--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4CC46592.9000903@cfl.rr.com">http://lists.debian.org/4CC46592.9000903@cfl.rr.com
 
Old 10-24-2010, 05:20 PM
Andreas Barth
 
Default Pre-approval request for dpkg sync() changes for squeeze

* Phillip Susi (psusi@cfl.rr.com) [101024 19:15]:
> True, but that seems a bit of a contrived corner case. Most of the time
> when people are upgrading, I'd wager that they don't have many gb of
> dirty cache buffers already sitting in the cache. Though that does make
> me wonder why there isn't a sync() variant that applies only to a
> specific mount point. That would at least keep other unrelated mounts
> out of the picture.

I quite often run dpkg installs / upgrades in pbuilder ramdisks. If
the sync could be reduced to only that ramdisk, everything would be
fine.


Andi


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101024172016.GQ15003@mails.so.argh.org">http://lists.debian.org/20101024172016.GQ15003@mails.so.argh.org
 
Old 10-25-2010, 08:28 AM
Goswin von Brederlow
 
Default Pre-approval request for dpkg sync() changes for squeeze

Phillip Susi <psusi@cfl.rr.com> writes:

> On 10/24/2010 07:16 AM, Goswin von Brederlow wrote:
>> Or 5 minutes because sync() also needs to write out the 16GB cache data
>> to my usb 1.0 drive that is not involved with dpkg at all.
>
> True, but that seems a bit of a contrived corner case. Most of the
> time when people are upgrading, I'd wager that they don't have many gb
> of dirty cache buffers already sitting in the cache. Though that does
> make me wonder why there isn't a sync() variant that applies only to a
> specific mount point. That would at least keep other unrelated mounts
> out of the picture.
>
> If you are going to use fsync() to make sure you don't flush any
> unrelated data, then maybe you should at least use aio and issue all
> of the fsync calls overlapped at once, so you can hopefully have the
> writes combined into fewer, larger writes, with fewer seeks between
> them.
>
> Then again, you could probably do better by using O_DIRECT when
> writing the file in the first place and avoid having to latter flush
> at all.

Except that the fsync() patches from RedHat were never added to the AIO
code in the kernel. The portable way for this is to mmap() the files and
msync() them with MS_ASYNC. But that makes catching write errors a
nightmare.

MfG
Goswin


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 874ocallqc.fsf@frosties.localdomain">http://lists.debian.org/874ocallqc.fsf@frosties.localdomain
 
Old 10-25-2010, 02:45 PM
Phillip Susi
 
Default Pre-approval request for dpkg sync() changes for squeeze

On 10/24/2010 1:20 PM, Andreas Barth wrote:
> I quite often run dpkg installs / upgrades in pbuilder ramdisks. If
> the sync could be reduced to only that ramdisk, everything would be
> fine.

That's exactly the environment where you would disable syncing entirely
since you don't care if the pbuilder chroot becomes corrupted if the
system crashes; you will just rebuild it from the tarball anyhow.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4CC597F5.7040705@cfl.rr.com">http://lists.debian.org/4CC597F5.7040705@cfl.rr.com
 
Old 10-25-2010, 03:02 PM
Phillip Susi
 
Default Pre-approval request for dpkg sync() changes for squeeze

On 10/25/2010 4:28 AM, Goswin von Brederlow wrote:
> Except that the fsync() patches from RedHat were never added to the AIO
> code in the kernel. The portable way for this is to mmap() the files and
> msync() them with MS_ASYNC. But that makes catching write errors a
> nightmare.

Blast. Then I guess O_DIRECT aio is the way to go. Either that or a
kernel interface change is needed to convey the right caching policy.
Seems like the latter would be the better choice, but also more
difficult to get off the ground.


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4CC59C1B.2060702@cfl.rr.com">http://lists.debian.org/4CC59C1B.2060702@cfl.rr.com
 
Old 10-26-2010, 06:10 AM
Andreas Barth
 
Default Pre-approval request for dpkg sync() changes for squeeze

* Phillip Susi (psusi@cfl.rr.com) [101025 17:00]:
> On 10/24/2010 1:20 PM, Andreas Barth wrote:
> > I quite often run dpkg installs / upgrades in pbuilder ramdisks. If
> > the sync could be reduced to only that ramdisk, everything would be
> > fine.
>
> That's exactly the environment where you would disable syncing entirely
> since you don't care if the pbuilder chroot becomes corrupted if the
> system crashes; you will just rebuild it from the tarball anyhow.

So, what is the appropriate configuration setting for dpkg, and where
are the bug reports for our usual tools to auto-set it correct? If
there is none, this argument is moot.


Andi


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20101026061013.GR15003@mails.so.argh.org">http://lists.debian.org/20101026061013.GR15003@mails.so.argh.org
 

Thread Tools




All times are GMT. The time now is 10:57 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org