FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Pacman Development

 
 
LinkBack Thread Tools
 
Old 03-14-2009, 04:24 AM
Dan McGee
 
Default libalpm: sync changes to disc when appropriate

On Fri, Mar 13, 2009 at 9:55 PM, Allan McRae <allan@archlinux.org> wrote:
> Dan McGee wrote:
>>
>> We don't make calls to fsync() or fdatasync() when we are attempting to do
>> something transactionally, such as writing out one of our DB entries. Add
>> a
>> call to fdatasync() when we write DB entries, and also ensure we sync our
>> log changes to disc whenever we close it.
>>
>> Another important thing to ensure is that we commit removals of DB
>> entries.
>> The method isn't necessarily pretty, but it works in _alpm_db_remove().
>>
>
> This looks good to me.
> Just wonding how much overhead the fsyncing adds? Have you noticed any speed
> decrease when running with this patch? Not that there is much choice here...

Little if any, actually, as we are syncing only our DB files and not
all of the files we lay down on disk. For the most part pacman is
already I/O bound, so this doesn't add any noticeable hiccups.

-Dan
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://www.archlinux.org/mailman/listinfo/pacman-dev
 
Old 03-14-2009, 11:40 AM
Thomas Bächler
 
Default libalpm: sync changes to disc when appropriate

Dan McGee schrieb:

We don't make calls to fsync() or fdatasync() when we are attempting to do
something transactionally, such as writing out one of our DB entries. Add a
call to fdatasync() when we write DB entries, and also ensure we sync our
log changes to disc whenever we close it.

Another important thing to ensure is that we commit removals of DB entries.
The method isn't necessarily pretty, but it works in _alpm_db_remove().


Can we get this in the maint branch as well? I had this several times
yesterday when experimenting with 2.6.29-rc8 and some experimental intel
driver stuff and it crashed a lot. I eventually had to disable delalloc
on my / to make this go away.



@@ -881,6 +880,15 @@ int _alpm_db_remove(pmdb_t *db, pmpkg_t *info)

ret = _alpm_rmrf(pkgpath);

free(pkgpath);
+ /* by syncing the parent directory, we can be sure the removal is
+ * committed to disk. */
+ fd = open(db->path, 0);
+ if(fd != -1) {
+ fsync(fd);
+ close(fd);
+ } else {
+ ret = -1;
+ }
if(ret != 0) {
ret = -1;
}


If I understood it correctly, we also have to sync the parent directory
when adding new files, not only when removing. Not entirely sure though.


_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://www.archlinux.org/mailman/listinfo/pacman-dev
 
Old 03-14-2009, 12:02 PM
Xavier
 
Default libalpm: sync changes to disc when appropriate

On Sat, Mar 14, 2009 at 1:40 PM, Thomas Bächler <thomas@archlinux.org> wrote:
> Dan McGee schrieb:
>>
>> We don't make calls to fsync() or fdatasync() when we are attempting to do
>> something transactionally, such as writing out one of our DB entries. Add
>> a
>> call to fdatasync() when we write DB entries, and also ensure we sync our
>> log changes to disc whenever we close it.
>>
>> Another important thing to ensure is that we commit removals of DB
>> entries.
>> The method isn't necessarily pretty, but it works in _alpm_db_remove().
>
> Can we get this in the maint branch as well? I had this several times
> yesterday when experimenting with 2.6.29-rc8 and some experimental intel
> driver stuff and it crashed a lot. I eventually had to disable delalloc on
> my / to make this go away.
>

>From http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/
:
"""
What’s the best path forward? For now, what I would recommend to
Ubuntu gamers whose systems crash all the time and who want to use
ext4, to use the nodelalloc mount option. I haven’t quantified what
the performance penalty will be for this mode of operation, but the
performance will be better than ext3, and at least this way they won’t
have to worry about files getting lost as a result of delayed
allocation. Long term, application writers who are worried about
files getting lost on an unclena shutdown really should use fsync.
""""

Short term : use nodelalloc if your system is likely to crash (using
experimental or unstable drivers)
Long term : fix the apps

I don't think that improving pacman on this side ASAP will solve the
general problem. You will still have many other apps affected on your
system, some of them also dealing with important files. Or not?

Anyway, I am not opposed to a new quick maint release, this decision
is up to Dan anyway. I am just saying I don't see a big need for it.
And in any cases, we should try to move forward for a 3.3 release
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://www.archlinux.org/mailman/listinfo/pacman-dev
 
Old 03-14-2009, 12:32 PM
Dan McGee
 
Default libalpm: sync changes to disc when appropriate

On Sat, Mar 14, 2009 at 8:02 AM, Xavier <shiningxc@gmail.com> wrote:
> On Sat, Mar 14, 2009 at 1:40 PM, Thomas Bächler <thomas@archlinux.org> wrote:
>> Dan McGee schrieb:
>>>
>>> We don't make calls to fsync() or fdatasync() when we are attempting to do
>>> something transactionally, such as writing out one of our DB entries. Add
>>> a
>>> call to fdatasync() when we write DB entries, and also ensure we sync our
>>> log changes to disc whenever we close it.
>>>
>>> Another important thing to ensure is that we commit removals of DB
>>> entries.
>>> The method isn't necessarily pretty, but it works in _alpm_db_remove().
>>
>> Can we get this in the maint branch as well? I had this several times
>> yesterday when experimenting with 2.6.29-rc8 and some experimental intel
>> driver stuff and it crashed a lot. I eventually had to disable delalloc on
>> my / to make this go away.
>>
>
> >From http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/
> :
> """
> What’s the best path forward? Â* For now, what I would recommend to
> Ubuntu gamers whose systems crash all the time and who want to use
> ext4, to use the nodelalloc mount option. Â* I haven’t quantified what
> the performance penalty will be for this mode of operation, but the
> performance will be better than ext3, and at least this way they won’t
> have to worry about files getting lost as a result of delayed
> allocation. Â* Â*Long term, application writers who are worried about
> files getting lost on an unclena shutdown really should use fsync.
> """"
>
> Short term : use nodelalloc if your system is likely to crash (using
> experimental or unstable drivers)
> Long term : fix the apps
>
> I don't think that improving pacman on this side ASAP will solve the
> general problem. You will still have many other apps affected on your
> system, some of them also dealing with important files. Or not?
>
> Anyway, I am not opposed to a new quick maint release, this decision
> is up to Dan anyway. I am just saying I don't see a big need for it.
> And in any cases, we should try to move forward for a 3.3 release

I never said quick with this stuff at all. Does your system crash all
the time? Or ever? Mine sure doesn't- I'm in little rush to get this
out.

Thomas points to one case where his system does, but in that case he
is going to have a lot of broken stuff as he says as other apps also
do it wrong as well. My main point here is just to do it right- I'm
not trying to address a specific problem, just the general concern
that is "if I write a file and my app exits, can I be sure it actually
got written?".

Thomas- I believe you are right, we should sync the db directory when
we write to it as well.

-Dan
_______________________________________________
pacman-dev mailing list
pacman-dev@archlinux.org
http://www.archlinux.org/mailman/listinfo/pacman-dev
 

Thread Tools




All times are GMT. The time now is 05:28 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org