FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 04-23-2010, 01:52 PM
Chan Chung Hang Christopher
 
Default OT: Caching synchronous writes

Jure Pečar wrote:
>>> Ray Van Dolson wrote:
>>>>> I think what you want is a proper storage array with mirrored write
>>>>> cache.
>
> When ext3 came into widespread use, a popular method to "cache" frequent fsyncs was to run it in a full data journaling mode, with external journal on a separate disk.
> This turned all random writes to a sequential write, limited to a very small piece of disk and a periodical journal flush to the real file system.
> This worked amazingly well for busy mail queues - throughput went up 10x and more. People were also reporting improvements in NFS scenarios. Don't know how this is relevant today in times of SSD, but it should be worth to test it.
>
>

separate disk only? Don't forget nvram sticks or bbu ramdrives.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-23-2010, 04:17 PM
Ray Van Dolson
 
Default OT: Caching synchronous writes

On Fri, Apr 23, 2010 at 10:20:01AM +0200, Jure Pečar wrote:
>
> > > Ray Van Dolson wrote:
> > > >> I think what you want is a proper storage array with mirrored write
> > > >> cache.
>
> When ext3 came into widespread use, a popular method to "cache"
> frequent fsyncs was to run it in a full data journaling mode, with
> external journal on a separate disk. This turned all random writes
> to a sequential write, limited to a very small piece of disk and a
> periodical journal flush to the real file system. This worked
> amazingly well for busy mail queues - throughput went up 10x and
> more. People were also reporting improvements in NFS scenarios. Don't
> know how this is relevant today in times of SSD, but it should be
> worth to test it.

Interesting. As long as the requirements of O_SYNC are met once the
data is written to the journal (I imagine it would be), then I could
definitely see this speeding up NFS...

On the other hand, if no write confirmation is sent until the data
actually flushes out of the journal and onto disk, then the wins
probably aren't as significant.

Sounds like it'd be worth trying though, thanks.

Ray
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-23-2010, 04:51 PM
Les Mikesell
 
Default OT: Caching synchronous writes

On 4/23/2010 11:17 AM, Ray Van Dolson wrote:
> On Fri, Apr 23, 2010 at 10:20:01AM +0200, Jure Pečar wrote:
>>
>>>> Ray Van Dolson wrote:
>>>>>> I think what you want is a proper storage array with mirrored write
>>>>>> cache.
>>
>> When ext3 came into widespread use, a popular method to "cache"
>> frequent fsyncs was to run it in a full data journaling mode, with
>> external journal on a separate disk. This turned all random writes
>> to a sequential write, limited to a very small piece of disk and a
>> periodical journal flush to the real file system. This worked
>> amazingly well for busy mail queues - throughput went up 10x and
>> more. People were also reporting improvements in NFS scenarios. Don't
>> know how this is relevant today in times of SSD, but it should be
>> worth to test it.
>
> Interesting. As long as the requirements of O_SYNC are met once the
> data is written to the journal (I imagine it would be), then I could
> definitely see this speeding up NFS...
>
> On the other hand, if no write confirmation is sent until the data
> actually flushes out of the journal and onto disk, then the wins
> probably aren't as significant.

Do any linux filesystems actually get this right now? In the past, the
filesystem cache was somewhat divorced from file writes so fsync() and
probably any write with O_SYNC would wait until the entire filesystem
cache was flushed to disk, not just the related file buffer.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 03:57 PM
Ross Walker
 
Default OT: Caching synchronous writes

On Apr 22, 2010, at 8:08 PM, Ray Van Dolson <rayvd@bludgeon.org> wrote:

> On Thu, Apr 22, 2010 at 03:57:01PM -0700, nate wrote:
>> John R Pierce wrote:
>>> Ray Van Dolson wrote:
>>>>> I think what you want is a proper storage array with mirrored
>>>>> write
>>>>> cache.
>>>>>
>>>>
>>>> Which is what we have with ZFS + SSD-based ZIL for far less money
>>>> than
>>>> a NetApp.
>>>>
>>>
>>> not unless you have a pair of them configured as an active/standby
>>> HA
>>> cluster, sharing dual port disk storage, and some how (magic?)
>>> mirroring
>>> the cache pool so that if the active storage controller/server
>>> fails,
>>> the standby can take over wthout losing a single write.
>>>
>>
>> OT too but really thought this was a good post/thread on ZFS
>>
>> http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg18898.html
>>
>> "ZFS is designed for high *reliability*"
>> [..]
>> "You want something completely different. You expect it to deliver
>> *availability*.
>>
>> And availability is something ZFS doesn't promise. It simply can't
>> deliver this."
>
> Yep... and something you of course know going in.
>
> Don't want to get off on a tangent on that -- am still interested what
> type of solutions in the Linux world are out there that can
> approximate
> what an SSD based ZIL does for ZFS.
>
> Kent Overstreet (from lkml) mentioned that his bcache patch is
> intented
> to do something very similar.
>
> So I guess that's my answer -- it's not here yet, so sounds like the
> controller is the only way to achieve this currently.

How about locating XFS journal on SSDs and using HW RAID controller
with big NVRAM cache.

That should be a lot faster than ZFS with SSD ZIL.

NFS should always be 'sync' if performance isn't good, then your
storage isn't good.

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 04:43 PM
Les Mikesell
 
Default OT: Caching synchronous writes

Ross Walker wrote:
>
> NFS should always be 'sync' if performance isn't good, then your
> storage isn't good.

Why demand sync on remote storage when you typically don't have it locally?
Programs that need transactional integrity should know when to fsync() and for
anything else there's not much difference whether you crash before or after a
write() was issued in terms of it not completing.

--
Les Mikesesll
lesmikesell@gmail.com

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 08:09 PM
Ross Walker
 
Default OT: Caching synchronous writes

On Apr 24, 2010, at 12:43 PM, Les Mikesell <lesmikesell@gmail.com>
wrote:

> Ross Walker wrote:
>>
>> NFS should always be 'sync' if performance isn't good, then your
>> storage isn't good.
>
> Why demand sync on remote storage when you typically don't have it
> locally?
> Programs that need transactional integrity should know when to fsync
> () and for
> anything else there's not much difference whether you crash before
> or after a
> write() was issued in terms of it not completing.

Yes, but 'async' ignores those fsyncs and returns immediately.

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 08:34 PM
Les Mikesell
 
Default OT: Caching synchronous writes

Ross Walker wrote:
> On Apr 24, 2010, at 12:43 PM, Les Mikesell <lesmikesell@gmail.com>
> wrote:
>
>> Ross Walker wrote:
>>> NFS should always be 'sync' if performance isn't good, then your
>>> storage isn't good.
>> Why demand sync on remote storage when you typically don't have it
>> locally?
>> Programs that need transactional integrity should know when to fsync
>> () and for
>> anything else there's not much difference whether you crash before
>> or after a
>> write() was issued in terms of it not completing.
>
> Yes, but 'async' ignores those fsyncs and returns immediately.

That sounds like a bug in the nfs client code if fsync() doesn't block until all
of the data is committed to disk.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 08:41 PM
Ross Walker
 
Default OT: Caching synchronous writes

On Apr 24, 2010, at 4:34 PM, Les Mikesell <lesmikesell@gmail.com> wrote:

> Ross Walker wrote:
>> On Apr 24, 2010, at 12:43 PM, Les Mikesell <lesmikesell@gmail.com>
>> wrote:
>>
>>> Ross Walker wrote:
>>>> NFS should always be 'sync' if performance isn't good, then your
>>>> storage isn't good.
>>> Why demand sync on remote storage when you typically don't have it
>>> locally?
>>> Programs that need transactional integrity should know when to fsync
>>> () and for
>>> anything else there's not much difference whether you crash before
>>> or after a
>>> write() was issued in terms of it not completing.
>>
>> Yes, but 'async' ignores those fsyncs and returns immediately.
>
> That sounds like a bug in the nfs client code if fsync() doesn't
> block until all
> of the data is committed to disk.

It's not the client side I'm talking about, but the server side. We
were talking NFS servers and exporting sync (obey fsyncs) vs async
(ignore fsyncs).

The client always mounts async, that's not the problem.

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 08:53 PM
Les Mikesell
 
Default OT: Caching synchronous writes

Ross Walker wrote:
> On Apr 24, 2010, at 4:34 PM, Les Mikesell <lesmikesell@gmail.com> wrote:
>
>> Ross Walker wrote:
>>> On Apr 24, 2010, at 12:43 PM, Les Mikesell <lesmikesell@gmail.com>
>>> wrote:
>>>
>>>> Ross Walker wrote:
>>>>> NFS should always be 'sync' if performance isn't good, then your
>>>>> storage isn't good.
>>>> Why demand sync on remote storage when you typically don't have it
>>>> locally?
>>>> Programs that need transactional integrity should know when to fsync
>>>> () and for
>>>> anything else there's not much difference whether you crash before
>>>> or after a
>>>> write() was issued in terms of it not completing.
>>> Yes, but 'async' ignores those fsyncs and returns immediately.
>> That sounds like a bug in the nfs client code if fsync() doesn't
>> block until all
>> of the data is committed to disk.
>
> It's not the client side I'm talking about, but the server side. We
> were talking NFS servers and exporting sync (obey fsyncs) vs async
> (ignore fsyncs).
>
> The client always mounts async, that's not the problem.

That's different. I thought the nfs spec was always sync on the server side and
the client says when async is OK. And there's some special case response to
handle the case where the server rebooted between the async writes and the
subsequent fsync().

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-24-2010, 09:17 PM
Ross Walker
 
Default OT: Caching synchronous writes

On Apr 24, 2010, at 4:53 PM, Les Mikesell <lesmikesell@gmail.com> wrote:

> Ross Walker wrote:
>> On Apr 24, 2010, at 4:34 PM, Les Mikesell <lesmikesell@gmail.com>
>> wrote:
>>
>>> Ross Walker wrote:
>>>> On Apr 24, 2010, at 12:43 PM, Les Mikesell <lesmikesell@gmail.com>
>>>> wrote:
>>>>
>>>>> Ross Walker wrote:
>>>>>> NFS should always be 'sync' if performance isn't good, then your
>>>>>> storage isn't good.
>>>>> Why demand sync on remote storage when you typically don't have it
>>>>> locally?
>>>>> Programs that need transactional integrity should know when to
>>>>> fsync
>>>>> () and for
>>>>> anything else there's not much difference whether you crash before
>>>>> or after a
>>>>> write() was issued in terms of it not completing.
>>>> Yes, but 'async' ignores those fsyncs and returns immediately.
>>> That sounds like a bug in the nfs client code if fsync() doesn't
>>> block until all
>>> of the data is committed to disk.
>>
>> It's not the client side I'm talking about, but the server side. We
>> were talking NFS servers and exporting sync (obey fsyncs) vs async
>> (ignore fsyncs).
>>
>> The client always mounts async, that's not the problem.
>
> That's different. I thought the nfs spec was always sync on the
> server side and
> the client says when async is OK. And there's some special case
> response to
> handle the case where the server rebooted between the async writes
> and the
> subsequent fsync().

All the NFS info you wanted, but were afraid to ask:

http://nfs.sourceforge.net/

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 04:59 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org