OT: Caching synchronous writes
 

[ Wish there was a generic, active Linux "storage" mailing list out
there -- something other than the kernel lists I mean ]

To frame the discussion, we use VMware ESX (vSphere) quite a bit with
NFS datastores. Often times with NetApp, but lately, more often with
Solaris 10 + ZFS + SSD's for ZIL (intent log or write cache).

The ZIL lets us use synchronous writes (safer) without the normal
delay. Were we to try and get the same level of performance with
Linux, we'd need to use async mode for our NFS shares -- and we'd lose
some reliability.

However, given the latest rumblings and ruminations about Oracle
potentially no longer selling entitlements for Solaris 10 on non-Sun
hardware -- and then turning around and no longer allowing you to run
Solaris 10 "freely", we're left with either OpenSolaris or looking at
Linux again (we run Solaris 10 on Silicon Mechanics hardware).

My question is, what are the various options for getting NFS in "sync"
mode to run fast on Linux?

Obviously we can buy a really nice disk controller with lots of cache,
but I'm thinking more at the filesystem, volume manager or block driver
layer. Is there a way to shunt write requests to a quicker medium like
an SLC-based SSD (or NVRAM)?

I don't see a way to do this with LVM, ext3/ext4 or even xfs... maybe
btrfs will have some options along this line down the road, but that's
tomorrow and not today.

So is a beefy disk controller our best option? Even using our 3Ware
9650's w/ BBU so we can enable write-back doesn't seem to give us as
good of write performance via NFS as ZFS+ZIL-on-SSD does...

Ray
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

On 4/22/2010 3:20 PM, Ray Van Dolson wrote:
> [ Wish there was a generic, active Linux "storage" mailing list out
> there -- something other than the kernel lists I mean ]
>
> To frame the discussion, we use VMware ESX (vSphere) quite a bit with
> NFS datastores. Often times with NetApp, but lately, more often with
> Solaris 10 + ZFS + SSD's for ZIL (intent log or write cache).
>
> The ZIL lets us use synchronous writes (safer) without the normal
> delay. Were we to try and get the same level of performance with
> Linux, we'd need to use async mode for our NFS shares -- and we'd lose
> some reliability.
>
> However, given the latest rumblings and ruminations about Oracle
> potentially no longer selling entitlements for Solaris 10 on non-Sun
> hardware -- and then turning around and no longer allowing you to run
> Solaris 10 "freely", we're left with either OpenSolaris or looking at
> Linux again (we run Solaris 10 on Silicon Mechanics hardware).

Is there some problem with OpenSolaris or NexentaStor?

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

On Thu, Apr 22, 2010 at 03:37:41PM -0500, Les Mikesell wrote:
> On 4/22/2010 3:20 PM, Ray Van Dolson wrote:
> > [ Wish there was a generic, active Linux "storage" mailing list out
> > there -- something other than the kernel lists I mean ]
> >
> > To frame the discussion, we use VMware ESX (vSphere) quite a bit with
> > NFS datastores. Often times with NetApp, but lately, more often with
> > Solaris 10 + ZFS + SSD's for ZIL (intent log or write cache).
> >
> > The ZIL lets us use synchronous writes (safer) without the normal
> > delay. Were we to try and get the same level of performance with
> > Linux, we'd need to use async mode for our NFS shares -- and we'd lose
> > some reliability.
> >
> > However, given the latest rumblings and ruminations about Oracle
> > potentially no longer selling entitlements for Solaris 10 on non-Sun
> > hardware -- and then turning around and no longer allowing you to run
> > Solaris 10 "freely", we're left with either OpenSolaris or looking at
> > Linux again (we run Solaris 10 on Silicon Mechanics hardware).
>
> Is there some problem with OpenSolaris or NexentaStor?
>

Maybe not, but am trying to see what options there are on the Linux
side.

The "delayed allocation" features in ext4 (and xfs, reiser4) sound
interesting. Might give a little performance boost for synchronous
write workloads....

[ We like Nexenta and OpenSolaris just fine, but really like the
stability guarantee Solaris gives us -- much like RHEL. Would rather
not have to worry (as much) about needing to reboot storage boxes and
even though I have confidence in OpenSolaris, it's still more of a
moving / changing target. Not to say we won't ultimately go in that
direction though. ]

Thanks,
Ray
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

Ray Van Dolson wrote:

> The "delayed allocation" features in ext4 (and xfs, reiser4) sound
> interesting. Might give a little performance boost for synchronous
> write workloads....

Doesn't delayed allocation defeat the purpose of a synchronous write?

I think what you want is a proper storage array with mirrored write
cache.

nate


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

On Thu, Apr 22, 2010 at 02:06:47PM -0700, nate wrote:
> Ray Van Dolson wrote:
>
> > The "delayed allocation" features in ext4 (and xfs, reiser4) sound
> > interesting. Might give a little performance boost for synchronous
> > write workloads....
>
> Doesn't delayed allocation defeat the purpose of a synchronous write?

I don't know for sure. From reading, it sounds like as far as data
integrity is concerned it would fall somewhere between complete
write-through synchronous writes and asynchronous writes.

> I think what you want is a proper storage array with mirrored write
> cache.

Which is what we have with ZFS + SSD-based ZIL for far less money than
a NetApp.

This[1] sounds interesting...

Ray

[1] http://lkml.org/lkml/2010/4/5/41
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

Ray Van Dolson wrote:
>> I think what you want is a proper storage array with mirrored write
>> cache.
>>
>
> Which is what we have with ZFS + SSD-based ZIL for far less money than
> a NetApp.
>

not unless you have a pair of them configured as an active/standby HA
cluster, sharing dual port disk storage, and some how (magic?) mirroring
the cache pool so that if the active storage controller/server fails,
the standby can take over wthout losing a single write.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

John R Pierce wrote:
> Ray Van Dolson wrote:
>>> I think what you want is a proper storage array with mirrored write
>>> cache.
>>>
>>
>> Which is what we have with ZFS + SSD-based ZIL for far less money than
>> a NetApp.
>>
>
> not unless you have a pair of them configured as an active/standby HA
> cluster, sharing dual port disk storage, and some how (magic?) mirroring
> the cache pool so that if the active storage controller/server fails,
> the standby can take over wthout losing a single write.
>

OT too but really thought this was a good post/thread on ZFS

http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg18898.html

"ZFS is designed for high *reliability*"
[..]
"You want something completely different. You expect it to deliver
*availability*.

And availability is something ZFS doesn't promise. It simply can't
deliver this."

--


nate


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

On Thu, Apr 22, 2010 at 03:57:01PM -0700, nate wrote:
> John R Pierce wrote:
> > Ray Van Dolson wrote:
> >>> I think what you want is a proper storage array with mirrored write
> >>> cache.
> >>>
> >>
> >> Which is what we have with ZFS + SSD-based ZIL for far less money than
> >> a NetApp.
> >>
> >
> > not unless you have a pair of them configured as an active/standby HA
> > cluster, sharing dual port disk storage, and some how (magic?) mirroring
> > the cache pool so that if the active storage controller/server fails,
> > the standby can take over wthout losing a single write.
> >
>
> OT too but really thought this was a good post/thread on ZFS
>
> http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg18898.html
>
> "ZFS is designed for high *reliability*"
> [..]
> "You want something completely different. You expect it to deliver
> *availability*.
>
> And availability is something ZFS doesn't promise. It simply can't
> deliver this."

Yep... and something you of course know going in.

Don't want to get off on a tangent on that -- am still interested what
type of solutions in the Linux world are out there that can approximate
what an SSD based ZIL does for ZFS.

Kent Overstreet (from lkml) mentioned that his bcache patch is intented
to do something very similar.

So I guess that's my answer -- it's not here yet, so sounds like the
controller is the only way to achieve this currently.

Ray
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

On Thu, Apr 22, 2010 at 03:50:11PM -0700, John R Pierce wrote:
> Ray Van Dolson wrote:
> >> I think what you want is a proper storage array with mirrored write
> >> cache.
> >>
> >
> > Which is what we have with ZFS + SSD-based ZIL for far less money than
> > a NetApp.
> >
>
> not unless you have a pair of them configured as an active/standby HA
> cluster, sharing dual port disk storage, and some how (magic?) mirroring
> the cache pool so that if the active storage controller/server fails,
> the standby can take over wthout losing a single write.

This is definitely tangental to what I was originally asking. :)

I'm not suggesting this perfectly replaces (or even comes close) a
clustered NetApp setup. But it can provide similiar NFS write
performance and I can buy three of them and replicate data for DR needs
for far less than the price of a NetApp SnapMirror setup.

Ray
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

OT: Caching synchronous writes
 

> > Ray Van Dolson wrote:
> > >> I think what you want is a proper storage array with mirrored write
> > >> cache.

When ext3 came into widespread use, a popular method to "cache" frequent fsyncs was to run it in a full data journaling mode, with external journal on a separate disk.
This turned all random writes to a sequential write, limited to a very small piece of disk and a periodical journal flush to the real file system.
This worked amazingly well for busy mail queues - throughput went up 10x and more. People were also reporting improvements in NFS scenarios. Don't know how this is relevant today in times of SSD, but it should be worth to test it.


--

Jure Pečar
http://jure.pecar.org
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos