Is glusterfs ready?
 

Hey,

since RH took control of glusterfs, I've been looking to convert our old independent RAID storage servers to several non RAID glustered ones.

The thing is that I, here and there, heard a few frightening stories from some users (even with latest release).
Any one has experienced with it long enough to think one can blindly trust it or if it is almost there but not yet ready?

Thx,
JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On 28 August 2012 11:14, John Doe <jdmls@yahoo.com> wrote:
> Hey,
>
> since RH took control of glusterfs, I've been looking to convert our old independent RAID storage servers to several non RAID glustered ones.
>
> The thing is that I, here and there, heard a few frightening stories from some users (even with latest release).
> Any one has experienced with it long enough to think one can blindly trust it or if it is almost there but not yet ready

I can't say anything about the RH Storage Appliance, but for us,
gluster up to 3.2.x was most definitely not ready. We went through a
lot of pain, and even after optimizing OS config with help of gluster
support, we were facing insurmountable problems. One of them was
kswapd instances going into overdrive, and once the machine reached a
certain load, all networking functions just stopped. I'm not saying
this is gluster's fault, but even with support we were unable to
configure the machines so that this doesn't happen. That was on CentOS
5.6/x86_64.

Another problem was that due to load and frequent updates (each new
version was supposed to fix bugs; some weren't fixed, and there were
plenty of new ones) the filesystems became inconsistent. In theory,
each file lives on a single brick. The reality was that in the end,
there were many files that existed on all bricks, one copy fully
intact, the others with zero size and funny permissions. You can guess
what happens if you're not aware of this and try to copy/rsync data
off all bricks to different storage. IIRC there were internal changes
that required going through a certain procedure during some upgrades
to ensure filesystem consistency, and these procedures were followed.

We only started out with 3.0.x, and my impression was that development
was focusing on new features rather than bug fixes.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

----- Original Message -----
> From: "John Doe" <jdmls@yahoo.com>
> To: "Cent O Smailinglist" <centos@centos.org>
> Sent: Tuesday, August 28, 2012 3:14:29 AM
> Subject: [CentOS] Is glusterfs ready?
>
> Hey,
>
> since RH took control of glusterfs, I've been looking to convert our
> old independent RAID storage servers to several non RAID glustered
> ones.
>
> The thing is that I, here and there, heard a few frightening stories
> from some users (even with latest release).
> Any one has experienced with it long enough to think one can blindly
> trust it or if it is almost there but not yet ready?
>
> Thx,
> JD


I'm using gluster 3.3.0-1 on two KVM host nodes. I have a 1TB logical volume used as a brick on each node to create a replicated volume. I store my VM's on this volume and have each node mount the gluster volume via localhost using the native FUSE gluster driver. I get about 75-105MB/s over 1Gb Ethernet. Been running this since 3.3 came out. I did quite a bit of failure testing before going live. So far it is working well. I'm only using it as a glorified network RAID1 to make live migration of my VM's fast.

David.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

From: isdtor <isdtor@gmail.com>
> I can't say anything about the RH Storage Appliance, but for us,
> gluster up to 3.2.x was most definitely not ready.
> ...
> We only started out with 3.0.x, and my impression was that development
> was focusing on new features rather than bug fixes.

From: David C. Miller <millerdc@fusion.gat.com>
> I'm using gluster 3.3.0-1 ...
> Been running this since 3.3 came out. I did quite a bit
> of failure testing before going live. So far it is working well.

I read that 3.3 was the first "RH" release.
Let's hope they did/will focus on bug fixing...
So I guess I will wait a little bit more.

Thx to both,
JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On 08/29/2012 04:07 AM, John Doe wrote:
> From: isdtor <isdtor@gmail.com>
>> I can't say anything about the RH Storage Appliance, but for us,
>> gluster up to 3.2.x was most definitely not ready.
>> ...
>> We only started out with 3.0.x, and my impression was that development
>> was focusing on new features rather than bug fixes.
> From: David C. Miller <millerdc@fusion.gat.com>
>> I'm using gluster 3.3.0-1 ...
>> Been running this since 3.3 came out. I did quite a bit
>> of failure testing before going live. So far it is working well.
> I read that 3.3 was the first "RH" release.
> Let's hope they did/will focus on bug fixing...
> So I guess I will wait a little bit more.

We use glusterfs in the CentOS build infrastructure ... and for the most
part it works fairly well.

It is sometimes very slow on file systems with lots of small files ...
especially for operations like find or chmod/chown on a large volume
with lots of small files.

BUT, that said, it is very convenient to use commodity hardware and have
redundant, large, failover volumes on the local network.

We started with version 3.2.5 and now use 3.3.0-3, which is faster than
3.2.5 ... so it should get better in the future.

I can recommend glusterfs as I have not found anything that does what it
does and does it better, but it is challenging and may not be good for
all situations, so test it before you use it.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

From: Johnny Hughes <johnny@centos.org>
> We use glusterfs in the CentOS build infrastructure ... and for the most
> part it works fairly well.
> It is sometimes very slow on file systems with lots of small files ...
> especially for operations like find or chmod/chown on a large volume
> with lots of small files.
> BUT, that said, it is very convenient to use commodity hardware and have
> redundant, large, failover volumes on the local network.
> We started with version 3.2.5 and now use 3.3.0-3, which is faster than
> 3.2.5 ... so it should get better in the future.
> I can recommend glusterfs as I have not found anything that does what it
> does and does it better, but it is challenging and may not be good for
> all situations, so test it before you use it.

I am not too worried about bad performances.
I am afraid to get paged one night because the 50+ TB of the storage
cluster are gone followinf a bug/crash...
It would take days/weeks to set it back up from the backups.
If we were rich, I guess we would have two (or more) "geo-replicated" glusters and
be able to withstand one failing...
I would like the same trust level that I have in RAID.

JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On 08/29/2012 05:16 AM, John Doe wrote:
> From: Johnny Hughes <johnny@centos.org>
>> We use glusterfs in the CentOS build infrastructure ... and for the most
>> part it works fairly well.
>> It is sometimes very slow on file systems with lots of small files ...
>> especially for operations like find or chmod/chown on a large volume
>> with lots of small files.
>> BUT, that said, it is very convenient to use commodity hardware and have
>> redundant, large, failover volumes on the local network.
>> We started with version 3.2.5 and now use 3.3.0-3, which is faster than
>> 3.2.5 ... so it should get better in the future.
>> I can recommend glusterfs as I have not found anything that does what it
>> does and does it better, but it is challenging and may not be good for
>> all situations, so test it before you use it.
> I am not too worried about bad performances.
> I am afraid to get paged one night because the 50+ TB of the storage
> cluster are gone followinf a bug/crash...
> It would take days/weeks to set it back up from the backups.
> If we were rich, I guess we would have two (or more) "geo-replicated" glusters and
> be able to withstand one failing...
> I would like the same trust level that I have in RAID.

I have routinely used DRBD for things like this ... 2 servers, one a
complete failover of the other one. Of course, that requires a 50+ TB
file system on each machine.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On Wed, Aug 29, 2012 at 7:49 AM, Johnny Hughes <johnny@centos.org> wrote:
> > If we were rich, I guess we would have two (or more) "geo-replicated" glusters and
>> be able to withstand one failing...
>> I would like the same trust level that I have in RAID.
>
> I have routinely used DRBD for things like this ... 2 servers, one a
> complete failover of the other one. Of course, that requires a 50+ TB
> file system on each machine.

How well do glusterfs or drbd deal with downtime of one of the
members? Do they catch up quickly with incremental updates and what
kind of impact does that have on performance as it happens? And is
either suitable for running over distances where there is some network
latency?

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On 08/29/12 6:06 AM, Les Mikesell wrote:
> How well do glusterfs or drbd deal with downtime of one of the
> members? Do they catch up quickly with incremental updates and what
> kind of impact does that have on performance as it happens? And is
> either suitable for running over distances where there is some network
> latency?

the extreme case is when one end fails, and you rebuild it,and have to
replicate the whole thing. how long does it take to move 50TB across
your LAN ? how fast can your file system write that much ?






--
john r pierce N 37, W 122
santa cruz ca mid-left coast

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Is glusterfs ready?
 

On 08/29/2012 08:06 AM, Les Mikesell wrote:
> On Wed, Aug 29, 2012 at 7:49 AM, Johnny Hughes <johnny@centos.org> wrote:
>>> If we were rich, I guess we would have two (or more) "geo-replicated" glusters and
>>> be able to withstand one failing...
>>> I would like the same trust level that I have in RAID.
>> I have routinely used DRBD for things like this ... 2 servers, one a
>> complete failover of the other one. Of course, that requires a 50+ TB
>> file system on each machine.
> How well do glusterfs or drbd deal with downtime of one of the
> members? Do they catch up quickly with incremental updates and what
> kind of impact does that have on performance as it happens? And is
> either suitable for running over distances where there is some network
> latency?
>

Well, DRBD is a tried and true solution, but it requires dedicated boxes
and crossover network connections, etc. I would consider it by far the
best method for providing critical failover.

I would consider gluserfs almost a different thing entirely ... it
provides the ability to string several partitions on different machines
into one shared network volume.

Glusterfs does also provide redundancy if you set it up that way ... and
if you have a fast network and enough volumes then the performance is
not very degraded when a gluster volume comes back, etc.

However, I don't think I would trust extremely critical things on
glusterfs at this point.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos