FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 08-29-2012, 02:09 PM
Dennis Jacobfeuerborn
 
Default Is glusterfs ready?

On 08/29/2012 03:17 PM, Johnny Hughes wrote:
> On 08/29/2012 08:06 AM, Les Mikesell wrote:
>> On Wed, Aug 29, 2012 at 7:49 AM, Johnny Hughes <johnny@centos.org> wrote:
>>>> If we were rich, I guess we would have two (or more) "geo-replicated" glusters and
>>>> be able to withstand one failing...
>>>> I would like the same trust level that I have in RAID.
>>> I have routinely used DRBD for things like this ... 2 servers, one a
>>> complete failover of the other one. Of course, that requires a 50+ TB
>>> file system on each machine.
>> How well do glusterfs or drbd deal with downtime of one of the
>> members? Do they catch up quickly with incremental updates and what
>> kind of impact does that have on performance as it happens? And is
>> either suitable for running over distances where there is some network
>> latency?
>>
>
> Well, DRBD is a tried and true solution, but it requires dedicated boxes
> and crossover network connections, etc. I would consider it by far the
> best method for providing critical failover.
>
> I would consider gluserfs almost a different thing entirely ... it
> provides the ability to string several partitions on different machines
> into one shared network volume.
>
> Glusterfs does also provide redundancy if you set it up that way ... and
> if you have a fast network and enough volumes then the performance is
> not very degraded when a gluster volume comes back, etc.
>
> However, I don't think I would trust extremely critical things on
> glusterfs at this point.

I think the keyword with solutions like glusterfs, ceph, sheepdog, etc. is
"elasticity". DRBD and RAID work well as long as you have a fixed size of
data to deal with but once you get to a consistent data growth you need
something that offers redundancy yet can be easily extended incrementally.

Glusterfs seems to aim to be a solution that works well right now because
it uses a simple file replication approach whereas ceph and sheepdog seem
to go deeper and provide better architectures but will take longer to mature.

Regards,
Dennis
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 08-29-2012, 03:50 PM
Rajagopal Swaminathan
 
Default Is glusterfs ready?

Greetings,

On Wed, Aug 29, 2012 at 7:39 PM, Dennis Jacobfeuerborn
<dennisml@conversis.de> wrote:


Where does openAFS stands in all these deleberations?
http://www.openafs.org/

--
Regards,

Rajagopal
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 08-29-2012, 03:52 PM
Rajagopal Swaminathan
 
Default Is glusterfs ready?

On Wed, Aug 29, 2012 at 9:20 PM, Rajagopal Swaminathan
<raju.rajsand@gmail.com> wrote:
> Greetings,
>
> On Wed, Aug 29, 2012 at 7:39 PM, Dennis Jacobfeuerborn
> <dennisml@conversis.de> wrote:
>
>
> Where does openAFS stands in all these deleberations?
> http://www.openafs.org/
>

oops, missed out this:
http://www.stacken.kth.se/project/arla/

--
Regards,

Rajagopal
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 08-29-2012, 04:06 PM
Les Mikesell
 
Default Is glusterfs ready?

On Wed, Aug 29, 2012 at 10:52 AM, Rajagopal Swaminathan
<raju.rajsand@gmail.com> wrote:
> >
>>
>> Where does openAFS stands in all these deleberations?
>> http://www.openafs.org/
>>
>
> oops, missed out this:
> http://www.stacken.kth.se/project/arla/
>

AFS isn't what you expect from a distributed file system. Each
machine works with cached copies of whole files and when one of them
writes and closes a file the others are notified to update their copy.
Last write wins.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-05-2012, 05:14 AM
Bob Hepple
 
Default Is glusterfs ready?

David C. Miller <millerdc@...> writes:

>
>
> ----- Original Message -----
> > From: "John Doe" <jdmls@...>
> > To: "Cent O Smailinglist" <centos@...>
> > Sent: Tuesday, August 28, 2012 3:14:29 AM
> > Subject: [CentOS] Is glusterfs ready?
> >
> > Hey,
> >
> > since RH took control of glusterfs, I've been looking to convert our
> > old independent RAID storage servers to several non RAID glustered
> > ones.
> >
> > The thing is that I, here and there, heard a few frightening stories
> > from some users (even with latest release).
> > Any one has experienced with it long enough to think one can blindly
> > trust it or if it is almost there but not yet ready?
> >

Heya,

Well I guess I'm one of the frightening stories, or at least a
previous employer was. They had a mere 0.1 petabyte store over 6
bricks yet they had incredible performance and reliability
difficulties. I'm talking about a mission critical system being
unavailable for weeks at a time. At least it wasn't customer
facing (there was another set of servers for that).

The system was down more than it was up. Reading was generally
OK (but very slow) but multiple threads writing caused mayhem -
I'm talking lost files and file system accesses going into the
multiple minutes.

In the end I implemented a 1-Tb store to be fuse-unioned over the top
of the thing to take the impact of multiple threads writing to it. A
single thread (overnight) brought the underlying glusterfs up to date.

That got us more or less running but the darned thing spent most of
its time re-indexing and balancing rather than serving files.

To be fair, some of the problems were undoubtedly of their own making
as 2 nodes were centos and 4 were fedora-12 - apparently the engineer
couldn't find the installation CD for the 2 new nodes and 'made do'
with what he had! I recall that a difference in the system 'sort'
command gave all sorts of grief until it was discovered, never mind
different versions of the gluster drivers.

I'd endorse Johnny's comments about it not handling large numbers of
small files well (ie <~ 10 Mb). I believe it was designed for large
multi-media files such as clinical X-Rays. ie a small number of large
files.

Another factor is that the available space is the physical space
divided by 4 due to the replication across the nodes on top of the
nodes being RAID'd themselves.

Lesse now - that was all of 6 months ago - unlike most of my war
stories, it's not ancient history!!

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-05-2012, 03:07 PM
Dennis Jacobfeuerborn
 
Default Is glusterfs ready?

On 09/05/2012 07:14 AM, Bob Hepple wrote:
> David C. Miller <millerdc@...> writes:
>
>>
>>
>> ----- Original Message -----
>>> From: "John Doe" <jdmls@...>
>>> To: "Cent O Smailinglist" <centos@...>
>>> Sent: Tuesday, August 28, 2012 3:14:29 AM
>>> Subject: [CentOS] Is glusterfs ready?
>>>
>>> Hey,
>>>
>>> since RH took control of glusterfs, I've been looking to convert our
>>> old independent RAID storage servers to several non RAID glustered
>>> ones.
>>>
>>> The thing is that I, here and there, heard a few frightening stories
>>> from some users (even with latest release).
>>> Any one has experienced with it long enough to think one can blindly
>>> trust it or if it is almost there but not yet ready?
>>>
>
> Heya,
>
> Well I guess I'm one of the frightening stories, or at least a
> previous employer was. They had a mere 0.1 petabyte store over 6
> bricks yet they had incredible performance and reliability
> difficulties. I'm talking about a mission critical system being
> unavailable for weeks at a time. At least it wasn't customer
> facing (there was another set of servers for that).
>
> The system was down more than it was up. Reading was generally
> OK (but very slow) but multiple threads writing caused mayhem -
> I'm talking lost files and file system accesses going into the
> multiple minutes.
>
> In the end I implemented a 1-Tb store to be fuse-unioned over the top
> of the thing to take the impact of multiple threads writing to it. A
> single thread (overnight) brought the underlying glusterfs up to date.
>
> That got us more or less running but the darned thing spent most of
> its time re-indexing and balancing rather than serving files.
>
> To be fair, some of the problems were undoubtedly of their own making
> as 2 nodes were centos and 4 were fedora-12 - apparently the engineer
> couldn't find the installation CD for the 2 new nodes and 'made do'
> with what he had! I recall that a difference in the system 'sort'
> command gave all sorts of grief until it was discovered, never mind
> different versions of the gluster drivers.

That is the problem with most of these stories in that the setups tend to
be of the "adventurous" kind. Not only was the setup very asymmetrical but
Fedora 12 was long outdated even 6 months ago.
This kind of setup should be categorized as "highly experimental" and not
something you actually use in production.

> I'd endorse Johnny's comments about it not handling large numbers of
> small files well (ie <~ 10 Mb). I believe it was designed for large
> multi-media files such as clinical X-Rays. ie a small number of large
> files.

That's a problem with all distributed filesystems. For a few large files
the additional time needed for round-trips is usually dwarfed by the actual
I/O requests themselves so you don't notice it (much). With a ton of small
files you incur lots of metadata fetching round-trips for every few kbyte
read/written which slows things down by a great deal.
So basically if you want top performance for lot of small files don't use
distributed filesystems.

> Another factor is that the available space is the physical space
> divided by 4 due to the replication across the nodes on top of the
> nodes being RAID'd themselves.

That really depends on your setup. I'm not sure what you mean by the nodes
being raided themselves. If you run a four node cluster and keep two copies
of each file you would probably create two pairs of nodes where one node is
replicated to the other and then create a stripe over these two pairs which
should actually improve performance. This would mean your available space
would be cut in half and not be divided by 4.

Regards,
Dennis

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-05-2012, 04:00 PM
John Doe
 
Default Is glusterfs ready?

From: Dennis Jacobfeuerborn <dennisml@conversis.de>

>On 09/05/2012 07:14 AM, Bob Hepple wrote:
>> Another factor is that the available space is the physical space
>> divided by 4 due to the replication across the nodes on top of the
>> nodes being RAID'd themselves.
>That really depends on your setup. I'm not sure what you mean by the nodes
>being raided themselves.
I think he meant gluster "RAID1" plus hardware RAID (10 I guess from the x2, instead of standalone disks).

JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-11-2012, 03:45 PM
"Jose P. Espinal"
 
Default Is glusterfs ready?

On Wed, Sep 5, 2012 at 12:00 PM, John Doe <jdmls@yahoo.com> wrote:

> From: Dennis Jacobfeuerborn <dennisml@conversis.de>
>
> >On 09/05/2012 07:14 AM, Bob Hepple wrote:
> >> Another factor is that the available space is the physical space
> >> divided by 4 due to the replication across the nodes on top of the
> >> nodes being RAID'd themselves.
> >That really depends on your setup. I'm not sure what you mean by the nodes
> >being raided themselves.
> I think he meant gluster "RAID1" plus hardware RAID (10 I guess from the
> x2, instead of standalone disks).
>
> JD
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>


Hello, this comment was posted on a site I administer, where
I chronologically publish an archive of some CentOS (and some other
distros) lists:

===== [comment] =====
A new comment on the post "Is Glusterfs Ready?"

Author : Jeff Darcy (IP: 173.48.139.36 ,
pool-173-48-139-36.bstnma.fios.verizon.net)
E-mail : jeff@pl.atyp.us
URL : http://pl.atyp.us
Whois : http://whois.arin.net/rest/ip/173.48.139.36
Comment:

Hi. I'm one of the GlusterFS developers, and I'll try to offer a slightly
different perspective.

First, sure GlusterFS has bugs. Some of them even make me cringe. If we
really wanted to get into a discussion of the things about GlusterFS that
suck, I'd probably be able to come up with more things than anybody, but
one of the lessons I learned early in my career is that seeing all of the
bugs for a piece of software leads to a skewed perspective. Some people
have had problems with GlusterFS but some people have been very happy with
it, and I guarantee that every alternative has its own horror stories.
GlusterFS and XtreemFS were the only two distributed filesystems that
passed some *very simple* tests I ran last year. Ceph crashed. MooseFS
hung (and also doesn't honor O_SYNC). OrangeFS corrupted data. HDFS
cheats by buffering writes locally, and doesn't even try to implement half
of the required behaviors for a general-purpose filesystem. I can go
through any of those codebases and find awful bug after horrible bug after
data-destroying bug . . . and yet each of them has their fans too, because
most users could never possibly hit the edge conditions where those bugs
exist. The lesson is that anecdotes do not equal data. Don't listen to
vendor hype, and don't listen to anti-vendor bashing either. Find out what
the *typical* experience across a large number of users is, and how well
the software works in your own testing.

Second, just as every piece of software has bugs, every truly distributed
filesystem (i.e. not NFS) struggles with lots of small files. There has
been some progress in this area with projects like Pomegranate and GIGA+,
we have some ideas for how to approach it in GlusterFS (see my talk at SDC
next week), but overall I think it's important to realize that such a
workload is likely to be problematic for *any* offering in the category.
You'll have to do a lot of tuning, maybe implement some special
workarounds yourself, but if you want to combine this I/O profile with the
benefits of scalable storage it can all be worth it.

Lastly, if anybody is paying a 4x disk-space penalty (at one site) I'd say
they're overdoing things. Once you have replication between servers,
RAID-1 on each server is overkill. I'd say even RAID-6 is overkill. How
many simultaneous disk failures do you need to survive? If the answer is
two, as it usually seems to be, then GlusterFS replication on top of RAID-5
is a fine solution and requires a maximum of 3x (more typically just a bit
more than 2x). In the future we're looking at various forms of compression
and deduplication and erasure codes that will all bring the multiple down
even further.

So I can't say whether it's ready or whether you can trust it. I'm not
objective enough for my opinion on that to count for much. What I'm saying
is that distributed filesystems are complex pieces of sofware, none of the
alternatives are where any of us working on them would like to be, and the
only way any of these projects get better is if users let us know of
problems they encounter. Blog posts or comments describing specific
issues, from people whose names appear nowhere on any email or bug report
the developers could have seen, don't help to advance the state of the art.

===== [/comment] =====


Regards,


--
J. Pavel Espinal
Skype: p.espinal
http://ww.pavelespinal.com
http://www.slackware-es.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-11-2012, 04:58 PM
Les Mikesell
 
Default Is glusterfs ready?

On Tue, Sep 11, 2012 at 10:45 AM, Jose P. Espinal <jose@pavelespinal.com> wrote:

> Blog posts or comments describing specific
> issues, from people whose names appear nowhere on any email or bug report
> the developers could have seen, don't help to advance the state of the art.

Just speaking for myself here, I'm less interested in 'advancing' the
state of the art' (which usually means running something painfully
broken) than in finding something that already works... You didn't
paint a very rosy picture there. Would it be better to just forget
filesystem semantics and use one of the distributed nosql databases
(riak, mongo, cassandra, etc.).

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-12-2012, 12:30 PM
John Doe
 
Default Is glusterfs ready?

From: Jose P. Espinal <jose@pavelespinal.com>
> First, sure GlusterFS has bugs.* Some of them even make me cringe.* If we
> really wanted to get into a discussion of the things about GlusterFS that
> suck, I'd probably be able to come up with more things than anybody, but
> one of the lessons I learned early in my career is that seeing all of the
> bugs for a piece of software leads to a skewed perspective.* Some people
> have had problems with GlusterFS but some people have been very happy with
> it, and I guarantee that every alternative has its own horror stories.
> ...
> So I can't say whether it's ready or whether you can trust it.* I'm
> not objective enough for my opinion on that to count for much.* What
> I'm saying is that distributed filesystems are complex pieces of sofware,
> none of the alternatives are where any of us working on them would like
> to be, and the only way any of these projects get better is if users let
> us know of problems they encounter.

By ready I just meant "safe enough to transfer all our production storage
on it and be 99.99% sure that it won't vanish one night"....
Again, the same level of trust that one can have with RAID storage.
It can still fail, but it is nowadays quite rare (luckily never happened to me).

I understand that developers need testers and feedback, and I am sure you
are doing an excellent job, but we will start with a small test cluster and
follow the project progress.

Thx for your input,
JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 06:32 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org