40TB File System Recommendations
 

On Tue, 12 Apr 2011, Marian Marinov wrote:


On Tuesday 12 April 2011 10:36:54 Alain Péan wrote:

Le 12/04/2011 09:23, Matthew Feinberg a écrit :

Hello All

I have a brand spanking new 40TB Hardware Raid6 array to play around
with. I am looking for recommendations for which filesystem to use. I am
trying not to break this up into multiple file systems as we are going
to use it for backups. Other factors is performance and reliability.

CentOS 5.6

array is /dev/sdb

So here is what I have tried so far
reiserfs is limited to 16TB
ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
support creating ext4 (strange)

Anyone work with large filesystems like this that have any
suggestions/recommendations?


Hi Matthew,

I would go for xfs, which is now supported in CentOS. This is what I use
for a 16 TB storage, with CentOS 5.3 (Rocks Cluster), and it woks fine.
No problem with lengthy fsck, as with ext3 (which does not support such
capacities). I did not try yet ext4...

Alain


I have Raid6 Arrays with 30TB. We have tested XFS and its write performance
was really dissapointing. So we looked at Ext4. It is really good for our
workloads, but it lacks the ability to grow over 16TB. So we crated two
partitions on the raid with ext4.

The RAID rebuild time is around 2 days, max 3 if the workload is higher. So I
presume that for 40TB it will be around 4 days.

Marian



For interest how much *memory* would you need in your raid management node
to support "fsck" on a 40TB array. I imagine it would be very high.


Steve
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

On Apr 12, 2011, at 3:23 AM, Matthew Feinberg wrote:

ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
support creating ext4 (strange)

The CentOS homepage states that ext4 is now a fully supported filesystem in 5.6.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

On Tuesday 12 April 2011 15:34:21 Torres, Giovanni (NIH/NINDS) [C] wrote:
> On Apr 12, 2011, at 3:23 AM, Matthew Feinberg wrote:
>
> ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
> support creating ext4 (strange)
>
> The CentOS homepage states that ext4 is now a fully supported filesystem in
> 5.6. _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos

Steve,
I'm managing machines with 30TB of storage for more then two years. And with
good reporting and reaction we have never had to run fsck.

However I'm sure that if you have to run fsck on so big file systems, it will
be fater to rebuild the array from other storage then waiting for a few weeks
to finish.

On machines like that I use CentOS but I'm pratitioning them before the
install with a rescue live cd that I have created for me.

Marian
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

>-----Original Message-----
>From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf
>Of Torres, Giovanni (NIH/NINDS) [C]
>Sent: Tuesday, April 12, 2011 2:34 PM
>To: CentOS mailing list
>Subject: Re: [CentOS] 40TB File System Recommendations
>
>On Apr 12, 2011, at 3:23 AM, Matthew Feinberg wrote:
>ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
>support creating ext4 (strange)
>
>The CentOS homepage states that ext4 is now a fully supported filesystem in
>5.6.

I finalized an install with CentOS 5.6 yesterday on a machine that will be our
department fileserver. Ext4 seems to work fine on this raid-array.

In what way is ext4 not "fully baked" on CentOS 5.6?

IIRC, gparted won't be able to manipulate eg ext4 partitions if you don't have
the appropriate ext4 fs-utils installed. I might be wrong though.

OTOH, gparted doesn't see my software raid array either. Gparted it rather
practical for regular plain vanilla partitions, but for more advanced stuff and
filesystems, fdisk is probably better.

My two oere.
--
/Sorin


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

On Tue, Apr 12, 2011 at 2:47 PM, Marian Marinov <mm@yuhu.biz> wrote:
>
> Steve,
> I'm managing machines with 30TB of storage for more then two years. And with
> good reporting and reaction we have never had to run fsck.
>
> However I'm sure that if you have to run fsck on so big file systems, it will
> be fater to rebuild the array from other storage then waiting for a few weeks
> to finish.
>
> On machines like that I use CentOS but I'm pratitioning them before the
> install with a rescue live cd that I have created for me.
>
> Marian
>
> _______________________________________________

As matter of interest, what hardware do you use? i.e. what CPU's, size
of RAM and RAID cards do you use on this size system?

Everyone always recommends to use smaller RAID arrays than one big fat
one. So, I'm interested to know what you use, and how effective it
works. i.e. if that 30TB was actively used by many hosts how does it
cope? Or is it just archival storage?



--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

> On Tuesday 12 April 2011 15:34:21 Torres, Giovanni (NIH/NINDS) [C] wrote:
>> On Apr 12, 2011, at 3:23 AM, Matthew Feinberg wrote:
>>
>> ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
>> support creating ext4 (strange)
>>
>> The CentOS homepage states that ext4 is now a fully supported filesystem
>> in
>> 5.6. _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>
> Steve,
> I'm managing machines with 30TB of storage for more then two years. And
> with
> good reporting and reaction we have never had to run fsck.

That's not the issue.
The issue is rebuild-time.
The longer it takes, the more likely is another failure in the array.
With RAID6, this does not instantly kill your RAID, as with RAID5 - but I
assume it will further decrease overall-performance and the rebuild-time
will go up significantly - adding the the risk.
Thus, it's generally advisable to do just use RAID10 (in this case, a
thin-striped array of RAID1-arrays).



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

On Tuesday 12 April 2011 15:56:54 rainer@ultra-secure.de wrote:
> > On Tuesday 12 April 2011 15:34:21 Torres, Giovanni (NIH/NINDS) [C] wrote:
> >> On Apr 12, 2011, at 3:23 AM, Matthew Feinberg wrote:
> >>
> >> ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not
> >> support creating ext4 (strange)
> >>
> >> The CentOS homepage states that ext4 is now a fully supported filesystem
> >> in
> >> 5.6. _______________________________________________
> >> CentOS mailing list
> >> CentOS@centos.org
> >> http://lists.centos.org/mailman/listinfo/centos
> >
> > Steve,
> > I'm managing machines with 30TB of storage for more then two years. And
> > with
> > good reporting and reaction we have never had to run fsck.
>
> That's not the issue.
> The issue is rebuild-time.
> The longer it takes, the more likely is another failure in the array.
> With RAID6, this does not instantly kill your RAID, as with RAID5 - but I
> assume it will further decrease overall-performance and the rebuild-time
> will go up significantly - adding the the risk.
> Thus, it's generally advisable to do just use RAID10 (in this case, a
> thin-striped array of RAID1-arrays).
>

Yes... but with such RAID10 solution you get only half of the disk space... so
from 10 2TB drives you get only 10TB instead of 16TB with RAID6.

Some of us really need the space. Rebuild time(while it is less then 4 days)
is considered good enough. In my case I'm using these servers for backups and
the raid rebuilds haven't made any changes to the performance of the backups.

I'm sure that if you use such storage with RAID6 for VMs it wont perform very
well.

Marian
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

> OTOH, gparted doesn't see my software raid array either. Gparted it rather
> practical for regular plain vanilla partitions, but for more advanced stuff and
> filesystems, fdisk is probably better.

For filersystems > 2TB, you're better off grabbing a copy of GPT fdisk.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

Rudi Ahlers wrote:
> On Tue, Apr 12, 2011 at 2:47 PM, Marian Marinov <mm@yuhu.biz> wrote:
>>
>> I'm managing machines with 30TB of storage for more then two years. And
>> with good reporting and reaction we have never had to run fsck.
>>
>> However I'm sure that if you have to run fsck on so big file systems, it
>> will be fater to rebuild the array from other storage then waiting for
a few
>> weeks to finish.
<snip>
Here's a question: which would be faster on that huge a filesystem: fsck,
or having a second 30TB filesystem, and rsyncing everything over?

mark

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

40TB File System Recommendations
 

>-----Original Message-----
>From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf
>Of Lars Hecking
>Sent: Tuesday, April 12, 2011 3:11 PM
>To: centos@centos.org
>Subject: Re: [CentOS] 40TB File System Recommendations
>
>
>> OTOH, gparted doesn't see my software raid array either. Gparted it rather
>> practical for regular plain vanilla partitions, but for more advanced stuff
and
>> filesystems, fdisk is probably better.
>
> For filersystems > 2TB, you're better off grabbing a copy of GPT fdisk.

Oh, there are two flavours of fdisk? Didn't know. Thanks.

--
/Sorin


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos