FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 04-12-2011, 01:47 PM
Marian Marinov
 
Default 40TB File System Recommendations

On Tuesday 12 April 2011 16:20:22 m.roth@5-cent.us wrote:
> Rudi Ahlers wrote:
> > On Tue, Apr 12, 2011 at 2:47 PM, Marian Marinov <mm@yuhu.biz> wrote:
> >> I'm managing machines with 30TB of storage for more then two years. And
> >> with good reporting and reaction we have never had to run fsck.
> >>
> >> However I'm sure that if you have to run fsck on so big file systems, it
> >> will be fater to rebuild the array from other storage then waiting for
>
> a few
>
> >> weeks to finish.
>
> <snip>
> Here's a question: which would be faster on that huge a filesystem: fsck,
> or having a second 30TB filesystem, and rsyncing everything over?

For us, it was faster to transfer the information again. At least this was
during the tests. We have never had to do it for real.

I guess the time for the fsck depends on the amount of errors that you have.
If it has to check only the jurnal the fsck will not take long. But i it has
to do a full check of the FS... an rsync may be faster.

Marian
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 01:48 PM
Markus Falb
 
Default 40TB File System Recommendations

On 12.4.2011 15:02, Marian Marinov wrote:
> On Tuesday 12 April 2011 15:56:54 rainer-RNrd0m5o0MABOiyIzIsiOw@public.gmane.org wrote:

> Yes... but with such RAID10 solution you get only half of the disk space... so
> from 10 2TB drives you get only 10TB instead of 16TB with RAID6.

From a somewhat theoretical view, this is true for standard raid10 but
Linux md raid10 is much more flexible as I understood it. You could do 2
copys over 2 disks, thats like standard 10. Or you could do 2 copys over
2 or 3 or ... x disks. Or you could do 3 copys over 3 or 4 or ... x
disks. Do the math. See the manpage for md(4) and
http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

However, I have to admit that I have no experience with that but would
like to hear about any disadvantages or if I am mislead. I am just
interested.

--
Kind Regards, Markus Falb

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 01:56 PM
Rudi Ahlers
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 3:48 PM, Markus Falb <markus.falb@fasel.at> wrote:


On 12.4.2011 15:02, Marian Marinov wrote:

> On Tuesday 12 April 2011 15:56:54 rainer-RNrd0m5o0MABOiyIzIsiOw@public.gmane.org wrote:



> Yes... but with such RAID10 solution you get only half of the disk space... so

> from 10 2TB drives you get only 10TB instead of 16TB with RAID6.



From a somewhat theoretical view, this is true for standard raid10 but

Linux md raid10 is much more flexible as I understood it. You could do 2

copys over 2 disks, thats like standard 10. Or you could do 2 copys over

2 or 3 or ... x disks. Or you could do 3 copys over 3 or 4 or ... x

disks. Do the math. See the manpage for md(4) and

http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10



However, I have to admit that I have no experience with that but would

like to hear about any disadvantages or if I am mislead. I am just

interested.



--





We only use RAID 10 (rather 1+0) and never even bothered with RAID6. *And we've had no data loss in the past 3 years with it yet, on hundreds of servers.*


But, our RAID10 is setup as a stripe of mirrors, i.e. sda1 & sdb1 -> md0, sdc1 + sdd1 ->md1, then sde1 + sdf1 ->md2, and finally md0 + md1 + md2 are stripped. The advantage of this is that we can add more disks to the whole RAID set with no downtime (all server have hot swap HDD cages) and very little performance degradation since the 2 new drives *have to be*mirrored*on their own first (take very little CPU / RAM resources) and then added to the RAID set. Rebuild is generally quick since it only rebuilds the broken mirror*


--
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com


Office: 087 805 9573
Cell: 082 554 7532

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 02:01 PM
Marian Marinov
 
Default 40TB File System Recommendations

On Tuesday 12 April 2011 16:48:14 Markus Falb wrote:
> On 12.4.2011 15:02, Marian Marinov wrote:
> > On Tuesday 12 April 2011 15:56:54
> > rainer-RNrd0m5o0MABOiyIzIsiOw@public.gmane.org wrote:
> >
> > Yes... but with such RAID10 solution you get only half of the disk
> > space... so from 10 2TB drives you get only 10TB instead of 16TB with
> > RAID6.
>
> From a somewhat theoretical view, this is true for standard raid10 but
> Linux md raid10 is much more flexible as I understood it. You could do 2
> copys over 2 disks, thats like standard 10. Or you could do 2 copys over
> 2 or 3 or ... x disks. Or you could do 3 copys over 3 or 4 or ... x
> disks. Do the math. See the manpage for md(4) and
> http://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
>
> However, I have to admit that I have no experience with that but would
> like to hear about any disadvantages or if I am mislead. I am just
> interested.
Its like doing RAID50 or RAID60... Again the cheapest solution is RAID6.
I really like the software raid in linux, it has good performance. But I have
never tested it on such big volumes. And usually it is really hard to put 10
or more drives on a machine without buying a sata controler.

Marian

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 02:21 PM
Boris Epstein
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 3:36 AM, Alain Péan <alain.pean@lpp.polytechnique.fr> wrote:

Le 12/04/2011 09:23, Matthew Feinberg a écrit :

> Hello All

>

> I have a brand spanking new 40TB Hardware Raid6 array to play around

> with. I am looking for recommendations for which filesystem to use. I am

> trying not to break this up into multiple file systems as we are going

> to use it for backups. Other factors is performance and reliability.

>

> CentOS 5.6

>

> array is /dev/sdb

>

> So here is what I have tried so far

> reiserfs is limited to 16TB

> ext4 does not seem to be fully baked in 5.6 yet. parted 1.8 does not

> support creating ext4 (strange)

>

> Anyone work with large filesystems like this that have any

> suggestions/recommendations?



Hi Matthew,



I would go for xfs, which is now supported in CentOS. This is what I use

for a 16 TB storage, with CentOS 5.3 (Rocks Cluster), and it woks fine.

No problem with lengthy fsck, as with ext3 (which does not support such

capacities). I did not try yet ext4...



Alain



--

================================================== ========

Alain Péan - LPP/CNRS

Administrateur Système/Réseau

Laboratoire de Physique des Plasmas - UMR 7648

Observatoire de Saint-Maur

4, av de Neptune, Bat. A

94100 Saint-Maur des Fossés

Tel : 01-45-11-42-39 - Fax : 01-48-89-44-33

================================================== ========



_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos


I fully second Alain's opinion. An fsck on a 6 TB RAID6 containing about 30 million files takes over 10 hours.

As for XFS, we are running it on a 25 TB array and so far there has been no trouble.


Boris.

Boris.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 02:25 PM
David Miller
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 8:56 AM, <rainer@ultra-secure.de> wrote:
That's not the issue.

The issue is rebuild-time.

The longer it takes, the more likely is another failure in the array.

With RAID6, this does not instantly kill your RAID, as with RAID5 - but I

assume it will further decrease overall-performance and the rebuild-time

will go up significantly - adding the the risk.

Thus, it's generally advisable to do just use RAID10 (in this case, a

thin-striped array of RAID1-arrays).
Statistically speaking that risk isn't there. �RAID6 arrays have a slightly higher mean time between dataloss than RAID10's. �But the difference here is very small. �So if you need the capacity and don't mind the performance difference between these two RAID levels then RAID6 is perfectly fine in my opinion.

Here's a great blog post on calculating Mean Time Between Dataloss and they have a spreadsheet that you can download to play with. �http://info.zetta.net/blog/bid/45661/Calculating-Mean-Time-To-Data-Loss-and-probability-of-silent-data-corruption

In my configuration which is 12 drives the chances of a dataloss event over a 10 year period with RAID10 is 2.51% and with RAID6 is 1.31%. �I would expect those numbers to go up a bit with 16 drive configuration.

--David
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 02:36 PM
John Jasen
 
Default 40TB File System Recommendations

On 04/12/2011 10:21 AM, Boris Epstein wrote:
> On Tue, Apr 12, 2011 at 3:36 AM, Alain Péan
> <alain.pean@lpp.polytechnique.fr
> <mailto:alain.pean@lpp.polytechnique.fr>> wrote:

<snipped: two recommendations for XFS>

I would chime in with a dis-commendation for XFS. At my previous
employer, two cases involving XFS resulted in irrecoverable data
corruption. These were on RAID systems running from 4 to 20 TB.


--
-- John E. Jasen (jjasen@realityfailure.org)
-- "Deserve Victory." -- Terry Goodkind, Naked Empire
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 03:00 PM
Marian Marinov
 
Default 40TB File System Recommendations

On Tuesday 12 April 2011 17:36:39 John Jasen wrote:
> On 04/12/2011 10:21 AM, Boris Epstein wrote:
> > On Tue, Apr 12, 2011 at 3:36 AM, Alain Péan
> > <alain.pean@lpp.polytechnique.fr
>
> > <mailto:alain.pean@lpp.polytechnique.fr>> wrote:
> <snipped: two recommendations for XFS>
>
> I would chime in with a dis-commendation for XFS. At my previous
> employer, two cases involving XFS resulted in irrecoverable data
> corruption. These were on RAID systems running from 4 to 20 TB.

Can someone(who actually knows) share with us, what is the state of xfs-utils,
how stable and usable are they for recovery of broken XFS filesystems?

Marian
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 03:30 PM
"James A. Peltier"
 
Default 40TB File System Recommendations

----- Original Message -----
| On Tuesday 12 April 2011 17:36:39 John Jasen wrote:
| > On 04/12/2011 10:21 AM, Boris Epstein wrote:
| > > On Tue, Apr 12, 2011 at 3:36 AM, Alain Péan
| > > <alain.pean@lpp.polytechnique.fr
| >
| > > <mailto:alain.pean@lpp.polytechnique.fr>> wrote:
| > <snipped: two recommendations for XFS>
| >
| > I would chime in with a dis-commendation for XFS. At my previous
| > employer, two cases involving XFS resulted in irrecoverable data
| > corruption. These were on RAID systems running from 4 to 20 TB.
|
| Can someone(who actually knows) share with us, what is the state of
| xfs-utils,
| how stable and usable are they for recovery of broken XFS filesystems?
|
| Marian
|
| _______________________________________________
| CentOS mailing list
| CentOS@centos.org
| http://lists.centos.org/mailman/listinfo/centos

On 64-bit platforms the tools are totally stable, but it does depend on the degree of "broken" state that the file system is in. I've had xfs_checks run for days and eat up 96GB of memory because of various degrees of "broken"-ness. These are on 35 and 45TB file systems. Be prepared to throw memory at the problem or lots of swap files if you get really buggered up.

--
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpeltier@sfu.ca
Website : http://www.sfu.ca/itservices
http://blogs.sfu.ca/people/jpeltier


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 03:30 PM
Les Mikesell
 
Default 40TB File System Recommendations

On 4/12/2011 9:36 AM, John Jasen wrote:
>
> <snipped: two recommendations for XFS>
>
> I would chime in with a dis-commendation for XFS. At my previous
> employer, two cases involving XFS resulted in irrecoverable data
> corruption. These were on RAID systems running from 4 to 20 TB.

Was this on a 32 or 64 bit system?

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 12:20 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org