FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 04-12-2011, 04:21 PM
Keith Keller
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 06:00:57PM +0300, Marian Marinov wrote:
>
> Can someone(who actually knows) share with us, what is the state of xfs-utils,
> how stable and usable are they for recovery of broken XFS filesystems?

I have done an XFS repair once or twice on a real filesystem (~4TB) in a
64bit kernel. It worked fine, but I don't think the filesystem was too
badly thrashed.

As another poster noted, be ready to throw memory or swap at the XFS
check and repair tools. (I read that it's slightly better memory-wise
to run xfs_repair -n than xfs_check, but I believe that's mainly for
32bit systems, and that may have been fixed anyway.)

--keith


--
kkeller@wombat.san-francisco.ca.us

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 04:39 PM
Pasi Kärkkäinen
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 10:36:39AM -0400, John Jasen wrote:
> On 04/12/2011 10:21 AM, Boris Epstein wrote:
> > On Tue, Apr 12, 2011 at 3:36 AM, Alain Péan
> > <alain.pean@lpp.polytechnique.fr
> > <mailto:alain.pean@lpp.polytechnique.fr>> wrote:
>
> <snipped: two recommendations for XFS>
>
> I would chime in with a dis-commendation for XFS. At my previous
> employer, two cases involving XFS resulted in irrecoverable data
> corruption. These were on RAID systems running from 4 to 20 TB.
>

Did you have these problems with XFS on 32bit Linux?

-- Pasi

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 05:50 PM
 
Default 40TB File System Recommendations

On Apr 12, 2011, at 12:31 AM, John R Pierce wrote:

> On 04/12/11 12:23 AM, Matthew Feinberg wrote:
>> Hello All
>>
>> I have a brand spanking new 40TB Hardware Raid6 array
>
> never mind file systems... is that one raid set? do you have any
> idea
> how LONG rebuilding that is going to take when there are any drive
> hiccups? or how painfully slow writes will be until its rebuilt? is
> that something like 22 x 2TB or 16 x 3TB? I'll bet a raid rebuild
> takes nearly a WEEK, maybe even longer..
>
> I am very strongly NOT in favor of raid6, even for nearline bulk
> backup
> storage. I would sacrifice the space and format that as raid10, and
> have at LEAST a couple hot spares too.

+1 for the 1+0 and a few hot spares.

Raid 6 + spare ran great but rebuilds took 2 days. The likely hood of
2+ failed drives is less then 1 failed drive but I actually had 2
failed drives so RAID6 + spare saved me.

Hence why I switched to RAID 1+0 + spares.

A tuned XFS fs will work great.

I run my large RAID XFS fs with logbufs=8, and no(atime.dirtime).

I also run iozone for testing my tuned options for optimum performance
in my env.

- aurf
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 06:51 PM
John R Pierce
 
Default 40TB File System Recommendations

On 04/12/11 6:02 AM, Marian Marinov wrote:
>
> Yes... but with such RAID10 solution you get only half of the disk space... so
> from 10 2TB drives you get only 10TB instead of 16TB with RAID6.

those disks are $100 each. whats your data worth?

The rebuild time goes way up as the number of drives in the raid stripe
goes up.

in this case, the OP is talking about a 40TB array, so thats a TWENTY
TWO drive raid. NOONE I know in the storage business will use larger
than a 8 or 10 drive raid set. If you really need such a massive
volume, you stripe several smaller raidsets, so the raid6 version would
be 2 x 12 x 2TB or 24 drives for raid6+0 == 40TB.

but the OP's application is backup. for backup, it really doesn't
matter what the volume size is, more smaller file systems is fine, so
you can partition your backups by date interval or whatever.

let me throw out another thing. I assume this 40TB backup server is
not just ONE backup of the current state, but an archive of
point-in-time backups? you better have more than one of them, where you
backup the backup on the 2nd. there's any number of scenarios the
raid6 won't protect against, including file system corruption, raid
controller failure where it dumps across a whole stripe, etc.




_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 08:54 PM
Lamar Owen
 
Default 40TB File System Recommendations

On Tuesday, April 12, 2011 02:51:45 PM John R Pierce wrote:
> On 04/12/11 6:02 AM, Marian Marinov wrote:
> >
> > Yes... but with such RAID10 solution you get only half of the disk space... so
> > from 10 2TB drives you get only 10TB instead of 16TB with RAID6.
>
> those disks are $100 each. whats your data worth?

Where can I get an enterprise-class 2TB drive for $100? Commodity SATA isn't enterprise-class. SAS is; FC is, SCSI is. A 500GB FC drive with EMC firmware new is going to set you back ten times that, at least. What's youre data worth indeed, putting it on commodity disk.... :-)

> in this case, the OP is talking about a 40TB array, so thats a TWENTY
> TWO drive raid. NOONE I know in the storage business will use larger
> than a 8 or 10 drive raid set.

EMC allows RAID groups up to 16 drives on Clariion storage. I've been doing this with EMC stuff for a while, with RAID6 plus a hotspare per DAE; that's a 14 drive RAID group plus the hotspare on one DAE. Some systems I forgo the dedicated per-DAE hotspare and spread a 16 drive RAID6 group and a 14 drive RAID6 group across two DAE's with hotspares on other DAE's. Works ok, and I've had double drive soft failures on a single RAID6 group that successfully hotspared (and back). This is partially due to the custom EMC firmware on the drives, and the interaction with the storage processor.

Rebuild time is several hours, but with more smaller drives it's not too bad.

> If you really need such a massive
> volume, you stripe several smaller raidsets, so the raid6 version would
> be 2 x 12 x 2TB or 24 drives for raid6+0 == 40TB.

Or you do metaLUNs, or similar using LVM.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 09:01 PM
 
Default 40TB File System Recommendations

On Apr 12, 2011, at 1:54 PM, Lamar Owen wrote:

> On Tuesday, April 12, 2011 02:51:45 PM John R Pierce wrote:
>> On 04/12/11 6:02 AM, Marian Marinov wrote:
>>>
>>> Yes... but with such RAID10 solution you get only half of the disk
>>> space... so
>>> from 10 2TB drives you get only 10TB instead of 16TB with RAID6.
>>
>> those disks are $100 each. whats your data worth?
>
> Where can I get an enterprise-class 2TB drive for $100?

This is a good point.

The cheapies are so called green as they spin down often which is not
what you want in a RAID setup.

While I've been able to tweak this in OSX, I haven't yet tried to see
what to do in Linux or Winblowz which I will eventually do as some
turd nugget bought a bunch of these for pro use.

> Commodity SATA isn't enterprise-class. SAS is; FC is, SCSI is. A
> 500GB FC drive with EMC firmware new is going to set you back ten
> times that, at least. What's youre data worth indeed, putting it on
> commodity disk.... :-)
>
>> in this case, the OP is talking about a 40TB array, so thats a TWENTY
>> TWO drive raid. NOONE I know in the storage business will use larger
>> than a 8 or 10 drive raid set.
>
> EMC allows RAID groups up to 16 drives on Clariion storage.

Yea, as does BlueArc, unsure of the rest but agreed.

- aurf
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 10:02 PM
Keith Keller
 
Default 40TB File System Recommendations

On Tue, Apr 12, 2011 at 02:01:42PM -0700, aurfalien@gmail.com wrote:
>
> The cheapies are so called green as they spin down often which is not
> what you want in a RAID setup.

The WD RE4-GP is a so-called 'green' disk that's suitable for RAID
arrays. It's marketed and priced as an enterprise drive.

--keith

--
kkeller@wombat.san-francisco.ca.us

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 10:14 PM
 
Default 40TB File System Recommendations

On Apr 12, 2011, at 3:02 PM, Keith Keller wrote:

> On Tue, Apr 12, 2011 at 02:01:42PM -0700, aurfalien@gmail.com wrote:
>>
>> The cheapies are so called green as they spin down often which is not
>> what you want in a RAID setup.
>
> The WD RE4-GP is a so-called 'green' disk that's suitable for RAID
> arrays. It's marketed and priced as an enterprise drive.

Well, it may either be BS marketing or is so called green for a diff
reason and not the freq spin downs.

I'm finding green can mean many things from the fact that there
product packaging is made from recycled material to there
manufacturing plant no longer uses mercury to there power consumption
is lower then previous models, etc...

I would say that the $100 price tag would be a caution but then again
one doesn't always get what one pays for.

Either way, it makes our jobs more challenging for sure.

- aurf


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 10:49 PM
Drew
 
Default 40TB File System Recommendations

> Where can I get an enterprise-class 2TB drive for $100? *Commodity SATA isn't enterprise-class. *SAS is; FC is, SCSI is. A 500GB FC drive with EMC firmware new is going to set you back ten times that, at least. *What's youre data worth indeed, putting it on commodity disk.... :-)

I can get Seagate's Constellation ES series SATA drives in 1TB for
$125. 2TB will run me around $225.

They're not something I'd run my database off, I have 15k SAS drives
for that, but for large amounts of storage on the cheap like our
backup system, it's just fine.

--
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-12-2011, 11:00 PM
"compdoc"
 
Default 40TB File System Recommendations

>The WD RE4-GP is a so-called 'green' disk that's suitable for RAID
>arrays. It's marketed and priced as an enterprise drive.

I've had good luck with green, 5400 rpm Samsung drives. They don't spin down
automatically and work fine in my raid 5 arrays. The cost is about $80 for
2TB drives.

I also have a few 5900 rpm Seagate ST32000542AS drives, but not currently in
raids. They don't spin down, so I'm sure they would be fine in a raid.

None of the drives in the raids have failed, although I've replaced a couple
that developed reallocated sectors as reported by smart.

Just because they are so tiny on the outside, 2.5 inch drives like the
Seagate Constellation and WD Raptors are great. Unfortunately, the don't
come any larger than 1TB, so I use them in special situations.




_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 10:10 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org