FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 04-13-2010, 05:05 PM
Boris Epstein
 
Default 12-15 TB RAID storage recommendations

Hello listmates,

I would like to build a 12-15 TB RAID 5 data server to run under
ContOS. Any recommendations as far as hardware, configuration, etc?

Thanks.

Boris.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 05:14 PM
Ryan Manikowski
 
Default 12-15 TB RAID storage recommendations

On 4/13/2010 1:05 PM, Boris Epstein wrote:

Hello listmates,

I would like to build a 12-15 TB RAID 5 data server to run under
ContOS. Any recommendations as far as hardware, configuration, etc?

Thanks.

Boris.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos




Chassis - CSE-836A-R1200B*Supermicro SC836 A-R1200B - Rack-mountable -
3U - SATA/SAS - hot-swap - power supply 1200 Watt



RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL
RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 -
Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA



BBU Module for RAID card - 3ware BBU-MODULE-03





Pick the cpu(s) and motherboard to fit the chassis. Obviously go with
ECC ram and ONLY enterprise grade hard drives. To ensure compatibility
check with 3ware to see which drives they recommend. Areca RAID cards
will get you a little better performance but the module for the 9650SE
series of 3ware cards is included with the Centos kernel. Getting the
Areca driver going is a bit more work, but nothing that would be
considered a huge hurdle for a competent sysadmin. Also, if you're
looking for advice on Areca products call their Tekram contact in the
USA. Their other distributors have been less than stellar on answering
pre-sales questions.





--
Ryan Manikowski


]] Devision Media Services LLC [[
www.devision.us
ryan@devision.us | 716.771.2282


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 05:19 PM
Boris Epstein
 
Default 12-15 TB RAID storage recommendations

On Tue, Apr 13, 2010 at 1:14 PM, Ryan Manikowski <ryan@devision.us> wrote:
> On 4/13/2010 1:05 PM, Boris Epstein wrote:
>
> Hello listmates,
>
> I would like to build a 12-15 TB RAID 5 data server to run under
> ContOS. Any recommendations as far as hardware, configuration, etc?
>
> Thanks.
>
> Boris.
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>
> Chassis - CSE-836A-R1200B*Supermicro SC836 A-R1200B - Rack-mountable - 3U -
> SATA/SAS - hot-swap - power supply 1200 Watt
>
> RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH
> SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4
> Serial ATA/300 - Serial ATA
>
> BBU Module for RAID card - 3ware BBU-MODULE-03
>
>
> Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC
> ram and ONLY enterprise grade hard drives. To ensure compatibility check
> with 3ware to see which drives they recommend. Areca RAID cards will get you
> a little better performance but the module for the 9650SE series of 3ware
> cards is included with the Centos kernel. Getting the Areca driver going is
> a bit more work, but nothing that would be considered a huge hurdle for a
> competent sysadmin. Also, if you're looking for advice on Areca products
> call their Tekram contact in the USA. Their other distributors have been
> less than stellar on answering pre-sales questions.
>
>
> --
> Ryan Manikowski
>
>
> ]] Devision Media Services LLC [[
> www.devision.us
> ryan@devision.us | 716.771.2282
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

Thanks Ryan!

So, basically, I take it you've had positive experience with that
Supermicro box. It does look good and seems fairly inexpensive, too.

Boris.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 05:19 PM
David Miller
 
Default 12-15 TB RAID storage recommendations

On Tue, Apr 13, 2010 at 1:14 PM, Ryan Manikowski <ryan@devision.us> wrote:







On 4/13/2010 1:05 PM, Boris Epstein wrote:

Hello listmates,

I would like to build a 12-15 TB RAID 5 data server to run under
ContOS. Any recommendations as far as hardware, configuration, etc?

Thanks.

Boris.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos




Chassis - CSE-836A-R1200B*Supermicro SC836 A-R1200B - Rack-mountable -
3U - SATA/SAS - hot-swap - power supply 1200 Watt



RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL
RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 -
Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA



BBU Module for RAID card - 3ware BBU-MODULE-03





Pick the cpu(s) and motherboard to fit the chassis. Obviously go with
ECC ram and ONLY enterprise grade hard drives. To ensure compatibility
check with 3ware to see which drives they recommend. Areca RAID cards
will get you a little better performance but the module for the 9650SE
series of 3ware cards is included with the Centos kernel. Getting the
Areca driver going is a bit more work, but nothing that would be
considered a huge hurdle for a competent sysadmin. Also, if you're
looking for advice on Areca products call their Tekram contact in the
USA. Their other distributors have been less than stellar on answering
pre-sales questions.





--
Ryan Manikowski


]] Devision Media Services LLC [[
www.devision.us
ryan@devision.us | 716.771.2282



_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



Ryan's hardware recommendations are good.* But I wouldn't run a RAID5 volume that large, software or hardware.* It's just too risky as rebuilds will
take days and the chances of hitting a non recoverable read error would
be near 100% on a volume that size.



Either run multiple smaller RAID5's and use LVM to manage the volumes
which the OS will use or choose a better RAID layout.* RAID6 or RAID10
are much better choices these days.

--

David
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 05:31 PM
Ryan Manikowski
 
Default 12-15 TB RAID storage recommendations

On 4/13/2010 1:19 PM, David Miller wrote:

<snip>
>
> Ryan's hardware recommendations are good. But I wouldn't run a RAID5
> volume that large, software or hardware. It's just too risky as
> rebuilds will take days and the chances of hitting a non recoverable
> read error would be near 100% on a volume that size.
>
> Either run multiple smaller RAID5's and use LVM to manage the volumes
> which the OS will use or choose a better RAID layout. RAID6 or RAID10
> are much better choices these days.
> --
> David

With the config mentioned above it would give the flexibility to run
RAID10 with a resulting data store of just under 14TB (8x RAID1 stripe
using 2TB drives).

Choice of RAID implementation (specifically RAID5) could be an
impediment to performance as noted above. Always good to have input from
more than one source. At that level of storage looking into spending a
bit more for redundancy (drbd/pacemkaker/heartbeat) may be a worthwhile
investment as well.


--
Ryan Manikowski


]] Devision Media Services LLC [[
www.devision.us
ryan@devision.us | 716.771.2282

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 06:30 PM
"Joseph L. Casale"
 
Default 12-15 TB RAID storage recommendations

>I would like to build a 12-15 TB RAID 5 data server to run under
>ContOS. Any recommendations as far as hardware, configuration, etc?

This comes up often
Things to keep in mind when using large discs if you suffer a failure,
while rebuilding in a degraded state it's not impossible or unlikely to
drop another disc loosing it all.

Rebuild times especially on busy arrays with large discs take lots of
time...
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 06:39 PM
Boris Epstein
 
Default 12-15 TB RAID storage recommendations

On Tue, Apr 13, 2010 at 2:30 PM, Joseph L. Casale
<jcasale@activenetwerx.com> wrote:
>>I would like to build a 12-15 TB RAID 5 data server to run under
>>ContOS. Any recommendations as far as hardware, configuration, etc?
>
> This comes up often
> Things to keep in mind when using large discs if you suffer a failure,
> while rebuilding in a degraded state it's not impossible or unlikely to
> drop another disc loosing it all.
>
> Rebuild times especially on busy arrays with large discs take lots of
> time...
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

Thanks Joseph!

In your opinion, if we are talking about 15 2-TB disks what does "lots
of time" translate to?

Boris.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 06:45 PM
"nate"
 
Default 12-15 TB RAID storage recommendations

Joseph L. Casale wrote:

> Rebuild times especially on busy arrays with large discs take lots of
> time...

Unless you have a good storage system..

a blog entry I wrote last year:
http://www.techopsguys.com/2009/11/24/81000-raid-arrays/

Another one where I ripped into Equallogic's claims:
http://www.techopsguys.com/2010/03/26/enterprise-equallogic/

Just checked my array again, nearly 200,000 RAID arrays on it,
makes for a massively parallel many:many RAID rebuild for fast
recovery times with *no* service impact.

Course this stuff may be out of the OP's budget, but just keep in
mind that there are such systems on the market.

IBM XIV is another such system, though it's scalability is too
limited to be useful IMO (~180 drives max).

You'd have to pull my toenails out with a rusty pair of pliers
before I go back to crap storage.

nate



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 06:49 PM
Eero Volotinen
 
Default 12-15 TB RAID storage recommendations

2010/4/13 Boris Epstein <borepstein@gmail.com>:
> Hello listmates,
>
> I would like to build a 12-15 TB RAID 5 data server to run under
> ContOS. Any recommendations as far as hardware, configuration, etc?

how about hitachi SMS 100 ? it is about that size and cost effective
iscsi solution?

--
Eero
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 04-13-2010, 06:51 PM
John R Pierce
 
Default 12-15 TB RAID storage recommendations

Boris Epstein wrote:
> On Tue, Apr 13, 2010 at 2:30 PM, Joseph L. Casale
> <jcasale@activenetwerx.com> wrote:
>
>>> I would like to build a 12-15 TB RAID 5 data server to run under
>>> ContOS. Any recommendations as far as hardware, configuration, etc?
>>>
>> This comes up often
>> Things to keep in mind when using large discs if you suffer a failure,
>> while rebuilding in a degraded state it's not impossible or unlikely to
>> drop another disc loosing it all.
>>
>> Rebuild times especially on busy arrays with large discs take lots of
>> time...
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>>
>
> Thanks Joseph!
>
> In your opinion, if we are talking about 15 2-TB disks what does "lots
> of time" translate to?
>


well, IF your controller totally screams and can rebuild the drives at
wire speeds with full overlap, you'll be reading 7 * 2TB of data at
around 100MB/sec average and writing the XOR of that to the 8th drive in
a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is
2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about
the shortest it possibly could be done.

if you have any access activity during this rebuild, you can figure on
nearly doubling that before you even get warm. if the controller can't
actually move 800MB/sec internally (believe me, that ain't easy), then
you can double or quadruple it a few more times. This 5.5 hours could
easily stretch to a week.


ANY previously unnoticed bad sector on those other 7 drives renders the
whole pile bad, so doing weekly raid sweeps is probably a good thing,
except THOSE will take some multiple of 5.5 hours when run in the
background.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 12:56 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org