Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo User (http://www.linux-archive.org/gentoo-user/)
-   -   Strategy for using SAN/NAS for storage with Gentoo... (http://www.linux-archive.org/gentoo-user/341836-strategy-using-san-nas-storage-gentoo.html)

Steve 03-15-2010 12:20 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
I have recently started looking at server resilience and availability in
the context of a hardware failure or hardware upgrade. I've come to the
conclusion that it would be very desirable if terrabyte-scale data did
not need to be restored from backup. This isn't a commercial server -
so I'm interested in minimum cost approaches.

With this in mind, I'm interested to discover what represents
state-of-the-art from the perspective of the OS and its configuration.
Issues I envisage are:

* With NAS, it would be desirable to have a Linux filesystem rather than
access files over CIFS - this raises further questions about protocol...
is NFS as hopelessly outdated as it seems? Are there any products that
offer NFS access? Are any of them secure?
* With a SAN, questions of filesystem features are diminished - but
questions of access protocol remain. What is best supported by gentoo?
* Do any gentooists have any inexpensive hardware configurations that
work especially well?

Any hints or tips?

Harry Putnam 03-15-2010 01:37 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
Steve <gentoo_sjh@shic.co.uk> writes:

> I have recently started looking at server resilience and availability in
> the context of a hardware failure or hardware upgrade. I've come to the
> conclusion that it would be very desirable if terrabyte-scale data did
> not need to be restored from backup. This isn't a commercial server -
> so I'm interested in minimum cost approaches.
>
> With this in mind, I'm interested to discover what represents
> state-of-the-art from the perspective of the OS and its configuration.
> Issues I envisage are:
>
> * With NAS, it would be desirable to have a Linux filesystem rather than
> access files over CIFS - this raises further questions about protocol...
> is NFS as hopelessly outdated as it seems? Are there any products that
> offer NFS access? Are any of them secure?
> * With a SAN, questions of filesystem features are diminished - but
> questions of access protocol remain. What is best supported by gentoo?
> * Do any gentooists have any inexpensive hardware configurations that
> work especially well?
>
> Any hints or tips?

Someone here, a yr or two ago recommended to me when I asked that
question to install opensolaris on a machine and set it up as NAS.

Opensolaris offers the zfs file system, that is really advanced
compared to others.

I'll admit I've had some issues along the way. And have to do lots of
boning up on opensolaris.

I access the zfs server by cifs from windows machines, and by NFS from
linux.

Opensolaris doesn't use samba by default. Instead they have their own
CIFS server which works fine. They do have samba pkgs but no one
hardly use it, preferring their CIFS server.

Mine is only a home lan setup, but even then the opensolaris server
has over a terabyte of capacity.

I have it setup in 3 mirrors of 2 disks each.

2 prs of 500gb sata hdd and one pr of 750 sata hdd

There are many ways so setup `zraid' systems on `zfs' that are excellent for
reliability. ( I have been told.... I haven't tried that route).

For me the mirror setup seemed good for my small needs. Even more
reliable I'm told, if a bit higher in disk usage.

opensolaris zfs fs offers a `timeslider' interface to a system of
snapshotting the filesystems in 15 min, 1hr 1day etc snapshots in a
very small footprint way. You'd have to read up on it.. It would take
too much off topic to cover here. The system takes an amazing small
amount of disk space for the snapshots based on COW (Copy on write).

There was talk of opensolaris going by the wayside with the Oracle
takeover of Sun... but Oracle has since announced its intention of
puttin even more resources into `opensolaris' development than Sun was
doing.

These may be a good intro to zfs:
http://www.sun.com/bigadmin/features/articles/zfs_part1.scalable.jsp
http://all-unix.blogspot.com/2007/03/zfs-cow-and-relate-features.html
http://wikis.sun.com/display/OpenSolarisInfoPL/How+to+Manage+the+Automatic+ZFS+Snapshot+Service

Stroller 03-15-2010 05:21 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On 15 Mar 2010, at 16:26, Steve wrote:

...
From ages ago, I remember iSCSI being bandied about. Did that ever go
anywhere (i.e. is this easy to do from Gentoo?)


I believe it is quite widely used - it is mentioned often on the linux-
poweredge list. I would imagine the Linux kernel allows mounting and
sharing by iSCSI - check `make menuconfig` and type "/iscsi".


It's hard to be more specific without knowing your usage.

For storage of a "mere terabyte" you can buy a networked storage
enclosure which will accommodate two drives. These are cheap, do
mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are
probably not too fast.


One reads a lot posted by people who have large movie collections
stored on the network, whether they be MythTV users or the mutineer
sailors of 17th century galleons. A PC-based solution gives you more
room for this - you can fit perhaps 4 drives in a standard PC case you
find at the tip, or you can get 12 or 16 drives in a dedicated
rackmount server case. This allows capacity of upto 32TB with current
drives, if you can afford that, or to use cheaper drives (1TB or 1.5TB
are best gigabytes-per-dollar at present, I think; 500gb drives seem
recently to have become disproportionately expensive) and have better
RAID levels.


The Norco one is popular amongst enthusiasts, because it's really
cheap [1]; it uses 2 x standard ATX power supplies, one for the
mainboard, one for the drives. You can get similar cases with the
option of hot-swap PSUs - Chenbro used to be the main brand for this,
I think, but in the last couple of years TST <http://TSTcom.com> have
started producing nicer cases; I use a TST ESR-316, which is utterly
lush, but which was expensive. I have one slight reservation about the
TST, which I will not spend time detailing unless you ask.


I use only half the TST's capacity at present, but it is a pleasure
and a relief to have so much room available; expansion of network
drive capacity is never a problem - just slap a drive in and you're
ready to go. Even with as many as 6 or 8 drive bays there are corner
cases which can make expansion a bit of a headache (at least if uptime
is important).


Since these cases accommodate standard ATX motherboards, you get to
use an old Pentium 4 motherboard salvaged from an old PC or an Atom-
based motherboard for 100 or so. The latter price is a bit shocking,
IMO, compared to (say) the Asus EE-PC, but it reflects the demand for
them; they're prolly only $100 in the US. These atom motherboards have
minimal expansion slots, but if you only want to use it for storage
then you're probably fine with just one.


If you build your own server you can use software or hardware RAID.
Fast hardware RAID, based on an PCIe controller card, is expensive.
You can get PCI or PCI-X hardware RAID very cheaply on eBay these
days, but it's slow. That is to say that PCI or PCI-X hardware RAID is
fast enough to stream a couple of movies at the same time, fast enough
to copy 5gb files only a couple of minutes, but production server
systems (if you were buying a database server for work) would be
expected to use a PCIe-based hard-drive controller. Hardware RAID is
nice in its ability to hot-swap out a failed hard-drive without
interruption. I have not found non-RAID SATA controllers that satisfy
me with their ability to do hot-swap (although I would love to).


Managing RAID on a PC-based server - rather than a dedicated NAS
enclosure - very easily allows expansion. With RAID5 or 6 you can just
add in another drive and expand on to it. I use an old PCI-X (fits in
a PCI slot) 3ware 9500 card, and it *seems* like if you have a RAID1
(haven't tried RAID5) on two drives of capacity X, then remove each of
those drives in turn, rebuilding onto drives of X+Y capacity, then
upon completion the array appears to the o/s as the larger X+Y size. I
think some LSI cards do this, also. I would not bet on the ability of
low-end NAS boxes to do this.


A company called Drobo makes some high-end NAS hardware with space for
plenty of drives (on some models) and some fancy features. I find UK
prices a bit shocking, but depending upon your application they might
be justified; the US prices seem quite reasonable to me.


I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can
be faster than NFS, even in an all-Linux environment. Other times it's
not. This seems pretty much random, depending upon whom is doing the
benchmarking. On an intellectual level, at least, I find neither
wholly satisfying - it would be really nice to have a Linux-native
network filesystem that does authentication / permissions properly.
But both do work.


I looked at ZFS, but decided that Solaris, from a look at the HCL, was
too picky over hardware. I think ZFS is great, I no longer think it's
the future. My selection of cheap hardware is far wider under Linux, I
can install Gentoo and just `emerge mediatomb` and stream movies to my
PS3.


So there ya go. Lots of options, budget from dead cheap to mega money.
Depends how much you can justify.


Stroller.



[1] http://www.newegg.com/product/product.aspx?item=n82e16811219021

Steve 03-15-2010 06:18 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On 15/03/2010 18:21, Stroller wrote:
> It's hard to be more specific without knowing your usage.
Yes... I was deliberately vague to see what options came up... but I can
be more specific. The budget is miniscule - and the performance demands
(bandwidth and latency) are completely non-challenging. It's in this
context that I'm looking for reliability and availability... and I'd
like to have unix permissions working properly. Security is a moderate
concern - the physical network is secured - but there is a broadband
connection which exposes various services.

> For storage of a "mere terabyte" you can buy a networked storage
> enclosure which will accommodate two drives. These are cheap, do
> mirroring, will accommodate standard 1TB, 1.5TB, 2TB drives, but are
> probably not too fast.

A cheap NAS enclosure is a definite possibility - there'd be no
performance issue - though this leaves three key questions:
1) Will it support unix file-permissions and can I be (fairly sure) it
will be secure if someone hacks my Wi-Fi?
2) Will I be able to put the (majority of the) gentoo filesystem on it -
or will I need to have a fully booted system to connect?
3) Can I use two entirely separate devices and mirror to both? (I
expect the failure of the enclosure to be at least as likely as the
failure of a drive.)

> If you build your own server you can use software or hardware RAID.

Hmmm... building my own server - I've done that in the past, but my plan
is to minimize DIY with a view to minimizing the number of components
that might fail. Ideally, I'd have four devices - one with a CPU and
memory (the server)... booting from Flash or CD or whatever (+a
replacement in the cupboard); two separate boxes with drives in them
(mirrored storage); one (wired) Ethernet hub and broadband gateway. I'd
connect to the network from a separate desktop/laptop to interact with
it - either locally or remotely.

> I wouldn't get too het up about Samba / CIFS vs NFS. Samba / CIFS can
> be faster than NFS, even in an all-Linux environment. Other times it's
> not. This seems pretty much random, depending upon whom is doing the
> benchmarking. On an intellectual level, at least, I find neither
> wholly satisfying - it would be really nice to have a Linux-native
> network filesystem that does authentication / permissions properly.
> But both do work.

Well the 'server' will be running Samba - and it's the back-end storage
for that I'm trying to resolve. CIFS definitely looks problematic -
since Unix permissions for server data are one valuable separation
between publicly accessible services and my private data. NFS might be
OK (it doesn't "feel" great) - though I *really* don't want to move from
one server to two when I'm aiming for reliability.

> I looked at ZFS, but decided that Solaris, from a look at the HCL, was
> too picky over hardware. I think ZFS is great, I no longer think it's
> the future. My selection of cheap hardware is far wider under Linux, I
> can install Gentoo and just `emerge mediatomb` and stream movies to my
> PS3.
I like ZFS, conceptually, though I don't like Solaris. I'm aware that
Apple have toyed with adopting ZFS and that it is available for BSD... A
*really* neat solution would be a (pair of) cheap NAS devices running an
appliance distribution of BSD with ZFS - exporting a NFS mount...
possibly over a VPN? Hmmm - I'm trying to avoid complexity, too. Hmmm.

Andrea Conti 03-15-2010 09:29 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
Hi,

> The budget is miniscule - and the performance demands
> (bandwidth and latency) are completely non-challenging.

This IMHO pretty much rules out any kind of server-class hardware, which
tends to be both costly and power-hungry. If you're thinking about
buying used stuff, be sure to factor in the cost and difficulty of
finding spares in some years' time.

Given the point above I would also stick with software RAID. True HW
RAID controllers are quite expensive and generally come with a x8 PCIe
interface which will require a server motherboard -- x16 PCIe video card
slots in commodity boards are usually only certified for x16 and x1
operation, so don't expect them to work reliably with other bus widths.

Linux software RAID also has the advantage that the kernel is not tied
to any specific piece of hardware. In case of a failure, your volumes
will be readable on any other Linux system -- provided the disks
themselves are not toast.

If reliability is your primary concern, I would go for a simple RAID1
setup; if your volumes need to be bigger than a physical disk you can
build a spanned volume over multiple mirrored pairs. Network throughput
will mostly likely be your primary bottleneck, so I'd avoid striping as
it would offer little in the way of performace at the expense of making
data recovery extremely difficult in case the worst should happen.

As for availability, I think the best strategy with a limited budget is
to focus on reducing downtime: make sure your data can survive the
failure of any single component, and choose hardware that you can get
easily and for a reasonable price. Sh*t happens, so make it painless to
clean up.

Network protocol:

If you do not need data sharing (i.e. if your volumes are only mounted
by one client at a time), the simplest solution is to completely avoid
having a FS on the storage server side -- just export the raw block
device via iSCSI, and do everything on the client. In my experience this
also works very well with Windows clients using the free MS iSCSI initiator.

Alternatively, you can consider good old NFS, which performs decently
and tends to behave a bit better -- especially when used over UDP -- in
case of network glitches, like accidentally powering off a switch,
yanking cables, losing wireless connectivity...

CIFS should be avoided at all costs if your clients are not Windows
machines.

File systems: avoid complexity. As technically superior as it might be,
in this kind of setup ZFS is only going to be resource hog and a
maintenance headache; your priority should be having a rock-solid
implementation and a reliable set of diagnostic/repair tools in case
disaster strikes. Tried-and-true ext3 fits the bill nicely if you ask
me; just remember to tune it properly according to your planned use --
eg. if a volume is going to be used to host huge VM disk images, be sure
to create its filesystem with -T largefile4.

Just my 2 cents,

Andrea

Steve 03-16-2010 03:32 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On 15/03/2010 22:29, Andrea Conti wrote:
> This IMHO pretty much rules out any kind of server-class hardware, which
> tends to be both costly and power-hungry. If you're thinking about
> buying used stuff, be sure to factor in the cost and difficulty of
> finding spares in some years' time.
>
I'm considering neither used equipment nor 'server-class' - the workload
simply doesn't demand it.
> Given the point above I would also stick with software RAID.
...
> If reliability is your primary concern, I would go for a simple RAID1
> setup;
Absolutely. Software raid is cheaper and implies less hardware to
fail. Similarly, RAID1 minimises the total number of disks required to
survive a failure. It's the only way for me to go.
> If you do not need data sharing (i.e. if your volumes are only mounted
> by one client at a time), the simplest solution is to completely avoid
> having a FS on the storage server side -- just export the raw block
> device via iSCSI, and do everything on the client.
This idea is on my wavelength. Has anyone on this tried this? My
concerns are:

1. Are there reliability issues surrounding this technology in Gentoo?
2. Are there any howtos about putting as much of the file-system as
possible onto an iSCSI device.
3. What's the best (most lightweight) way to expose the disk as a block
device. I don't want to manage three fully-fledged Linux boxes. Can
(cheap) NAS devices be used to export iSCSI to Gentoo?
4. What would be the strategy to 'secure' this iSCSI device... it would
be a disaster if my WiFi were cracked and my data corrupted from a
non-authorised host.

> In my experience this also works very well with Windows clients using the free MS iSCSI initiator.
>
That's fantastic - I had no idea that such software existed. Now, I
wonder, what's the most lightweight solution to get a couple of iSCSI
devices? Does it help that MS supports attaching devices this way?
> File systems: avoid complexity. As technically superior as it might be,
> in this kind of setup ZFS is only going to be resource hog and a
> maintenance headache; your priority should be having a rock-solid
> implementation and a reliable set of diagnostic/repair tools in case
> disaster strikes.
Yes. Separate arguments for snapshot support are compelling... but there
are alternatives without tackling the additional complexity. That said,
the iSCSI approach would work as well with ZFS as something mundane.
Snap-shots, of course, are only really valuable for non-archive data...
so, in future, I could add a ZFS volume using the same iSCSI strategy.

Stroller 03-16-2010 06:57 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On 16 Mar 2010, at 16:32, Steve wrote:

...

Given the point above I would also stick with software RAID.

...

If reliability is your primary concern, I would go for a simple RAID1
setup;

Absolutely. Software raid is cheaper and implies less hardware to
fail. Similarly, RAID1 minimises the total number of disks required
to

survive a failure. It's the only way for me to go.


How does your system boot if your RAID1 system volume fails? The one
you have grub on? I think you mentioned a flash drive, which I've seen
mentioned before. This seems sound, but just to point out that's
another, different, single point of failure.


If you do not need data sharing (i.e. if your volumes are only
mounted
by one client at a time), the simplest solution is to completely
avoid

having a FS on the storage server side -- just export the raw block
device via iSCSI, and do everything on the client.

...
Snap-shots, of course, are only really valuable for non-archive
data...

so, in future, I could add a ZFS volume using the same iSCSI strategy.


I have wondered if it might be possible to create a large file (`dd
if=/dev/zero of=/path/to/large/file` constrain at a size of 20gig or
100gig or whatever) and treat it as a loopback device for stuff like
this. It's not true snapshotting (in the ZFS / BTFS sense), but you
can unmount it and make a copy quite quickly.


Stroller.

Neil Bothwick 03-16-2010 07:04 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:

> How does your system boot if your RAID1 system volume fails?

You put GRUB on both disks, then you can boot from either on its own.


--
Neil Bothwick

Growing old is mandatory; growing up is optional!!

Stroller 03-16-2010 07:13 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On 16 Mar 2010, at 20:04, Neil Bothwick wrote:


On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:


How does your system boot if your RAID1 system volume fails?


You put GRUB on both disks, then you can boot from either on its own.


Is this reliable? I don't contest it, I'm just asking. It's just this
was one of my considerations when choosing hardware RAID.


Stroller.

"J. Roeleveld" 03-16-2010 07:44 PM

Strategy for using SAN/NAS for storage with Gentoo...
 
On Tuesday 16 March 2010 21:13:29 Stroller wrote:
> On 16 Mar 2010, at 20:04, Neil Bothwick wrote:
> > On Tue, 16 Mar 2010 19:57:49 +0000, Stroller wrote:
> >> How does your system boot if your RAID1 system volume fails?
> >
> > You put GRUB on both disks, then you can boot from either on its own.
>
> Is this reliable? I don't contest it, I'm just asking. It's just this
> was one of my considerations when choosing hardware RAID.
>
> Stroller.
>

This is the generally recommended method and I found this method in the Gentoo
documentation. If this wouldn't be reliable, I would have expected this not to
be in the docs for long.

I have /boot mirrored, not actually tested with removing a disk yet, but I am
confident it will work.

If not, I have other methods of booting the system and getting to the data.

--
Joost


All times are GMT. The time now is 09:32 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.