Linux Archive - Storage server

Linux Archive (http://www.linux-archive.org/)

- Debian User (http://www.linux-archive.org/debian-user/)

- - Storage server (http://www.linux-archive.org/debian-user/701467-storage-server.html)

Hi!

I'm in the process of making new backup server, so I'm thinking of best
way of doing it. I have 4 3TB disks and I'm thinking of puting them in
software RAID10.

I created 2 500MB partitions for /boot (RAID1) and the rest it RAID10.

Next step is LVM, so I can expand storage if necessary. What is the best
way to do it? One VG and then create LV for root, swap, storage space or
I should create 2 VG, one for system and other one for storage space?

Every LV will have enough space for few months, but I wouldn't allocate
all available space, saving it for LV that will actualy need it.

I would put monitoring and file sharing server here as well, so there
will be LV for them too.

LVM will provide me a way to expand storage with extra disks, but if
there is no more room for that kind of expansion, I was thinking of
GlusterFS for scaling-out.

I would use KVM virtual machines (another LV for virtual disks) for
client part that would use network file system from host machine and
when need for more storage servers arise, I would just build another
similar storage server and add it to GFS I'm already using. I'm not sure
though if it is possible to use GFS on a just single server.

Inside of virtual machine will be rsnapshot, that would do actual backup
job.

I was thinking of puting samba/FTP and monitoring server on virtual
machines too (and some others for testing and that kind of things). They
would also use network storage provided by host machine.

What do you think of this setup? Good sides? Bad sides of this approach?

OS I would use is Wheezy. Guess he will be stable soon enough and I
don't want to reinstall everything again in one year, when support for
old stable is dropped.

Regards,
Veljko

P.S. Sorry for my bad english, I hope you understood what I was trying
to say. ;)

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120907162935.GA22966@angelina.example.org">http://lists.debian.org/20120907162935.GA22966@angelina.example.org

On Fri, Sep 07, 2012 at 06:29:35PM +0200, Veljko wrote:
>
> Hi!
>
> I'm in the process of making new backup server, so I'm thinking of best
> way of doing it. I have 4 3TB disks and I'm thinking of puting them in
> software RAID10.
>
> I created 2 500MB partitions for /boot (RAID1) and the rest it RAID10.

So far, so good.

> LVM will provide me a way to expand storage with extra disks, but if
> there is no more room for that kind of expansion, I was thinking of
> GlusterFS for scaling-out.

Let me suggest a different approach. It sounds like you're
planning on a lot of future expansion.

Get a high-end SAS RAID card. One with two external SFF8088
connectors.

When you start running out of places to put disks, buy external
chassis that take SFF8088 and have daisy-chaining ports. 2U
boxes often hold 12 3.5" disks.

You can put cheap SATA disks in, instead of expensive SAS disks.
The performance may not be as good, but I suspect you are
looking at sheer capacity rather than IOPS.

Now, the next thing: I know it's tempting to make a single
filesystem over all these disks. Don't. The fsck times will be
horrendous. Make filesystems which are the size you need, plus a
little extra. It's rare to actually need a single gigantic fs.

> OS I would use is Wheezy. Guess he will be stable soon enough and I
> don't want to reinstall everything again in one year, when support for
> old stable is dropped.

This is Debian. Since 1997 or so, you have had the ability to
upgrade from major version n to version n+1 without
reinstalling. You won't need to reinstall unless you change
architectures (i.e. from x86_32 to x86_64).

-dsr-

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120907174232.GY4974@randomstring.org">http://lists.debian.org/20120907174232.GY4974@randomstring.org

On 9/7/2012 11:29 AM, Veljko wrote:

> I'm in the process of making new backup server, so I'm thinking of best
> way of doing it. I have 4 3TB disks and I'm thinking of puting them in
> software RAID10.

["what if" stream of consciousness rambling snipped for brevity]

> What do you think of this setup? Good sides? Bad sides of this approach?

Applying the brakes...

As with many tech geeks with too much enthusiasm for various tools and
not enough common sense and seasoning, you've made the mistake of
approaching this backwards. Always start here:

1. What are the requirements of the workload?
2. What is my budget and implementation date?
3. How can I accomplish #1 given #2 with the
4. Least complexity and
5. Highest reliability and
6. Easiest recovery if the system fails?

You've described a dozen or so overly complex technical means to some
end that tend to violate #4 through #6.

Slow down, catch your breath, and simply describe #1 and #2. We'll go
from there.

--
Stan

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504A3C45.1050303@hardwarefreak.com">http://lists.debian.org/504A3C45.1050303@hardwarefreak.com

On Fri, Sep 07, 2012 at 01:42:32PM -0400, Dan Ritter wrote:
> On Fri, Sep 07, 2012 at 06:29:35PM +0200, Veljko wrote:
> >
> > Hi!
> >
> > I'm in the process of making new backup server, so I'm thinking of best
> > way of doing it. I have 4 3TB disks and I'm thinking of puting them in
> > software RAID10.
> >
> > I created 2 500MB partitions for /boot (RAID1) and the rest it RAID10.
>
> So far, so good.
>
> > LVM will provide me a way to expand storage with extra disks, but if
> > there is no more room for that kind of expansion, I was thinking of
> > GlusterFS for scaling-out.
>
> Let me suggest a different approach. It sounds like you're
> planning on a lot of future expansion.

Well, it is possible that this will be quite enough for a long time, but
I just want to be able to expand it if necessary.

>
> Get a high-end SAS RAID card. One with two external SFF8088
> connectors.
>
> When you start running out of places to put disks, buy external
> chassis that take SFF8088 and have daisy-chaining ports. 2U
> boxes often hold 12 3.5" disks.
>
> You can put cheap SATA disks in, instead of expensive SAS disks.
> The performance may not be as good, but I suspect you are
> looking at sheer capacity rather than IOPS.

I failed to provide enough information. It's desktop class machine in
desktop case. Hardware I have is what I must use, no space for
additional hardware. I can only use on board SATA connectors.

Since it's basic usage will be backup server, performance is not most
important thing, but it will contain some virtual machines (with modest
usage though) so RAID10 will provide quite sufficient IO performance.

> Now, the next thing: I know it's tempting to make a single
> filesystem over all these disks. Don't. The fsck times will be
> horrendous. Make filesystems which are the size you need, plus a
> little extra. It's rare to actually need a single gigantic fs.

Yes, I was thinking the same thing. Just a little extra then already
used.

> > OS I would use is Wheezy. Guess he will be stable soon enough and I
> > don't want to reinstall everything again in one year, when support for
> > old stable is dropped.
>
> This is Debian. Since 1997 or so, you have had the ability to
> upgrade from major version n to version n+1 without
> reinstalling. You won't need to reinstall unless you change
> architectures (i.e. from x86_32 to x86_64).
>
> -dsr-

But, isn't complete reintall safest way? Dist-upgrade can go wrong
sometime.

Regards,
Veljko

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120907182948.GA29042@angelina.example.org">http://lists.debian.org/20120907182948.GA29042@angelina.example.org

On 9/7/2012 12:42 PM, Dan Ritter wrote:

> You can put cheap SATA disks in, instead of expensive SAS disks.
> The performance may not be as good, but I suspect you are
> looking at sheer capacity rather than IOPS.

Stick with enterprise quality SATA disks. Throwing "drive of the week"
consumer models, i.e. WD20EARS, in the chassis simply causes unnecessary
heartache down the road.

> Now, the next thing: I know it's tempting to make a single
> filesystem over all these disks. Don't. The fsck times will be
> horrendous. Make filesystems which are the size you need, plus a
> little extra. It's rare to actually need a single gigantic fs.

Whjat? Are you talking crash recovery boot time "fsck"? With any
modern journaled FS log recovery is instantaneous. If you're talking
about an actual structure check, XFS is pretty quick regardless of inode
count as the check is done in parallel. I can't speak to EXTx as I
don't use them. For a multi terabyte backup server, XFS is the only way
to go anyway. Using XFS also allows infinite growth without requiring
array reshapes nor LVM, while maintaining striped write alignment and
thus maintaining performance.

There are hundreds of 30TB+ and dozens of 100TB+ XFS filesystems in
production today, and I know of one over 300TB and one over 500TB,
attached to NASA's two archival storage servers.

When using correctly architected reliable hardware there's no reason one
can't use a single 500TB XFS filesystem.

--
Stan

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504A41C1.7040702@hardwarefreak.com">http://lists.debian.org/504A41C1.7040702@hardwarefreak.com

Veljko wrote:
> Dan Ritter wrote:
> > > OS I would use is Wheezy. Guess he will be stable soon enough and I
> > > don't want to reinstall everything again in one year, when support for
> > > old stable is dropped.
> >
> > This is Debian. Since 1997 or so, you have had the ability to
> > upgrade from major version n to version n+1 without
> > reinstalling. You won't need to reinstall unless you change
> > architectures (i.e. from x86_32 to x86_64).
>
> But, isn't complete reintall safest way? Dist-upgrade can go wrong
> sometime.

If you follow the release notes there is no reason you shouldn't be
able to upgrade from one major release to another. I have systems
that started out as Woody that are currently running Squeeze.
Upgrades work great. Debian, unlike some other distros, is all about
being able to successfully upgrade. Upgrades work just fine. I have
upgraded many systems and will be upgrading many more.

But it is important to follow the release notes for the upgrade for
each major release because there is special handling needed and it
will be fully documented.

Sometimes this special handling annoys me because the required manual
cleanup mostly seems unnecessary to me if the packaging was done
better. I would call them bugs. But regardless of the small
packaging issues here and there the overall system upgrades just fine.

Bob

On 9/7/2012 2:43 PM, Bob Proulx wrote:
> I have systems
> that started out as Woody that are currently running Squeeze.

The Postfix box which relayed this email started at Potato and is
running Squeeze. Only a couple of minor hiccups during dist-upgrades,
mostly due to my failure to read the release notes thoroughly on occasion.

--
Stan

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504A5497.9030209@hardwarefreak.com">http://lists.debian.org/504A5497.9030209@hardwarefreak.com

Stan Hoeppner wrote:
> Dan Ritter wrote:
> > You can put cheap SATA disks in, instead of expensive SAS disks.
> > The performance may not be as good, but I suspect you are
> > looking at sheer capacity rather than IOPS.
>
> Stick with enterprise quality SATA disks. Throwing "drive of the week"
> consumer models, i.e. WD20EARS, in the chassis simply causes unnecessary
> heartache down the road.

There are inexpensive disks and then there are cheap disks. Those
"green" drives are in definitely the "cheap" category. Much harder to
deal with than the "inexpensive" category. I think there are three
big classifications of drives. Good quality enterprise drives.
Inexpensive drives. Cheap drives. The cheap drives are really
terrible!

> > Now, the next thing: I know it's tempting to make a single
> > filesystem over all these disks. Don't. The fsck times will be
> > horrendous. Make filesystems which are the size you need, plus a
> > little extra. It's rare to actually need a single gigantic fs.

Agreed. But for me it isn't about the fsck time. It is about the
size of the problem. If you have full 100G filesystem and there is a
problem then you have a 100G problem. It is painful. But you can
handle it. If you have a full 10T filesystem and there is a problem
then there is a *HUGE* problem. It is so much more than painful.

Therefore when practical I like to compartmentalize things so that
there is isolation between problems. Whether the problem is due to a
hardware failure, a software failure or a human failure. All of which
are possible. Having compartmentalization makes dealing with the
problem easier and smaller.

> Whjat? Are you talking crash recovery boot time "fsck"? With any
> modern journaled FS log recovery is instantaneous. If you're talking
> about an actual structure check, XFS is pretty quick regardless of inode
> count as the check is done in parallel. I can't speak to EXTx as I
> don't use them.

You should try an experiment and set up a terabyte ext3 and ext4
filesystem and then perform a few crash recovery reboots of the
system. It will change your mind. :-)

> For a multi terabyte backup server, XFS is the only way to go
> anyway. Using XFS also allows infinite growth without requiring
> array reshapes nor LVM, while maintaining striped write alignment
> and thus maintaining performance.

I agree that XFS is a superior filesystem for large filesystems. I
have used it there for years.

XFS has one unfortunate missing feature. You can't resize a
filesystem to be smaller. You can resize them larger. But not
smaller. This is a missing feature that I miss as compared to other
filesystems.

Unfortunately I have some recent FUD concerning xfs. I have had some
recent small idle xfs filesystems trigger kernel watchdog timer
recoveries recently. Emphasis on idle. Active filesystems are always
fine. I used /tmp as a large xfs filesystem but swapped it to be ext4
due to these lockups. Squeeze. Everything current. But when idle it
would periodically lock up and the only messages in the syslog and on
the system console were concerning xfs threads timed out. When the
kernel froze it always had these messages displayed[1]. It was simply
using /tmp as a hundred gig or so xfs filesystem. Doing nothing but
changing /tmp from xfs to ext4 resolved the problem and it hasn't seen
a kernel lockup since. I saw that problem on three different machines
but effectively all mine and very similar software configurations.
And by kernel lockup I mean unresponsive and it took a power cycle to
free it.

I hesitated to say anything because of lacking real data but it means
I can't completely recommend xfs today even though I have given it
strong recommendations in the past. I am thinking that recent kernels
are not completely clean specifically for idle xfs filesystems.
Meanwhile active ones seem to be just fine. Would love to have this
resolved one way or the other so I could go back to recommending xfs
again without reservations.

> There are hundreds of 30TB+ and dozens of 100TB+ XFS filesystems in
> production today, and I know of one over 300TB and one over 500TB,
> attached to NASA's two archival storage servers.

Definitely XFS can handle large filesystems. And definitely when
there is a good version of everything all around it has been a very
good and reliable performer for me. I wish my recent bad experiences
were resolved.

But for large filesystems such as that I think you need a very good
and careful administrator to manage the disk farm. And that includes
disk use policies as much as it includes managing kernel versions and
disk hardware. Huge problems of any sort need more careful management.

> When using correctly architected reliable hardware there's no reason one
> can't use a single 500TB XFS filesystem.

Although I am sure it would work I would hate to have to deal with a
problem that large when there is a need for disaster recovery. I
guess that is why *I* don't manage storage farms that are that large. :-)

Bob

[1] Found an old log trace. Stock Squeeze. Everything current. /tmp
was the only xfs filesystem on the machine. Most of the time the
recovery would work fine. But whenever the machine was locked up
frozen this was always displayed on the console. Doing nothing
but replacing xfs /tmp with ext4 /tmp and the system freeze
problem disappeared. I could put it back and see if the kernel
freeze reappears but I don't want to.

May 21 09:05:38 fs kernel: [3865560.844047] INFO: task xfssyncd:1794 blocked for more than 120 seconds.
May 21 09:05:38 fs kernel: [3865560.925322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 21 09:05:38 fs kernel: [3865561.021188] xfssyncd D 00000010 0 1794 2 0x00000000
May 21 09:05:38 fs kernel: [3865561.021204] f618d940 00000046 f6189980 00000010 f8311020 c13fc000 c13fc000 c13f7604
May 21 09:05:38 fs kernel: [3865561.021234] f618dafc c3508000 00000001 00036b3f 00000000 00000000 c3554700 d21b3280
May 21 09:05:38 fs kernel: [3865561.021265] c3503604 f618dafc 3997fc05 c35039c8 c106ef0b 00000000 00000000 00000000
May 21 09:05:38 fs kernel: [3865561.021295] Call Trace:
May 21 09:05:38 fs kernel: [3865561.021326] [<c106ef0b>] ? rcu_process_gp_end+0x27/0x63
May 21 09:05:38 fs kernel: [3865561.021339] [<c125d891>] ? schedule_timeout+0x20/0xb0
May 21 09:05:38 fs kernel: [3865561.021352] [<c1132b9b>] ? __lookup_tag+0x8e/0xee
May 21 09:05:38 fs kernel: [3865561.021362] [<c125d79a>] ? wait_for_common+0xa4/0x100
May 21 09:05:38 fs kernel: [3865561.021374] [<c102daad>] ? default_wake_function+0x0/0x8
May 21 09:05:38 fs kernel: [3865561.021405] [<f8bad6d2>] ? xfs_reclaim_inode+0xca/0x117 [xfs]
May 21 09:05:38 fs kernel: [3865561.021425] [<f8bade3c>] ? xfs_inode_ag_walk+0x44/0x73 [xfs]
May 21 09:05:38 fs kernel: [3865561.021445] [<f8bad71f>] ? xfs_reclaim_inode_now+0x0/0x4c [xfs]
May 21 09:05:38 fs kernel: [3865561.021465] [<f8badea1>] ? xfs_inode_ag_iterator+0x36/0x58 [xfs]
May 21 09:05:38 fs kernel: [3865561.021484] [<f8bad71f>] ? xfs_reclaim_inode_now+0x0/0x4c [xfs]
May 21 09:05:38 fs kernel: [3865561.021504] [<f8baded1>] ? xfs_reclaim_inodes+0xe/0x10 [xfs]
May 21 09:05:38 fs kernel: [3865561.021530] [<f8badef6>] ? xfs_sync_worker+0x23/0x5c [xfs]
May 21 09:05:38 fs kernel: [3865561.021549] [<f8bad901>] ? xfssyncd+0x134/0x17d [xfs]
May 21 09:05:38 fs kernel: [3865561.021569] [<f8bad7cd>] ? xfssyncd+0x0/0x17d [xfs]
May 21 09:05:38 fs kernel: [3865561.021580] [<c10441e0>] ? kthread+0x61/0x66
May 21 09:05:38 fs kernel: [3865561.021590] [<c104417f>] ? kthread+0x0/0x66
May 21 09:05:38 fs kernel: [3865561.021601] [<c1003d47>] ? kernel_thread_helper+0x7/0x10

>> I'm in the process of making new backup server, so I'm thinking of best>> way of doing it. I have 4 3TB disks and I'm thinking of puting them in>> software RAID10.
>>>> I created 2 500MB partitions for /boot (RAID1) and the rest it RAID10.>> So far, so good.>>> LVM will provide me a way to expand storage with extra disks, but if
>> there is no more room for that kind of expansion, I was thinking of>> GlusterFS for scaling-out.>> Let me suggest a different approach. It sounds like you're
> planning on a lot of future expansion.>> Get a high-end SAS RAID card. One with two external SFF8088> connectors.
I would 2nd the suggestion of investing in a high-end SAS RAID card. *I would also avoid any kind of software raid if at all possible, at least for partitions that see a lot of I/O. *I guess /boot would be ok, but I def would not put my root or /home under software raid if I had a discrete controller. *However, you did say this is a storage/backup server and not a "main" machine... so I don't know... just something to think I guess. *Software raid is free, so if performance becomes an issue you can always upgrade :)

There are a lot of discrete hard disk drive controllers on the market. *If you go this route, try to be sure that it supports SMART passthrough so that you can at least get _some_ kind status your drives.

--tdowg1

On Fri, Sep 07, 2012 at 01:26:13PM -0500, Stan Hoeppner wrote:
> On 9/7/2012 11:29 AM, Veljko wrote:
>
> > I'm in the process of making new backup server, so I'm thinking of best
> > way of doing it. I have 4 3TB disks and I'm thinking of puting them in
> > software RAID10.
>
> ["what if" stream of consciousness rambling snipped for brevity]
>
> > What do you think of this setup? Good sides? Bad sides of this approach?
>
> Applying the brakes...
>
> As with many tech geeks with too much enthusiasm for various tools and
> not enough common sense and seasoning, you've made the mistake of
> approaching this backwards. Always start here:
>
> 1. What are the requirements of the workload?
> 2. What is my budget and implementation date?
> 3. How can I accomplish #1 given #2 with the
> 4. Least complexity and
> 5. Highest reliability and
> 6. Easiest recovery if the system fails?
>
> You've described a dozen or so overly complex technical means to some
> end that tend to violate #4 through #6.
>
> Slow down, catch your breath, and simply describe #1 and #2. We'll go
> from there.
>
> --
> Stan

Well, it did sound a little to complex and that is why I posted to this
list, hoping to hear some other opinions.

1. This machine will be used for
a) backup (backup server for several dedicated (mainly) web servers).
It will contain incremental backups, so only first running will take a
lot of time, rsnapshot will latter download only changed/added files
and will run from cron every day. Files that will be added later are
around 1-10 MB in size. I expect ~20 GB daily, but that number can
grow. Some files fill be deleted, other will be added.
Dedicated servers that will be backed up are ~500GB in size.
b) monitoring (Icinga or Zabbix) of dedicated servers.
c) file sharing for employees (mainly small text files). I don't
expect this to be resource hog.
d) Since there is enough space (for now), and machine have four cores
and 4GB RAM (that can be easily increased), I figured I can use it for
test virtual machines. I usually work with 300MB virtual machines and
no intensive load. Just testing some software.

2. There is no fixed implementation date, but I'm expected to start
working on it. Sooner the better, but no dead lines.
Equipment I have to work with is desktop class machine: Athlon X4,
4GB RAM and 4 3TB Seagate ST3000DM001 7200rpm. Server will be in my
office and will perform backup over internet. I do have APC UPS to
power off machine in case of power loss (apcupsd will take care of
that).

In next few months it is expected that size of files on dedicated
servers will grow and it case that really happen I'd like to be able to
expand this system. Hardware RAID controllers are expensive and managers
always want to go with least expenses possible, so I'm stuck with
software RAID only.

But, one of dedicated server is slowly running out of space, so I don't
think they will go for cheapest option there. I'll have to take care of
that too, but first things first.

And, of course, thanks for your time and valuable advices, Stan, I've read
some of your previous posts on this list and know you're storage guru.

Rregards,
Veljko

--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120908164945.GA12245@angelina.example.org">http://lists.debian.org/20120908164945.GA12245@angelina.example.org