FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 09-08-2012, 07:56 PM
Martin Steigerwald
 
Default Storage server

Am Samstag, 8. September 2012 schrieb Martin Steigerwald:
> > And, of course, thanks for your time and valuable advices, Stan, I've
> > read some of your previous posts on this list and know you're storage
> > guru.
>
> It wasn´t Stan who wrote the mail you replied to here, but yes I think
> I can learn a lot from him regarding storage setups, too.

Ah, forget this. Of course you answered to Stans mail with the good
questions in it and I replied to your reply to Stan a second time with
some more considerations and questions for you.

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201209082156.22556.Martin@lichtvoll.de">http://lists.debian.org/201209082156.22556.Martin@lichtvoll.de
 
Old 09-08-2012, 07:59 PM
Martin Steigerwald
 
Default Storage server

Am Samstag, 8. September 2012 schrieb Veljko:
> Well, it did sound a little to complex and that is why I posted to this
> list, hoping to hear some other opinions.
>
> 1. This machine will be used for
> a) backup (backup server for several dedicated (mainly) web servers).
> It will contain incremental backups, so only first running will take
> a lot of time, rsnapshot will latter download only changed/added files
> and will run from cron every day. Files that will be added later are
> around 1-10 MB in size. I expect ~20 GB daily, but that number can
> grow. Some files fill be deleted, other will be added.
> Dedicated servers that will be backed up are ~500GB in size.
> b) monitoring (Icinga or Zabbix) of dedicated servers.
> c) file sharing for employees (mainly small text files). I don't
> expect this to be resource hog.
> d) Since there is enough space (for now), and machine have four cores
> and 4GB RAM (that can be easily increased), I figured I can use it
> for test virtual machines. I usually work with 300MB virtual machines
> and no intensive load. Just testing some software.

Could it be that you intend to provide hosted monitoring, backup and
fileservices for an customer and while at it use the same machine for
testing own stuff?

If so: Don´t.

Thats at least my advice. (In addition to what I wrote already.)

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201209082159.36172.Martin@lichtvoll.de">http://lists.debian.org/201209082159.36172.Martin@lichtvoll.de
 
Old 09-09-2012, 06:37 AM
Stan Hoeppner
 
Default Storage server

On 9/7/2012 3:16 PM, Bob Proulx wrote:

> Agreed. But for me it isn't about the fsck time. It is about the
> size of the problem. If you have full 100G filesystem and there is a
> problem then you have a 100G problem. It is painful. But you can
> handle it. If you have a full 10T filesystem and there is a problem
> then there is a *HUGE* problem. It is so much more than painful.

This depends entirely on the nature of the problem. Most filesystem
problems are relatively easy to fix even on 100TB+ filesystems,
sometimes with some data loss, often with only a file or few being lost
or put in lost+found. If you have a non-redundant hardware device
failure that roasts your FS, then you replace the hardware make a new
FS, and restore from D2D or tape. That's not painful, that's procedure.

> Therefore when practical I like to compartmentalize things so that
> there is isolation between problems. Whether the problem is due to a
> hardware failure, a software failure or a human failure. All of which
> are possible. Having compartmentalization makes dealing with the
> problem easier and smaller.

Sounds like you're mostly trying to mitigate human error. When you
identify that solution, let me know, then patent it.

>> Whjat? Are you talking crash recovery boot time "fsck"? With any
>> modern journaled FS log recovery is instantaneous. If you're talking
>> about an actual structure check, XFS is pretty quick regardless of inode
>> count as the check is done in parallel. I can't speak to EXTx as I
>> don't use them.
>
> You should try an experiment and set up a terabyte ext3 and ext4
> filesystem and then perform a few crash recovery reboots of the
> system. It will change your mind. :-)

As I've never used EXT3/4 and thus have no opinion, it'd be a bit
difficult to change my mind. That said, putting root on a 1TB
filesystem is a brain dead move, regardless of FS flavor. A Linux
server doesn't need more than 5GB of space for root. With /var, /home/
and /bigdata on other filesystems, crash recovery fsck should be quick.

> XFS has one unfortunate missing feature. You can't resize a
> filesystem to be smaller. You can resize them larger. But not
> smaller. This is a missing feature that I miss as compared to other
> filesystems.

If you ever need to shrink a server filesystem: "you're doing IT wrong".

> Unfortunately I have some recent FUD concerning xfs. I have had some
> recent small idle xfs filesystems trigger kernel watchdog timer
> recoveries recently. Emphasis on idle.

If this is the bug I'm thinking of, "Idle" has nothing to do with the
problem, which was fixed in 3.1 and backported to 3.0. The fix didn't
hit Debian 2.6.32. I'm not a Debian kernel dev, ask them why--likely
too old. Upgrading to the BPO 3.2 kernel should fix this and give you
some nice additional performance enhancements. 2.6.32 is ancient BTW,
released almost 3 years ago. That's 51 in Linux development years.

If you're going to recommend to someone against XFS, please
qualify/clarify that you're referring to 3 year old XFS, not the current
release.

> Definitely XFS can handle large filesystems. And definitely when
> there is a good version of everything all around it has been a very
> good and reliable performer for me. I wish my recent bad experiences
> were resolved.

The fix is quick and simple, install BPO 3.2. Why haven't you already?

> But for large filesystems such as that I think you need a very good
> and careful administrator to manage the disk farm. And that includes
> disk use policies as much as it includes managing kernel versions and
> disk hardware. Huge problems of any sort need more careful management.

Say I have a 1.7TB filesystem and a 30TB filesystem. How do you feel
the two should be managed differently, or that the 30TB filesystem needs
kid gloves?

>> When using correctly architected reliable hardware there's no reason one
>> can't use a single 500TB XFS filesystem.
>
> Although I am sure it would work I would hate to have to deal with a
> problem that large when there is a need for disaster recovery. I
> guess that is why *I* don't manage storage farms that are that large. :-)

The only real difference at this scale is that your backup medium is
tape, not disk, and you have much phatter pipes to the storage host. A
500TB filesystem will reside on over 1000 disk drives. It isn't going
to be transactional or primary storage, but nearline or archival
storage. It takes a tape silo and intelligent software to back it up,
but a full restore after catastrophe doesn't have (many) angry users
breathing down your neck.

On the other hand, managing a 7TB transactional filesystem residing on
48x 300GB SAS drives in a concatenated RADI10 setup, housing, say,
corporate mailboxes for 10,000 employees, including the CxOs, is a much
trickier affair. If you wholesale lose this filesystem and must do a
full restore, you are red meat, and everyone is going to take a bite out
of your ass. And you very well may get a pink slip depending on the
employer.

Size may matter WRT storage/filesystem management, but it's the type of
data you're storing and the workload that's more meaningful.

--
Stan



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504C3922.30201@hardwarefreak.com">http://lists.debian.org/504C3922.30201@hardwarefreak.com
 
Old 09-09-2012, 08:42 AM
Stan Hoeppner
 
Default Storage server

On 9/8/2012 11:49 AM, Veljko wrote:

> Well, it did sound a little to complex and that is why I posted to this
> list, hoping to hear some other opinions.
>
> 1. This machine will be used for
> a) backup (backup server for several dedicated (mainly) web servers).
> It will contain incremental backups, so only first running will take a
> lot of time, rsnapshot will latter download only changed/added files
> and will run from cron every day. Files that will be added later are
> around 1-10 MB in size. I expect ~20 GB daily, but that number can
> grow. Some files fill be deleted, other will be added.
> Dedicated servers that will be backed up are ~500GB in size.
> b) monitoring (Icinga or Zabbix) of dedicated servers.
> c) file sharing for employees (mainly small text files). I don't
> expect this to be resource hog.

Stop here. Never use a production system as a test rig.

> d) Since there is enough space (for now), and machine have four cores
> and 4GB RAM (that can be easily increased), I figured I can use it for
> test virtual machines. I usually work with 300MB virtual machines and
> no intensive load. Just testing some software.

You can build a complete brand new AMD dedicated test machine with parts
from Newegg for $238 USD, sans KB/mouse/monitor, which you already have.
Boot it up then run it headless, use a KVM switch, etc.

http://www.newegg.com/Product/Product.aspx?Item=N82E16813186189
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148262
http://www.newegg.com/Product/Product.aspx?Item=N82E16819103888
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136771
http://www.newegg.com/Product/Product.aspx?Item=N82E16827106289
http://www.newegg.com/Product/Product.aspx?Item=N82E16811121118

If ~$250 stretches the wallet of your employer, it's time for a new job.

> 2. There is no fixed implementation date, but I'm expected to start
> working on it. Sooner the better, but no dead lines.
> Equipment I have to work with is desktop class machine: Athlon X4,
> 4GB RAM and 4 3TB Seagate ST3000DM001 7200rpm. Server will be in my
> office and will perform backup over internet. I do have APC UPS to
> power off machine in case of power loss (apcupsd will take care of
> that).

Get yourself an Adaptec 8 port PCIe x8 RAID card kit for $250:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816103231

The Seagate ST3000DM001 is certified. It can't do RAID5 so you'll use
RAID10, giving you 6TB of raw capacity, but much better write
performance than RAID5. You can add 4 more of these drives, doubling
capacity to 12TB. Comes with all cables, manuals, etc. Anyone who has
tried to boot a server after the BIOS configured boot drive that is
mdraid mirrored knows why $250 is far more than worth the money. A
drive failure with a RAID card doesn't screw up your boot order. It
just works.

> In next few months it is expected that size of files on dedicated
> servers will grow and it case that really happen I'd like to be able to
> expand this system.

See above.

> And, of course, thanks for your time and valuable advices, Stan, I've read
> some of your previous posts on this list and know you're storage guru.

You're welcome. And thank you. Recommending the above Adaptec card
is the best advice you'll get. It'll make your life much easier, with
better performance to boot.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504C5664.8060206@hardwarefreak.com">http://lists.debian.org/504C5664.8060206@hardwarefreak.com
 
Old 09-09-2012, 08:51 AM
Stan Hoeppner
 
Default Storage server

On 9/8/2012 1:10 PM, Martin Steigerwald wrote:
> Am Freitag, 7. September 2012 schrieb Stan Hoeppner:
>> On 9/7/2012 12:42 PM, Dan Ritter wrote:
> […]
>>> Now, the next thing: I know it's tempting to make a single
>>> filesystem over all these disks. Don't. The fsck times will be
>>> horrendous. Make filesystems which are the size you need, plus a
>>> little extra. It's rare to actually need a single gigantic fs.
>>
>> Whjat? Are you talking crash recovery boot time "fsck"? With any
>> modern journaled FS log recovery is instantaneous. If you're talking
>> about an actual structure check, XFS is pretty quick regardless of
>> inode count as the check is done in parallel. I can't speak to EXTx
>> as I don't use them. For a multi terabyte backup server, XFS is the
>> only way to go anyway. Using XFS also allows infinite growth without
>> requiring array reshapes nor LVM, while maintaining striped write
>> alignment and thus maintaining performance.
>>
>> There are hundreds of 30TB+ and dozens of 100TB+ XFS filesystems in
>> production today, and I know of one over 300TB and one over 500TB,
>> attached to NASA's two archival storage servers.
>>
>> When using correctly architected reliable hardware there's no reason
>> one can't use a single 500TB XFS filesystem.
>
> I assume that such correctly architected hardware contains a lot of RAM in
> order to be able to xfs_repair the filesystem in case of any filesystem
> corruption.
>
> I know RAM usage of xfs_repair has been lowered, but still such a 500 TiB
> XFS filesystem can contain a lot of inodes.

The system I've been referring to with the ~500TB XFS is an IA64 SGI
Altix with 64P and 128GB RAM. I'm pretty sure 128GB is plenty for
xfs_repair on filesystems much larger than 500TB.

> But for upto 10 TiB XFS filesystem I wouldn´t care too much about those
> issues.

Yeah, an 8GB machine typically allows for much larger than 10TB xfs_repair.

--
Stan



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504C587B.90908@hardwarefreak.com">http://lists.debian.org/504C587B.90908@hardwarefreak.com
 
Old 09-09-2012, 09:09 AM
Stan Hoeppner
 
Default Storage server

On 9/8/2012 2:53 PM, Martin Steigerwald wrote:

> I would love to learn more about those really big XFS installations and
> how there were made. I never dealt with more than about 4 TiB big XFS
> setups.

About the only information that's still available is at the link below,
and it lacks configuration details. I read those long ago when this
system was fresh. The detailed configuration info has since disappeared.

http://www.nas.nasa.gov/hecc/resources/storage_systems.html

NAS cannibalized the Columbia super quite some time ago and recycled
some nodes into these archive systems (and other systems) after they
installed the big Pleiades cluster and the users flocked to it. A bit
of a shame as Columbia had 60TF capability, and for shared memory
applications to boot. No system on earth had that capability until SGI
released the Altix UV, albeit with half the sockets/node of the IA64
Altix machines.

The coolest part about both 512P IA64 and 256P x86-64 Altix? Debian
will install and run with little to no modifications required, just as
shrink wrapped SLES and RHEL run out of the box, thanks to the Linux
Scalability Effort in the early 2000s.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504C5CB7.9060701@hardwarefreak.com">http://lists.debian.org/504C5CB7.9060701@hardwarefreak.com
 
Old 09-09-2012, 08:25 PM
Paul E Condon
 
Default Storage server

On 20120909_040911, Stan Hoeppner wrote:
> On 9/8/2012 2:53 PM, Martin Steigerwald wrote:
>
> > I would love to learn more about those really big XFS installations and
> > how there were made. I never dealt with more than about 4 TiB big XFS
> > setups.
>
> About the only information that's still available is at the link below,
> and it lacks configuration details. I read those long ago when this
> system was fresh. The detailed configuration info has since disappeared.
>
> http://www.nas.nasa.gov/hecc/resources/storage_systems.html
>
> NAS cannibalized the Columbia super quite some time ago and recycled
> some nodes into these archive systems (and other systems) after they
> installed the big Pleiades cluster and the users flocked to it. A bit
> of a shame as Columbia had 60TF capability, and for shared memory
> applications to boot. No system on earth had that capability until SGI
> released the Altix UV, albeit with half the sockets/node of the IA64
> Altix machines.
>
> The coolest part about both 512P IA64 and 256P x86-64 Altix? Debian
> will install and run with little to no modifications required, just as
> shrink wrapped SLES and RHEL run out of the box, thanks to the Linux
> Scalability Effort in the early 2000s.
>
> --
> Stan

Stan,

I've been following this thread from its beginning. My initial reading
of OP's post was to marvel at the thought that so many things/tasks
could be done with a single box in a single geek's cubicle. I resolved
to follow the thread that would surely follow closely. I think you,
Stan, did OP an enormous service with your list of questions to be
answered.

This thread drifted onto the topic of XFS. I first learned of the
existence of XFS from earlier post by you, and I have ever since been
curious about it. But I am retired, and live at home in an environment
where there is very little opportunity to make use of its features.
Perhaps you could take OP's original specification as a user wish list
and sketch a design that would fulfill the wishlist and list how XFS
would change or resolve issues that were/are troubling him.

In particular, the typical answers to questions about backup on this list
involve rsync, or packages that depend on rsync, and on having a file
system that uses inodes and supports hard links. How would an XFS design
handle "de-duplication"? Or is de-duplication simply a bad idea in very
large systems?

Sincerely,
--
Paul E Condon
pecondon@mesanetworks.net


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120909202535.GA3164@big.lan.gnu">http://lists.debian.org/20120909202535.GA3164@big.lan.gnu
 
Old 09-10-2012, 10:37 AM
Stan Hoeppner
 
Default Storage server

On 9/9/2012 3:25 PM, Paul E Condon wrote:

> I've been following this thread from its beginning. My initial reading
> of OP's post was to marvel at the thought that so many things/tasks
> could be done with a single box in a single geek's cubicle.

One consumer quad core AMD Linux box of today can do a whole lot more
than what has been mentioned.

> I resolved
> to follow the thread that would surely follow closely. I think you,
> Stan, did OP an enormous service with your list of questions to be
> answered.

I try to prevent other from shooting themselves in the foot when I see
the loaded gun in their hand.

> This thread drifted onto the topic of XFS. I first learned of the
> existence of XFS from earlier post by you, and I have ever since been
> curious about it. But I am retired, and live at home in an environment
> where there is very little opportunity to make use of its features.

You might be surprised. The AG design and xfs_fsr make it useful for
home users.

> Perhaps you could take OP's original specification as a user wish list
> and sketch a design that would fulfill the wishlist and list how XFS
> would change or resolve issues that were/are troubling him.

The OP's issues don't revolve around filesystem choice, but basic system
administration concepts.

> In particular, the typical answers to questions about backup on this list
> involve rsync, or packages that depend on rsync, and on having a file
> system that uses inodes and supports hard links.

rsync works with any filesystem, but some work better with rsync
workloads. If one has concurrent rsync jobs running XFS is usually best.

> How would an XFS design
> handle "de-duplication"?

Deduplication isn't an appropriate function of a filesystem.

> Or is de-duplication simply a bad idea in very
> large systems?

That's simply a bad, very overly broad question.

--
Stan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 504DC2FA.1080309@hardwarefreak.com">http://lists.debian.org/504DC2FA.1080309@hardwarefreak.com
 
Old 09-10-2012, 10:44 AM
Veljko
 
Default Storage server

On Sat, Sep 08, 2012 at 09:59:35PM +0200, Martin Steigerwald wrote:
> Could it be that you intend to provide hosted monitoring, backup and
> fileservices for an customer and while at it use the same machine for
> testing own stuff?
>
> If so: Don´t.
>
> Thats at least my advice. (In addition to what I wrote already.)
>
> --
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7

No, I'm not doing this for an customer, but for my boss. Someone with
idea like the one you implied to me has no place in job like this, I'm
sure you'll agree.

Regards,
Veljko


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120910104403.GA9096@angelina.example.org">http://lists.debian.org/20120910104403.GA9096@angelina.example.org
 
Old 09-10-2012, 10:45 AM
Veljko
 
Default Storage server

On Sat, Sep 08, 2012 at 09:28:09PM +0200, Martin Steigerwald wrote:
> Consider the consequenzes:
>
> If the server fails, you possibly wouldn´t know why cause the monitoring
> information wouldn´t be available anymore. So at least least Nagios /
> Icingo send out mails, in case these are not stored on the server as well,
> or let it relay the information to another Nagios / Icinga instance.

Ideally, Icinga/Nagios/any server would be on HA system but that,
unfortunately is not an option. But of course, Icinga can't monitor
system it's on, so I plan to monitor it from my own machine.

> What data do you backup? From where does it come?

Like I said, it's several dedicated, mostly web servers with users
uploaded content on one of them (that part is expected to grow). None of
them is in the same data center.

> I still think backup should be separate from other stuff. By design.
> Well for more fact based advice we´d require a lot more information on
> your current setup and what you want to achieve.
>
> I recommend to have a serious talk about acceptable downtimes and risks
> for the backup with the customer if you serve one or your boss if you work
> for one.

I talked to my boss about it. Since this is backup server, downtime is
acceptable to him. Regarding risks of data loss, isn't that the reason
to implement RAID configuration? "R" stands for redundancy. If hard disk
fails, it will be replaced and RAID will be rebuild with no data loss.
If processor or something else fails, it will be replaced with expected
downtime of course.

> --
> Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
> GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


Regards,
Veljko


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120910104516.GB9096@angelina.example.org">http://lists.debian.org/20120910104516.GB9096@angelina.example.org
 

Thread Tools




All times are GMT. The time now is 01:25 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org