FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 09-27-2012, 06:15 PM
Keith Keller
 
Default 11TB ext4 filesystem - filesystem alternatives?

On 2012-09-27, John R Pierce <pierce@hogranch.com> wrote:
>
> XFS is fairly memory intensive. 11TB file systems tend to mean
> millions and millions of files.
>
> frankly, I wouldn't run this on CentOS 5.6, I would upgrade to CentOS
> 6.latest and then I would use XFS.... support for EXT4 and XFS is
> rather sketchy with the old kernel in 5.x (and why aren't you at 5.8 or
> whatever is current in the 5 series anyways?!?)

I have a ~20TB XFS filesystem on CentOS 5. Support for xfs in the
CentOS 5 kernels is now built-in, so you don't have to rely on the old
buggy XFS modules from centosplus. (I have yet to xfs_repair this
filesystem; I did repair it back when it was ~12TB, and it ran fine.)

I have also run xfs_repair on a 17TB XFS filesystem on a machine with
about 4GB of memory. It ran fine in less than one hour (~30m IIRC; that
filesystem is on CentOS 6).

I definitely agree that CentOS 6 is a better way to go, but XFS can be
done on CentOS 5 too. Just make sure you are completely up to date.

For the OP, what are the fsck times currently like for your ext4
filesystem? If they are already less than one hour, you may not see any
benefit from switching.

--keith

--
kkeller@wombat.san-francisco.ca.us


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-27-2012, 06:46 PM
John R Pierce
 
Default 11TB ext4 filesystem - filesystem alternatives?

On 09/27/12 11:15 AM, Keith Keller wrote:
> I have also run xfs_repair on a 17TB XFS filesystem on a machine with
> about 4GB of memory. It ran fine in less than one hour (~30m IIRC; that
> filesystem is on CentOS 6).

with XFS at least (and probably ext4) what counts is how many files are
in the file system more than the absolute size. if you have 17000 1gb
files, its one thing, if you have 17,000,000,000 1K files, its another
thing entirely.



--
john r pierce N 37, W 122
santa cruz ca mid-left coast

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-27-2012, 07:22 PM
Keith Keller
 
Default 11TB ext4 filesystem - filesystem alternatives?

On 2012-09-27, John R Pierce <pierce@hogranch.com> wrote:
> On 09/27/12 11:15 AM, Keith Keller wrote:
>> I have also run xfs_repair on a 17TB XFS filesystem on a machine with
>> about 4GB of memory. It ran fine in less than one hour (~30m IIRC; that
>> filesystem is on CentOS 6).
>
> with XFS at least (and probably ext4) what counts is how many files are
> in the file system more than the absolute size. if you have 17000 1gb
> files, its one thing, if you have 17,000,000,000 1K files, its another
> thing entirely.

Good point. Just for a data point, I've probably got on the order of a
few dozen million files of widely varying sizes on this particular
filesystem. (So more than 10 million files but fewer than 100 million.)

--keith


--
kkeller@wombat.san-francisco.ca.us


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-27-2012, 10:34 PM
SilverTip257
 
Default 11TB ext4 filesystem - filesystem alternatives?

Current CentOS 6 is 2.6.32, not 2.6.36

In that XFS Youtube video, Dave Chinner says upstream 3.0 kernel or
RHEL 6.2 [at 45:20 of the video].

Other sources [0] [1] agree.

[0] http://lwn.net/Articles/476616/
[1] http://jira.funtoo.org/browse/FL-38

---~~.~~---
Mike
// SilverTip257 //


On Thu, Sep 27, 2012 at 8:46 AM, SilverTip257 <silvertip257@gmail.com> wrote:
> Definitely shoot for CentOS 6.3 ...
>
> XFS with a kernel _more recent_ than 2.6.36 (currently shipped with
> CentOS6) has more improvements to the XFS code. Youtube video on XFS
> [0] - I believe the kernel version noted is 2.6.39 (watch the video!)
> [2].
>
> And there's also a Youtube video on BTRFS [1] that was linked
> to/shared by Fernando.
>
> [0] http://lists.centos.org/pipermail/centos/2012-August/128119.html
> [1] http://lists.centos.org/pipermail/centos/2012-August/128110.html
> [2] http://lwn.net/Articles/438671/
>
> ---~~.~~---
> Mike
> // SilverTip257 //
>
>
> On Thu, Sep 27, 2012 at 5:08 AM, John R Pierce <pierce@hogranch.com> wrote:
>> On 09/27/12 1:52 AM, Nux! wrote:
>>> Never had to deal with such a large filesystem, yet, but I'd try XFS on
>>> it.
>>
>> XFS is fairly memory intensive. 11TB file systems tend to mean
>> millions and millions of files.
>>
>> frankly, I wouldn't run this on CentOS 5.6, I would upgrade to CentOS
>> 6.latest and then I would use XFS.... support for EXT4 and XFS is
>> rather sketchy with the old kernel in 5.x (and why aren't you at 5.8 or
>> whatever is current in the 5 series anyways?!?)
>>
>>
>>
>>
>>
>> --
>> john r pierce N 37, W 122
>> santa cruz ca mid-left coast
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-28-2012, 07:09 PM
Ilyas --
 
Default 11TB ext4 filesystem - filesystem alternatives?

Hello,

One day our servers farm rebooted unexpectedly (power fail happened)
and on centos 6.3 with up2date kernel we lost few hundred files (which
probably was opened for reading, NOT writing) on XFS.

Unexpected power lost follow to situation when some files get a zero size.

On Fri, Sep 28, 2012 at 2:34 AM, SilverTip257 <silvertip257@gmail.com> wrote:
> Current CentOS 6 is 2.6.32, not 2.6.36
>
> In that XFS Youtube video, Dave Chinner says upstream 3.0 kernel or
> RHEL 6.2 [at 45:20 of the video].
>
> Other sources [0] [1] agree.
>
> [0] http://lwn.net/Articles/476616/
> [1] http://jira.funtoo.org/browse/FL-38
>
> ---~~.~~---
> Mike
> // SilverTip257 //
>
>
> On Thu, Sep 27, 2012 at 8:46 AM, SilverTip257 <silvertip257@gmail.com> wrote:
>> Definitely shoot for CentOS 6.3 ...
>>
>> XFS with a kernel _more recent_ than 2.6.36 (currently shipped with
>> CentOS6) has more improvements to the XFS code. Youtube video on XFS
>> [0] - I believe the kernel version noted is 2.6.39 (watch the video!)
>> [2].
>>
>> And there's also a Youtube video on BTRFS [1] that was linked
>> to/shared by Fernando.
>>
>> [0] http://lists.centos.org/pipermail/centos/2012-August/128119.html
>> [1] http://lists.centos.org/pipermail/centos/2012-August/128110.html
>> [2] http://lwn.net/Articles/438671/
>>
>> ---~~.~~---
>> Mike
>> // SilverTip257 //
>>
>>
>> On Thu, Sep 27, 2012 at 5:08 AM, John R Pierce <pierce@hogranch.com> wrote:
>>> On 09/27/12 1:52 AM, Nux! wrote:
>>>> Never had to deal with such a large filesystem, yet, but I'd try XFS on
>>>> it.
>>>
>>> XFS is fairly memory intensive. 11TB file systems tend to mean
>>> millions and millions of files.
>>>
>>> frankly, I wouldn't run this on CentOS 5.6, I would upgrade to CentOS
>>> 6.latest and then I would use XFS.... support for EXT4 and XFS is
>>> rather sketchy with the old kernel in 5.x (and why aren't you at 5.8 or
>>> whatever is current in the 5 series anyways?!?)
>>>
>>>
>>>
>>>
>>>
>>> --
>>> john r pierce N 37, W 122
>>> santa cruz ca mid-left coast
>>>
>>> _______________________________________________
>>> CentOS mailing list
>>> CentOS@centos.org
>>> http://lists.centos.org/mailman/listinfo/centos
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos



--
GPG Key ID: 6EC5EB27
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-28-2012, 08:19 PM
John R Pierce
 
Default 11TB ext4 filesystem - filesystem alternatives?

On 09/28/12 12:09 PM, Ilyas -- wrote:
> Hello,
>
> One day our servers farm rebooted unexpectedly (power fail happened)
> and on centos 6.3 with up2date kernel we lost few hundred files (which
> probably was opened for reading, NOT writing) on XFS.
>
> Unexpected power lost follow to situation when some files get a zero size.

what sort of physical storage? are you sure it is write-safe ?
write-back caches without battery backup are often a cause of data loss.



--
john r pierce N 37, W 122
santa cruz ca mid-left coast

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-28-2012, 08:29 PM
Keith Keller
 
Default 11TB ext4 filesystem - filesystem alternatives?

On 2012-09-28, Ilyas -- <umask00@gmail.com> wrote:
>
> One day our servers farm rebooted unexpectedly (power fail happened)
> and on centos 6.3 with up2date kernel we lost few hundred files (which
> probably was opened for reading, NOT writing) on XFS.
>
> Unexpected power lost follow to situation when some files get a zero size.

No filesystem can fully protect against power failures--that's not its
job. That's why higher-end RAID controllers have battery backups, and
why important servers should be on a UPS. If you are really paranoid,
you can probably tweak the kernel (e.g., using sysctl) to flush disk
writes more frequently, but then you might drag down performance with
it.

IOW, it's nice to have fsck, but it's better to take steps to avoid
needing it.

That being said, through a series of unfortunate events, I've lost power
on some of my larger XFS filesystems, and in those rare events I have not
seen or heard about any files lost. So I strongly suspect other factors
in your data loss--if XFS was involved, there were probably also other
issues involved as well.

--keith

--
kkeller@wombat.san-francisco.ca.us


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-28-2012, 11:22 PM
"James A. Peltier"
 
Default 11TB ext4 filesystem - filesystem alternatives?

----- Original Message -----
| Hi All.
|
| I have a CentOS server:
|
| CentOS 5.6 x86_64
| 2.6.18-238.12.1.el5.centos.plus
| e4fsprogs-1.41.12-2.el5.x86_64
|
| which has a 11TB ext4 filesystem. I have problems with running fsck
| on it
| and would like to change the filesystem because I do not like the
| possibility of running long fsck on it, it's a production machine.
| Also I
| have some problems with running fsck (not enough RAM, problem with
| scratch_files option) and if the filesystem will need intervention I
| will
| be in a problematic situation.
|
| Which other mature and stable filesystem can you recommend for such
| large
| storage?
|
| Best regards,
| Rafal Radecki.
| _______________________________________________
| CentOS mailing list
| CentOS@centos.org
| http://lists.centos.org/mailman/listinfo/centos
|

As someone who is working with 15-30TB volumes, use XFS, but be sure you have a lot of memory. 48GB at least and more if you have directories with 10s of thousands of files in them.

--
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpeltier@sfu.ca
Website : http://www.sfu.ca/itservices
http://blogs.sfu.ca/people/jpeltier

Success is to be measured not so much by the position that one has reached
in life but as by the obstacles they have overcome. - Booker T. Washington
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-28-2012, 11:30 PM
"James A. Peltier"
 
Default 11TB ext4 filesystem - filesystem alternatives?

----- Original Message -----
| Hello,
|
| One day our servers farm rebooted unexpectedly (power fail happened)
| and on centos 6.3 with up2date kernel we lost few hundred files
| (which
| probably was opened for reading, NOT writing) on XFS.
|
| Unexpected power lost follow to situation when some files get a zero
| size.

This is not uncommon with a file system like XFS, where the file system makes EXTENSIVE use of file system caching and memory and internal semantics that will make your head spin. Fact of the matter is, that in spite of this "possibility" of loss, XFS is by far the best file system for large volumes at the moment and especially during initialization time. You *can* use EXT4 with you can speed this up if you use the -E lazy_itable_init=1 -O dir_index,extent,flex_bg,uninit_bg options.

--
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone : 778-782-6573
Fax : 778-782-3045
E-Mail : jpeltier@sfu.ca
Website : http://www.sfu.ca/itservices
http://blogs.sfu.ca/people/jpeltier

Success is to be measured not so much by the position that one has reached
in life but as by the obstacles they have overcome. - Booker T. Washington
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 09-29-2012, 12:19 PM
Ilyas --
 
Default 11TB ext4 filesystem - filesystem alternatives?

Backend storage is 2 SATA directly attached disks. No any caches on
SATA controller.
Both disks run in mdraid mirror.

Zeroed files have written many days (some files was written and closed
2 weeks ago) ago before power fail.

On Sat, Sep 29, 2012 at 12:19 AM, John R Pierce <pierce@hogranch.com> wrote:
> On 09/28/12 12:09 PM, Ilyas -- wrote:
>> Hello,
>>
>> One day our servers farm rebooted unexpectedly (power fail happened)
>> and on centos 6.3 with up2date kernel we lost few hundred files (which
>> probably was opened for reading, NOT writing) on XFS.
>>
>> Unexpected power lost follow to situation when some files get a zero size.
>
> what sort of physical storage? are you sure it is write-safe ?
> write-back caches without battery backup are often a cause of data loss.
>
>
>
> --
> john r pierce N 37, W 122
> santa cruz ca mid-left coast
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos



--
GPG Key ID: 6EC5EB27
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 07:23 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org