FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu User

 
 
LinkBack Thread Tools
 
Old 02-13-2012, 03:40 PM
Steve Flynn
 
Default Best filesystem to use for a specific type of application

On 13 February 2012 16:20, Rashkae <ubuntu@tigershaunt.com> wrote:
> On 02/13/2012 11:08 AM, Steve Flynn wrote:

> ReiserFS is deceptive. *It was advertised as being very high performance
> with small files. *But that was only taking into account benchmarks that
> favor it. *In my experience with this kind of workload, one very important
> 'benchmark' is the speed in which you can read all files in the order they
> are returned by Readdir. *(The function that lists all files in a
> directory.) *For the reading of small files to be efficient, they have to be
> read in the order they are laid out on the hard drive. *Otherwise, the head
> has to thrash with lots of random seeks, and that will drop your read
> performance to only a few MB/s.

Yup - we've seen that kind of pain already with much smaller datasets
(just over a million files). So much so that we ended up splitting
the directory structure up into "no more than 1000 files in a
directory" and effectively building a tree structure of directories
within directories to make sure we actually spend time reading the
contents of the files rather than reading the contents of the
directories. Worked OK for the smaller stuff but now I'm looking at 40
times more files so we need a better method.

> I haven't tested it much for this, but I think EXT4 has overcome this
> limitation. *XFS was by far the fastest filesystem I tested for this
> workload (before EXT4 was released.) *Reiser, and EXT3 were practically
> unusable.

Perfect. I'll move those two up my list a bit!

Cheers.

--
Steve

When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-13-2012, 04:11 PM
Ioannis Vranos
 
Default Best filesystem to use for a specific type of application

On Mon, Feb 13, 2012 at 6:40 PM, Steve Flynn <anothermindbomb@gmail.com> wrote:
> On 13 February 2012 16:20, Rashkae <ubuntu@tigershaunt.com> wrote:
>> On 02/13/2012 11:08 AM, Steve Flynn wrote:
>
>> ReiserFS is deceptive. *It was advertised as being very high performance
>> with small files. *But that was only taking into account benchmarks that
>> favor it. *In my experience with this kind of workload, one very important
>> 'benchmark' is the speed in which you can read all files in the order they
>> are returned by Readdir. *(The function that lists all files in a
>> directory.) *For the reading of small files to be efficient, they have to be
>> read in the order they are laid out on the hard drive. *Otherwise, the head
>> has to thrash with lots of random seeks, and that will drop your read
>> performance to only a few MB/s.
>
> Yup - we've seen that kind of pain already with much smaller datasets
> (just over a million files). So much so that we ended up *splitting
> the directory structure up into "no more than 1000 files in a
> directory" and effectively building a tree structure of directories
> within directories to make sure we actually spend time reading the
> contents of the files rather than reading the contents of the
> directories. Worked OK for the smaller stuff but now I'm looking at 40
> times more files so we need a better method.
>
>> I haven't tested it much for this, but I think EXT4 has overcome this
>> limitation. *XFS was by far the fastest filesystem I tested for this
>> workload (before EXT4 was released.) *Reiser, and EXT3 were practically
>> unusable.
>
> Perfect. I'll move those two up my list a bit!

Keep in mind that ext4 (and btrfs) are aggressively developed, and
kernel 3.2 comes with even more optimisations for ext4.

Perhaps what you are looking for is the Phoronix web site:

http://www.phoronix.com

Search for comparison benchmarks there.


--
Ioannis Vranos

http://cppsoftware.binhoster.com

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-13-2012, 06:17 PM
Koh Choon Lin
 
Default Best filesystem to use for a specific type of application

Hi

>> Can anyone point me to some stats for how differing filesystems
>> (Reiser, XFS, JFS, Ext3, Ext4, BTRfs, etc) stack up against each other
>> when dealing with a lot of very small files. I have a little bell
>> tinkling away at the back of my mind that Reiser was particularly good
>> for small files, but I could well be making that up... plus I don't
>> know how well that stacks up these days against the advances made in
>> other filesystems.

I am also interested in the performance of NTFS, ZFS, UFS, etc.. on
their native OS in rw-ing of small files.



--
Regards
Koh Choon Lin

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 11:10 AM
Greg Zeng
 
Default Best filesystem to use for a specific type of application

imo http://www.phoronix.com seems to pretend that NTFS does not exist.
Personally I use M$ Win7-64 NTFS-compressed partitions. The sector
size can also be flexible.

I assume that you want journalling as well (not available in Ext2;
BTRFS & Reiser are still aphaware)? If you use compression in the
file system, there are fewer hard-drice accesses. Not sure if RAID,
SSD would assist. One advantage of NTFS is the ability to defrag the
HDD (not needed for SSD drives). Grudgingly, Linux seems to admit
that magnetic drives have fragmentation & speed access variations.

Benchmarks (independent) on NTFS show that defrag greatly speeds HDD
access. There are urls which independently confirm this).


On 2012-02-14, Koh Choon Lin <2choonlin@gmail.com> wrote:
> Hi
>
>>> Can anyone point me to some stats for how differing filesystems
>>> (Reiser, XFS, JFS, Ext3, Ext4, BTRfs, etc) stack up against each other
>>> when dealing with a lot of very small files. I have a little bell
>>> tinkling away at the back of my mind that Reiser was particularly good
>>> for small files, but I could well be making that up... plus I don't
>>> know how well that stacks up these days against the advances made in
>>> other filesystems.
>
> I am also interested in the performance of NTFS, ZFS, UFS, etc.. on
> their native OS in rw-ing of small files.
>
>
>
> --
> Regards
> Koh Choon Lin
>
> --
> ubuntu-users mailing list
> ubuntu-users@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 11:24 AM
Avi Greenbury
 
Default Best filesystem to use for a specific type of application

Greg Zeng wrote:
> imo http://www.phoronix.com seems to pretend that NTFS does not exist.

Well, it's a Linux orientated site, that's hardly more surprising than
a Windows one benchmarking filesystems and not mentioning XFS.

> One advantage of NTFS is the ability to defrag the HDD (not needed
> for SSD drives). Grudgingly, Linux seems to admit that magnetic
> drives have fragmentation & speed access variations.

I was under the impression that, in general, the model under Linux
is that the filesystems are defragmented using idle I/O time. Is this
not the case?

--
Avi.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 02:31 PM
Steve Flynn
 
Default Best filesystem to use for a specific type of application

On 15 February 2012 12:10, Greg Zeng <gregzeng@gmail.com> wrote:

> I assume that you want journalling as well (not available in Ext2;
> BTRFS & Reiser are still aphaware)? *If you use compression in the
> file system, there are fewer hard-drice accesses. *Not sure if RAID,
> SSD would assist. *One advantage of NTFS is the ability to defrag the
> HDD (not needed for SSD drives). *Grudgingly, Linux seems to admit
> that magnetic drives have fragmentation & speed access variations.

Journalling is not required for my situation. My company will only
read the files and will never write to the disk. Each image is also
written to disk by the client with a SHA1 checksum, which we read and
check as we're processing the data. Anything which doesn't match is
reported on and ignored. If the filesystem I select includes
journalling it'd be nice to be able to turn it off. I suspect it won't
make any difference for my read only work however.

I'm not sure compression will give us much of a benefit either as much
of the data is already compressed. Every file is (likely) to be
smaller than the block size of the filesystem so I suspect the gains
would be marginal... only a test will prove it however.


--
Steve

When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 02:46 PM
Ioannis Vranos
 
Default Best filesystem to use for a specific type of application

On Wed, Feb 15, 2012 at 2:10 PM, Greg Zeng <gregzeng@gmail.com> wrote:
> imo http://www.phoronix.com seems to pretend that NTFS does not exist.
> *Personally I use M$ Win7-64 NTFS-compressed partitions. *The sector
> size can also be flexible.
>
> I assume that you want journalling as well (not available in Ext2;
> BTRFS & Reiser are still aphaware)? *If you use compression in the
> file system, there are fewer hard-drice accesses. *Not sure if RAID,
> SSD would assist. *One advantage of NTFS is the ability to defrag the
> HDD (not needed for SSD drives). *Grudgingly, Linux seems to admit
> that magnetic drives have fragmentation & speed access variations.
>
> Benchmarks (independent) on NTFS show that defrag greatly speeds HDD
> access. *There are urls which independently confirm this).

Not wanting to start an argument here, however AFAIK, NTFS has no
ability "to defrag the HDD", it just gets badly fragmented, and a user
defragmentation application/service is available to defrag it. Also it
takes a lot of time to perform a filesystem check.

NTFS is a 1993-era filesystem. I suggest the OP use ext4 instead,
unless he has some very specific needs, like reading the filesystem
under Windows.


--
Ioannis Vranos

http://cppsoftware.binhoster.com

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 02:58 PM
Liam Proven
 
Default Best filesystem to use for a specific type of application

On 15 February 2012 15:46, Ioannis Vranos <ioannis.vranos@gmail.com> wrote:
>
> Not wanting to start an argument here, however AFAIK, NTFS has no
> ability "to defrag the HDD", it just gets badly fragmented, and a user
> defragmentation application/service is available to defrag it. Also it
> takes a lot of time to perform a filesystem check.

NT offers an API explicitly for defragging.

> NTFS is a 1993-era filesystem. I suggest the OP use ext4 instead,
> unless he has some very specific needs, like reading the filesystem
> under Windows.

It is aging & MS is introducing a replacement, but 2012 NTFS is not
the same as 1993 NTFS. It is on at least version 5 now, I believe.


--
Liam Proven • Profile: http://lproven.livejournal.com/profile
Email: lproven@cix.co.uk • GMail/G+/Twitter/Flickr/Facebook: lproven
MSN: lproven@hotmail.com • Skype/AIM/Yahoo/LinkedIn: liamproven
Tel: +44 20-8685-0498 • Cell: +44 7939-087884

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 04:40 PM
Koh Choon Lin
 
Default Best filesystem to use for a specific type of application

Hi

>> SSD would assist. *One advantage of NTFS is the ability to defrag the
>> HDD (not needed for SSD drives). *Grudgingly, Linux seems to admit
>> that magnetic drives have fragmentation & speed access variations.

That advantage is not limited to NTFS as FAT16/32, ext1/2/3/4/5,
etc... can be defrag too with a defragger. SSDs, like flash drives,
would be destroyed pretty soon if there exists too much RW operations.

I believe all modern file systems have built in technologies to
prevent excessive fragmentation, e.g. leaving space for a file to
grow. Microsoft Windows seems to run the defragger on a fixed schedule
during idle periods.



--
Regards
Koh Choon Lin

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 02-15-2012, 04:45 PM
"compdoc"
 
Default Best filesystem to use for a specific type of application

> Microsoft Windows seems to run the defragger on a fixed schedule during
idle periods.

I've never seen it do that...




--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 

Thread Tools




All times are GMT. The time now is 05:13 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org