FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 08-15-2010, 11:05 PM
Andreas Dilger
 
Default Squid and first-level subdirectories & second-level subdirectories on ext3/4

On 2010-08-13, at 04:00, Ralf Hildebrandt wrote:
> Squid, a proxy, is by its nature, storing large amounts of relatively
> small files in it's cache.
>
> As config optins it's offering:
>
> # 'Level-1' is the number of first-level subdirectories which
> # will be created under the 'Directory'. The default is 16.
> #
> # 'Level-2' is the number of second-level subdirectories which
> # will be created under each first-level directory. The default
> # is 256.
>
> Meaning one has
>
> /squid-cache/(16 dirs)/(256 dirs)/(the small files)
>
> so the total number of small files in the cache is (hopefully) evenly
> distributed to 16*256 directories.
>
> But is that optimal for an ext3/4 filesystem? What is the point of
> using 16 for the first level and 256 for the second?

In ext3/4 the top-level inodes are spread around the filesystem, on the assumtion that something like /home or / is allocating trees of unrelated subdirectories at the top level, but that files within those subdirectories ARE related and should be allocated together.

Depending on how many files are in your cache, the 256 * {small files} is likely too big to fit into a single block group (32k inodes, 32k blocks) so you may want to consider marking the first level of subdirectories with the "TOPDIR" flag, that indicates the second-level (256) subdirs should also be spread around the disk.

> Wouldn't 64*64 (which equals 16*256) be better when it comes to
> finding the files on disk?

Benchmarking tells all...

Cheers, Andreas






_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 12-06-2010, 06:34 PM
Ralf Hildebrandt
 
Default Squid and first-level subdirectories & second-level subdirectories on ext3/4

* Andreas Dilger <adilger@dilger.ca>:

> In ext3/4 the top-level inodes are spread around the filesystem, on the
> assumtion that something like /home or / is allocating trees of
> unrelated subdirectories at the top level, but that files within those
> subdirectories ARE related and should be allocated together.
>
> Depending on how many files are in your cache, the 256 * {small files}
> is likely too big to fit into a single block group (32k inodes, 32k
> blocks) so you may want to consider marking the first level of
> subdirectories with the "TOPDIR" flag, that indicates the second-level
> (256) subdirs should also be spread around the disk.

How do I mark subdirectories with the "TOPDIR" flag?

--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hildebrandt@charite.de | http://www.charite.de


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 12-06-2010, 07:33 PM
Ralf Hildebrandt
 
Default Squid and first-level subdirectories & second-level subdirectories on ext3/4

* Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
> * Andreas Dilger <adilger@dilger.ca>:
>
> > In ext3/4 the top-level inodes are spread around the filesystem, on the
> > assumtion that something like /home or / is allocating trees of
> > unrelated subdirectories at the top level, but that files within those
> > subdirectories ARE related and should be allocated together.
> >
> > Depending on how many files are in your cache, the 256 * {small files}
> > is likely too big to fit into a single block group (32k inodes, 32k
> > blocks) so you may want to consider marking the first level of
> > subdirectories with the "TOPDIR" flag, that indicates the second-level
> > (256) subdirs should also be spread around the disk.
>
> How do I mark subdirectories with the "TOPDIR" flag?

Found it: chattr +T

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 06:28 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org