Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   EXT3 Users (http://www.linux-archive.org/ext3-users/)
-   -   Squid and first-level subdirectories & second-level subdirectories on ext3/4 (http://www.linux-archive.org/ext3-users/413319-squid-first-level-subdirectories-second-level-subdirectories-ext3-4-a.html)

Andreas Dilger 08-15-2010 11:05 PM

Squid and first-level subdirectories & second-level subdirectories on ext3/4
 
On 2010-08-13, at 04:00, Ralf Hildebrandt wrote:
> Squid, a proxy, is by its nature, storing large amounts of relatively
> small files in it's cache.
>
> As config optins it's offering:
>
> # 'Level-1' is the number of first-level subdirectories which
> # will be created under the 'Directory'. The default is 16.
> #
> # 'Level-2' is the number of second-level subdirectories which
> # will be created under each first-level directory. The default
> # is 256.
>
> Meaning one has
>
> /squid-cache/(16 dirs)/(256 dirs)/(the small files)
>
> so the total number of small files in the cache is (hopefully) evenly
> distributed to 16*256 directories.
>
> But is that optimal for an ext3/4 filesystem? What is the point of
> using 16 for the first level and 256 for the second?

In ext3/4 the top-level inodes are spread around the filesystem, on the assumtion that something like /home or / is allocating trees of unrelated subdirectories at the top level, but that files within those subdirectories ARE related and should be allocated together.

Depending on how many files are in your cache, the 256 * {small files} is likely too big to fit into a single block group (32k inodes, 32k blocks) so you may want to consider marking the first level of subdirectories with the "TOPDIR" flag, that indicates the second-level (256) subdirs should also be spread around the disk.

> Wouldn't 64*64 (which equals 16*256) be better when it comes to
> finding the files on disk?

Benchmarking tells all...

Cheers, Andreas






_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Ralf Hildebrandt 12-06-2010 06:34 PM

Squid and first-level subdirectories & second-level subdirectories on ext3/4
 
* Andreas Dilger <adilger@dilger.ca>:

> In ext3/4 the top-level inodes are spread around the filesystem, on the
> assumtion that something like /home or / is allocating trees of
> unrelated subdirectories at the top level, but that files within those
> subdirectories ARE related and should be allocated together.
>
> Depending on how many files are in your cache, the 256 * {small files}
> is likely too big to fit into a single block group (32k inodes, 32k
> blocks) so you may want to consider marking the first level of
> subdirectories with the "TOPDIR" flag, that indicates the second-level
> (256) subdirs should also be spread around the disk.

How do I mark subdirectories with the "TOPDIR" flag?

--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hildebrandt@charite.de | http://www.charite.de


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Ralf Hildebrandt 12-06-2010 07:33 PM

Squid and first-level subdirectories & second-level subdirectories on ext3/4
 
* Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>:
> * Andreas Dilger <adilger@dilger.ca>:
>
> > In ext3/4 the top-level inodes are spread around the filesystem, on the
> > assumtion that something like /home or / is allocating trees of
> > unrelated subdirectories at the top level, but that files within those
> > subdirectories ARE related and should be allocated together.
> >
> > Depending on how many files are in your cache, the 256 * {small files}
> > is likely too big to fit into a single block group (32k inodes, 32k
> > blocks) so you may want to consider marking the first level of
> > subdirectories with the "TOPDIR" flag, that indicates the second-level
> > (256) subdirs should also be spread around the disk.
>
> How do I mark subdirectories with the "TOPDIR" flag?

Found it: chattr +T

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users


All times are GMT. The time now is 10:19 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.