problems with large directories?
 

Hi,
I have an application that deals with 100,000 to 1,000,000 image files.
I initially structured it to use multiple directories, so that file 123456 would be stored in /12/34/123456. *I'm now wondering if that's pointless, as it would simplify things to simply store the file in /123456.*

Can anyone indicate whether I'm gaining anything by using smaller directories in ext3/ext4? *Thanks.
Mitch*
_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

On 03/08/2010 08:23 PM, Mitch Trachtenberg wrote:

Hi,

I have an application that deals with 100,000 to 1,000,000 image files.

I initially structured it to use multiple directories, so that file
123456 would be stored in /12/34/123456. I'm now wondering if that's
pointless, as it would simplify things to simply store the file in /123456.

Can anyone indicate whether I'm gaining anything by using smaller
directories in ext3/ext4? Thanks.

Mitch



I think that breaking up your files into subdirectories makes it easier to
navigate the tree and find files from a human point of view. Even better if the
bytes reflect something like year/month/day/hour/min (assuming your pathname has
a date based guid or similar encoding).


You can have a million files in one large directory, but be careful to iterate
and copy them in a sorted order (sorted by inode) to avoid nasty performance
issues that are side effects of the way we hash file names in ext3/4.


Good luck!

Ric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

Mitch Trachtenberg wrote:
> Hi,
>
> I have an application that deals with 100,000 to 1,000,000 image files.
>
> I initially structured it to use multiple directories, so that file
> 123456 would be stored in /12/34/123456. I'm now wondering if that's
> pointless, as it would simplify things to simply store the file in /123456.
>
> Can anyone indicate whether I'm gaining anything by using smaller
> directories in ext3/ext4? Thanks.
>
> Mitch
>

If you have one file per dir, that's a lot of dirs, and the time to search for
new dir inode locations can get rather expensive as the fs fills, in my experience.

You may also want to toy with setting the "topdir" flag on a dir; new directories
-under- that topdir get spread around the block groups. New dirs under a non-topdir
tend to stay closer to the parent.

Finally, remember that ext2/3 has a limit of 32000 or so files per dir.
ext4 lifts this restriction.

-Eric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

--On 8 March 2010 21:55:12 -0600 Eric Sandeen <sandeen@redhat.com> wrote:


Finally, remember that ext2/3 has a limit of 32000 or so files per dir.


My IMAP spool suggests this is false w.r.t. ext3

--
Alex Bligh

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

On Tue, 9 Mar 2010 at 07:11, Alex Bligh wrote:
> > Finally, remember that ext2/3 has a limit of 32000 or so files per dir.
> My IMAP spool suggests this is false w.r.t. ext3

Did you use any special options when creating this ext3 filesystem? I've
just tried but was only able to create 31998 directories in one directory.

Christian.
--
BOFH excuse #443:

Zombie processes detected, machine is haunted.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

On Tue, Mar 09, 2010 at 01:14:40 -0800,
Christian Kujau <lists@nerdbynature.de> wrote:
> On Tue, 9 Mar 2010 at 07:11, Alex Bligh wrote:
> > > Finally, remember that ext2/3 has a limit of 32000 or so files per dir.
> > My IMAP spool suggests this is false w.r.t. ext3
>
> Did you use any special options when creating this ext3 filesystem? I've
> just tried but was only able to create 31998 directories in one directory.

You can create a lot of files (though things work slowly). I think there
was a typo above and that it should have said dirs per dir, not files per dir.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

--On 9 March 2010 01:14:40 -0800 Christian Kujau <lists@nerdbynature.de>
wrote:



> Finally, remember that ext2/3 has a limit of 32000 or so files per dir.
My IMAP spool suggests this is false w.r.t. ext3


Did you use any special options when creating this ext3 filesystem? I've
just tried but was only able to create 31998 directories in one directory.


As Ted pointed out off list, the limit is the number of subdirectories
in a directory, not the number of files in a directory.

--
Alex Bligh

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

problems with large directories?
 

Alex Bligh wrote:
>
> --On 9 March 2010 01:14:40 -0800 Christian Kujau <lists@nerdbynature.de>
> wrote:
>
>>>> Finally, remember that ext2/3 has a limit of 32000 or so files per dir.
>>> My IMAP spool suggests this is false w.r.t. ext3
>> Did you use any special options when creating this ext3 filesystem? I've
>> just tried but was only able to create 31998 directories in one directory.
>
> As Ted pointed out off list, the limit is the number of subdirectories
> in a directory, not the number of files in a directory.

Argh you are right, sorry, brain burp. :)

-Eric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users