At my university we run fluid dynamic simulations. These simulations
create many small files (30,000) per hour. Their size is very small
(20k to 200k). Instead of having this on the filesystem since it take
up inode space, I would like to tar them per day into one tar file. I
would then like an interface similar to zsh/ksh to "cd tar.file" and
use it as a typeical shell. Do tools like that exist? That would be
very benefical for us and our system admin does not have to yell at
us.
Any thoughts or ideas?

TIA


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

2008/8/20 Mag Gam <magawake@gmail.com>:
> At my university we run fluid dynamic simulations. These simulations
> create many small files (30,000) per hour. Their size is very small
> (20k to 200k). Instead of having this on the filesystem since it take
> up inode space, I would like to tar them per day into one tar file. I
> would then like an interface similar to zsh/ksh to "cd tar.file" and
> use it as a typeical shell. Do tools like that exist? That would be
> very benefical for us and our system admin does not have to yell at
> us.
> Any thoughts or ideas?
>
> TIA

Hey,

There are a few fuse drivers that can do what you're after[0]. There's
an article showing how one of them, archivemount, works on
linux.com[1].

cheers,
Owen.

[0] http://fuse.sourceforge.net/wiki/index.php/ArchiveFileSystems
[1] http://www.linux.com/feature/132196


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

You can browse tar archives with Midnight Commander (mc).
--
John Hasler


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:
> At my university we run fluid dynamic simulations. These simulations
> create many small files (30,000) per hour. Their size is very small
> (20k to 200k). Instead of having this on the filesystem since it take

My approach:

make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
count=1000

size so that you have enough room, and room for growth, of course

Make a filesystem inside of that file (reiserfs might be a good choice
since it is well-designed to handle lots of smallish files, although
"small" by that definition may be much smaller than 200k)

Mount that file in loopback mode prior to running your simulations,
and (after moving the files over to the new filesystem) direct all
filesystem traffic to use that 'filesystem' which may entail only
something simple as cd'ing into the 'filesystem' and starting work.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

WOW!

Very nice ideas.

I like the dd idea. What command would I use for that? Also, the files
are coming from NFS; how can I help this? Any ideas for this?



On Tue, Aug 19, 2008 at 10:24 PM, David Fox <dfox94085@gmail.com> wrote:
> On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:
>> At my university we run fluid dynamic simulations. These simulations
>> create many small files (30,000) per hour. Their size is very small
>> (20k to 200k). Instead of having this on the filesystem since it take
>
> My approach:
>
> make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
> count=1000
>
> size so that you have enough room, and room for growth, of course
>
> Make a filesystem inside of that file (reiserfs might be a good choice
> since it is well-designed to handle lots of smallish files, although
> "small" by that definition may be much smaller than 200k)
>
> Mount that file in loopback mode prior to running your simulations,
> and (after moving the files over to the new filesystem) direct all
> filesystem traffic to use that 'filesystem' which may entail only
> something simple as cd'ing into the 'filesystem' and starting work.
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
>
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

David:

Do you have some sort of script to manage this? I am a little hesitate
to give professors mkfs and mount sudo access. Is there a way around
this?

On Wed, Aug 20, 2008 at 12:13 AM, Mag Gam <magawake@gmail.com> wrote:
> WOW!
>
> Very nice ideas.
>
> I like the dd idea. What command would I use for that? Also, the files
> are coming from NFS; how can I help this? Any ideas for this?
>
>
>
> On Tue, Aug 19, 2008 at 10:24 PM, David Fox <dfox94085@gmail.com> wrote:
>> On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:
>>> At my university we run fluid dynamic simulations. These simulations
>>> create many small files (30,000) per hour. Their size is very small
>>> (20k to 200k). Instead of having this on the filesystem since it take
>>
>> My approach:
>>
>> make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
>> count=1000
>>
>> size so that you have enough room, and room for growth, of course
>>
>> Make a filesystem inside of that file (reiserfs might be a good choice
>> since it is well-designed to handle lots of smallish files, although
>> "small" by that definition may be much smaller than 200k)
>>
>> Mount that file in loopback mode prior to running your simulations,
>> and (after moving the files over to the new filesystem) direct all
>> filesystem traffic to use that 'filesystem' which may entail only
>> something simple as cd'ing into the 'filesystem' and starting work.
>>
>>
>> --
>> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
>> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
>>
>>
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On 08/20/08 05:50, Mag Gam wrote:

David:

Do you have some sort of script to manage this? I am a little hesitate
to give professors mkfs and mount sudo access. Is there a way around
this?


In that case, go back to Owen Townend's suggestion.


On Wed, Aug 20, 2008 at 12:13 AM, Mag Gam <magawake@gmail.com> wrote:

WOW!

Very nice ideas.

I like the dd idea. What command would I use for that? Also, the files
are coming from NFS; how can I help this? Any ideas for this?



On Tue, Aug 19, 2008 at 10:24 PM, David Fox <dfox94085@gmail.com> wrote:

On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:

At my university we run fluid dynamic simulations. These simulations
create many small files (30,000) per hour. Their size is very small
(20k to 200k). Instead of having this on the filesystem since it take

My approach:

make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
count=1000

size so that you have enough room, and room for growth, of course

Make a filesystem inside of that file (reiserfs might be a good choice
since it is well-designed to handle lots of smallish files, although
"small" by that definition may be much smaller than 200k)

Mount that file in loopback mode prior to running your simulations,
and (after moving the files over to the new filesystem) direct all
filesystem traffic to use that 'filesystem' which may entail only
something simple as cd'ing into the 'filesystem' and starting work.


--
Ron Johnson, Jr.
Jefferson LA USA

"Do not bite at the bait of pleasure till you know there is no
hook beneath it." -- Thomas Jefferson


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

After solving the problem in the immediate consider telling the developer of
that simulation software to use a database!

On Wednesday 20 August 2008 03:40, Mag Gam wrote:
> At my university we run fluid dynamic simulations. These simulations
> create many small files (30,000) per hour. Their size is very small
> (20k to 200k). Instead of having this on the filesystem since it take
> up inode space, I would like to tar them per day into one tar file. I
> would then like an interface similar to zsh/ksh to "cd tar.file" and
> use it as a typeical shell. Do tools like that exist? That would be
> very benefical for us and our system admin does not have to yell at
> us.
> Any thoughts or ideas?
>
> TIA

--
Shachar Or | שחר �ור
http://ox.freeallweb.org/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Databases aren't filesystems, and they shouldn't be treated as such.
Especially if the text BLOBs need to be analyzed, summarized, etc.


On 08/20/08 06:34, Shachar Or wrote:
After solving the problem in the immediate consider telling the developer of
that simulation software to use a database!


On Wednesday 20 August 2008 03:40, Mag Gam wrote:

At my university we run fluid dynamic simulations. These simulations
create many small files (30,000) per hour. Their size is very small
(20k to 200k). Instead of having this on the filesystem since it take
up inode space, I would like to tar them per day into one tar file. I
would then like an interface similar to zsh/ksh to "cd tar.file" and
use it as a typeical shell. Do tools like that exist? That would be
very benefical for us and our system admin does not have to yell at
us.
Any thoughts or ideas?


--
Ron Johnson, Jr.
Jefferson LA USA

"Do not bite at the bait of pleasure till you know there is no
hook beneath it." -- Thomas Jefferson


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

I'm not sure what you're saying here.

It seems more appropriate to use a database instead of files in this case.

On Wednesday 20 August 2008 15:05, Ron Johnson wrote:
> Databases aren't filesystems, and they shouldn't be treated as such.
> Especially if the text BLOBs need to be analyzed, summarized, etc.
>
> On 08/20/08 06:34, Shachar Or wrote:
> > After solving the problem in the immediate consider telling the developer
> > of that simulation software to use a database!
> >
> > On Wednesday 20 August 2008 03:40, Mag Gam wrote:
> >> At my university we run fluid dynamic simulations. These simulations
> >> create many small files (30,000) per hour. Their size is very small
> >> (20k to 200k). Instead of having this on the filesystem since it take
> >> up inode space, I would like to tar them per day into one tar file. I
> >> would then like an interface similar to zsh/ksh to "cd tar.file" and
> >> use it as a typeical shell. Do tools like that exist? That would be
> >> very benefical for us and our system admin does not have to yell at
> >> us.
> >> Any thoughts or ideas?
>
> --
> Ron Johnson, Jr.
> Jefferson LA USA
>
> "Do not bite at the bait of pleasure till you know there is no
> hook beneath it." -- Thomas Jefferson

--
Shachar Or | שחר �ור
http://ox.freeallweb.org/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org