On Wed, Aug 20, 2008 at 02:34:59PM +0300, Shachar Or wrote:
> After solving the problem in the immediate consider telling the developer of
> that simulation software to use a database!

Err... I guess you meant:

"tell the developer of the application to use a zip archive, as it
allows random access to files in it".

--
Tzafrir Cohen | tzafrir@jabber.org | VIM is
http://tzafrir.org.il | | a Mutt's
tzafrir@cohens.org.il | | best
ICQ# 16849754 | | friend


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

I've been a DBA for 10 (11?) years, but I know that they aren't good
for everything. 30,000file/hr * 6 hrs/day * 6 days = 1.08*10^6
files. *Many* more, if the simulations run for weeks.


Hierarchical structures are much better at storing such data than
RDBMSs. (I don't know how MySQL stores them, but PostgreSQL puts
them all in one directory.) Maybe if you port IMS to Linux...


http://en.wikipedia.org/wiki/Information_Management_System

On 08/20/08 07:12, Shachar Or wrote:

I'm not sure what you're saying here.

It seems more appropriate to use a database instead of files in this case.

On Wednesday 20 August 2008 15:05, Ron Johnson wrote:

Databases aren't filesystems, and they shouldn't be treated as such.
Especially if the text BLOBs need to be analyzed, summarized, etc.

On 08/20/08 06:34, Shachar Or wrote:

After solving the problem in the immediate consider telling the developer
of that simulation software to use a database!

On Wednesday 20 August 2008 03:40, Mag Gam wrote:

At my university we run fluid dynamic simulations. These simulations
create many small files (30,000) per hour. Their size is very small
(20k to 200k). Instead of having this on the filesystem since it take
up inode space, I would like to tar them per day into one tar file. I
would then like an interface similar to zsh/ksh to "cd tar.file" and
use it as a typeical shell. Do tools like that exist? That would be
very benefical for us and our system admin does not have to yell at
us.
Any thoughts or ideas?


--
Ron Johnson, Jr.
Jefferson LA USA

"Do not bite at the bait of pleasure till you know there is no
hook beneath it." -- Thomas Jefferson


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Le Wednesday 20 August 2008 vers 02:40, Mag Gam("Mag Gam"
<magawake@gmail.com>) a écrit:

Hello,

> I would like to tar them
> per day into one tar file. I would then like an interface similar
> to zsh/ksh to "cd tar.file" and use it as a typeical shell.

Try fuse[1]. It has a driver for tar(ArchiveFileSystems) files[2].

1: http://fuse.sourceforge.net/
2: http://fuse.sourceforge.net/wiki/index.php/FileSystems
--
http://www.glennie.fr
The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Thank you all.

I am very interested in the fuse AND the fs image solution. Is it
possible to integrate that into auto mounter or autofs type solution?
I don't want too many open mounts. If the /tmp/mountpoint it not open,
I would like to automatically disregard the mount point.



On Wed, Aug 20, 2008 at 11:06 AM, Glennie Vignarajah <glennie@glennie.fr> wrote:
> Le Wednesday 20 August 2008 vers 02:40, Mag Gam("Mag Gam"
> <magawake@gmail.com>) a écrit:
>
> Hello,
>
>> I would like to tar them
>> per day into one tar file. I would then like an interface similar
>> to zsh/ksh to "cd tar.file" and use it as a typeical shell.
>
> Try fuse[1]. It has a driver for tar(ArchiveFileSystems) files[2].
>
> 1: http://fuse.sourceforge.net/
> 2: http://fuse.sourceforge.net/wiki/index.php/FileSystems
> --
> http://www.glennie.fr
> The reasonable man adapts himself to the world; the unreasonable one
> persists in trying to adapt the world to himself. Therefore all
> progress depends on the unreasonable man.
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
>
>

On Wednesday 20 August 2008 13:50, Mag Gam wrote:
> David:
>
> Do you have some sort of script to manage this? I am a little hesitate
> to give professors mkfs and mount sudo access. Is there a way around
> this?

You can specify the 'user' option in fstab so that usres can mount the
relevant filesystem.

If you precreate the files with the filesystems in them, it may cover it.
>
> On Wed, Aug 20, 2008 at 12:13 AM, Mag Gam <magawake@gmail.com> wrote:
> > WOW!
> >
> > Very nice ideas.
> >
> > I like the dd idea. What command would I use for that? Also, the files
> > are coming from NFS; how can I help this? Any ideas for this?
> >
> > On Tue, Aug 19, 2008 at 10:24 PM, David Fox <dfox94085@gmail.com> wrote:
> >> On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:
> >>> At my university we run fluid dynamic simulations. These simulations
> >>> create many small files (30,000) per hour. Their size is very small
> >>> (20k to 200k). Instead of having this on the filesystem since it take
> >>
> >> My approach:
> >>
> >> make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
> >> count=1000
> >>
> >> size so that you have enough room, and room for growth, of course
> >>
> >> Make a filesystem inside of that file (reiserfs might be a good choice
> >> since it is well-designed to handle lots of smallish files, although
> >> "small" by that definition may be much smaller than 200k)
> >>
> >> Mount that file in loopback mode prior to running your simulations,
> >> and (after moving the files over to the new filesystem) direct all
> >> filesystem traffic to use that 'filesystem' which may entail only
> >> something simple as cd'ing into the 'filesystem' and starting work.
> >>
> >>
> >> --
> >> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> >> with a subject of "unsubscribe". Trouble? Contact
> >> listmaster@lists.debian.org

--
Shachar Or | שחר �ור
http://ox.freeallweb.org/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Sharchar:

Can I do that with autofs? Lets say I have a directory called
/home/$userid/image_files; can I have autofs to look for
/home/$userid/& and automatically mount and unmount these fs images?
Lets say I keep the fs images in /home/$user/.isos

ANy thoughts about this?

Either way, thanks for all of your help guys!

TIA

On Thu, Aug 21, 2008 at 1:18 AM, Shachar Or <dawnlight@lavabit.com> wrote:
> On Wednesday 20 August 2008 13:50, Mag Gam wrote:
>> David:
>>
>> Do you have some sort of script to manage this? I am a little hesitate
>> to give professors mkfs and mount sudo access. Is there a way around
>> this?
>
> You can specify the 'user' option in fstab so that usres can mount the
> relevant filesystem.
>
> If you precreate the files with the filesystems in them, it may cover it.
>>
>> On Wed, Aug 20, 2008 at 12:13 AM, Mag Gam <magawake@gmail.com> wrote:
>> > WOW!
>> >
>> > Very nice ideas.
>> >
>> > I like the dd idea. What command would I use for that? Also, the files
>> > are coming from NFS; how can I help this? Any ideas for this?
>> >
>> > On Tue, Aug 19, 2008 at 10:24 PM, David Fox <dfox94085@gmail.com> wrote:
>> >> On Tue, Aug 19, 2008 at 5:40 PM, Mag Gam <magawake@gmail.com> wrote:
>> >>> At my university we run fluid dynamic simulations. These simulations
>> >>> create many small files (30,000) per hour. Their size is very small
>> >>> (20k to 200k). Instead of having this on the filesystem since it take
>> >>
>> >> My approach:
>> >>
>> >> make a sufficiently-sized file using dd if=/dev/zero of=/bigfile bs=1m
>> >> count=1000
>> >>
>> >> size so that you have enough room, and room for growth, of course
>> >>
>> >> Make a filesystem inside of that file (reiserfs might be a good choice
>> >> since it is well-designed to handle lots of smallish files, although
>> >> "small" by that definition may be much smaller than 200k)
>> >>
>> >> Mount that file in loopback mode prior to running your simulations,
>> >> and (after moving the files over to the new filesystem) direct all
>> >> filesystem traffic to use that 'filesystem' which may entail only
>> >> something simple as cd'ing into the 'filesystem' and starting work.
>> >>
>> >>
>> >> --
>> >> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
>> >> with a subject of "unsubscribe". Trouble? Contact
>> >> listmaster@lists.debian.org
>
> --
> Shachar Or | שחר �ור
> http://ox.freeallweb.org/
>
>

2008/8/21 Mag Gam <magawake@gmail.com>:
> Sharchar:
>
> Can I do that with autofs? Lets say I have a directory called
> /home/$userid/image_files; can I have autofs to look for
> /home/$userid/& and automatically mount and unmount these fs images?
> Lets say I keep the fs images in /home/$user/.isos
>
> ANy thoughts about this?
>
> Either way, thanks for all of your help guys!
>
> TIA
>

This is one of the major motivating factors for creating fuse.
Why take the hard road here?

fuse allows users with filesystem access to an image to mount it
within one of their own directories.
There are fuse drivers for iso (fuseiso), ext (fuseext2), tar
(archivemount, avfs), tar, gzip, bzip2, rar (avfs), network (fusesmb).
e.g. iso
mount:
$ fuseiso /path/to/image.iso ~/mnt
unmount:
$ fusermount -u ~/mnt

or with avfs, there's an example of using it here:
http://linuxwindows.org/2008/05/how-to-use-avfs.html

>From the article:
"
After installation of AVFS,establish the folder .avfs in the home directory.
Then run mountavfs,which does not require root privileges.
Open .avfs folder, you will see a mirror structure of the root
directory,which the cabinet can be open as the folder,but not directly
access to them, need with the #, such as
/home/yourname/garro.rar

should be visited as follows:

cd /home/yourname/.avfs/home/yourname/garro.rar#/
"

This works for tar, gzip, rar, bzip2 and others.

cheers,
Owen.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thursday 21 August 2008 14:44, Owen Townend wrote:
> fuse allows users with filesystem access to an image to mount it
> within one of their own directories.
> There are fuse drivers for iso (fuseiso), ext (fuseext2), tar
> (archivemount, avfs), tar, gzip, bzip2, rar (avfs), network (fusesmb).
> e.g. iso
> mount:
> $ fuseiso /path/to/image.iso ~/mnt
> unmount:
> $ fusermount -u ~/mnt
>
> or with avfs, there's an example of using it here:
> http://linuxwindows.org/2008/05/how-to-use-avfs.html
>
> >From the article:
>
> "
> After installation of AVFS,establish the folder .avfs in the home
> directory. Then run mountavfs,which does not require root privileges.
> Open .avfs folder, you will see a mirror structure of the root
> directory,which the cabinet can be open as the folder,but not directly
> access to them, need with the #, such as
> /home/yourname/garro.rar
>
> should be visited as follows:
>
> cd /home/yourname/.avfs/home/yourname/garro.rar#/
> "
>
> This works for tar, gzip, rar, bzip2 and others.

This sounds good.
>
> cheers,
> Owen.

--
Shachar Or | שחר �ור
http://ox.freeallweb.org/


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Le Thursday 21 August 2008 vers 03:58, Mag Gam("Mag Gam"
<magawake@gmail.com>) a écrit:


Hello,

> I am very interested in the fuse AND the fs image solution. Is it
> possible to integrate that into auto mounter or autofs type
> solution?
Afuse seems do this sort things. Have a look at
http://afuse.sourceforge.net
Moreover, there is a debian package.
HTH,
--
http://www.glennie.fr
The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, Aug 21, 2008 at 06:29:00AM -0400, Mag Gam wrote:
> Sharchar:
>
> Can I do that with autofs? Lets say I have a directory called
> /home/$userid/image_files; can I have autofs to look for
> /home/$userid/& and automatically mount and unmount these fs images?
> Lets say I keep the fs images in /home/$user/.isos

Here is what we have for auto-mounting ISO images.

$ cat /etc/auto.master
#
/var/www/netinst/cds /etc/auto.netinst-cd

$ cat /etc/auto.netinst-cd
cd1 -fstype=iso9660,ro,user,loop :/path/to/image1.iso
cd2 -fstype=iso9660,ro,user,loop :/path/to/image2.iso

This will still be require a file system to mount.

--
Tzafrir Cohen | tzafrir@jabber.org | VIM is
http://tzafrir.org.il | | a Mutt's
tzafrir@cohens.org.il | | best
ICQ# 16849754 | | friend


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org