FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 09-20-2012, 12:33 PM
Andrew Lowe
 
Default Recreating a directory structure and indicating a files presence

Hi all,
I have the situation where I have a large amount of data, many TB's,
made up of many, many files. This information has now been archived but
I've got people who want to be able to see what data does/does not
exist, filling in gaps where they may exist.


As this data used to be available/visible in a directory structure, ie
via a file browser, I thought the easiest way for them to see if
something existed would be to create a mirror of the directory structure
and then populate this dir structure with 1 - 5 byte files with the same
name as the "real" data files that now reside in the archive. I've seen
some scripts on the interweb that allow me to create the dir structure,
but does anyone have any ideas how to do the creation of the "marker"
files in the active file system?


Just buying more hard disks and keeping the data on line also isn't an
option.


Any thoughts, greatly appreciated,

Andrew
 
Old 09-20-2012, 02:13 PM
Alan McKinnon
 
Default Recreating a directory structure and indicating a files presence

On Thu, 20 Sep 2012 20:33:39 +0800
Andrew Lowe <agl@wht.com.au> wrote:

> Hi all,
> I have the situation where I have a large amount of data,
> many TB's, made up of many, many files. This information has now been
> archived but I've got people who want to be able to see what data
> does/does not exist, filling in gaps where they may exist.
>
> As this data used to be available/visible in a directory
> structure, ie via a file browser, I thought the easiest way for them
> to see if something existed would be to create a mirror of the
> directory structure and then populate this dir structure with 1 - 5
> byte files with the same name as the "real" data files that now
> reside in the archive. I've seen some scripts on the interweb that
> allow me to create the dir structure, but does anyone have any ideas
> how to do the creation of the "marker" files in the active file
> system?
>
> Just buying more hard disks and keeping the data on line also
> isn't an option.
>
> Any thoughts, greatly appreciated,
>
> Andrew
>

I don't understand why you specify 1-5 byte files. Those few bytes will
always be useless. Rather use 0-length files.

On the archive:

find /root/of/dir/structure -type d > dirs.txt
find /root/of/dir/structure -type f > files.txt

Copy those two files to the on-line system:

for I in `cat dirs.txt` ; do mkdir -p $I ; done
for I in `cat files.txt` ; do touch -p $I ; done

Do that in the appropriate top-level directory of course. You can
probably make it more efficient using decent options to xargs, but what
the hell, I'd do it as-is. It's a once off action and finding the xargs
man page will take longer than the mkdirs....

--
Alan McKinnon
alan.mckinnon@gmail.com
 
Old 09-20-2012, 02:48 PM
Neil Bothwick
 
Default Recreating a directory structure and indicating a files presence

On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:

> On the archive:
>
> find /root/of/dir/structure -type d > dirs.txt
> find /root/of/dir/structure -type f > files.txt

This will add '/root/of/dir/structure' to the start of each path. would
it be better to do?

cd /root/of/dir/structure
find -type d > ../dirs.txt
find -type f > ../files.txt


--
Neil Bothwick

All generalizations are false.
 
Old 09-20-2012, 02:57 PM
Michael Mol
 
Default Recreating a directory structure and indicating a files presence

On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>
>> On the archive:
>>
>> find /root/of/dir/structure -type d > dirs.txt
>> find /root/of/dir/structure -type f > files.txt
>
> This will add '/root/of/dir/structure' to the start of each path. would
> it be better to do?
>
> cd /root/of/dir/structure
> find -type d > ../dirs.txt
> find -type f > ../files.txt

I see your path correction, and raise you:
* whitespace-safe folders
* Automatic copy to remote system.
* Automatic new file and folder creation
* Using those pretty xargs parameters.

cd /root/of/dir/structure
find . -type d -print0 > ~/dirs.txt
find . -type d -print0 > ~/files.txt

scp dirs.txt files.txt remote.system:

ssh remote.system <<ENDSSH
cd /root/of/new/structure
cat ~/dirs.txt|xargs -0 mkdir -p
cat ~/files.txt|xargs -0 touch
ENDSSH


--
:wq
 
Old 09-20-2012, 03:54 PM
Pandu Poluan
 
Default Recreating a directory structure and indicating a files presence

On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:

>

> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk> wrote:

> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:

> >

> >> On the archive:

> >>

> >> find /root/of/dir/structure -type d > dirs.txt

> >> find /root/of/dir/structure -type f > files.txt

> >

> > This will add '/root/of/dir/structure' to the start of each path. would

> > it be better to do?

> >

> > cd /root/of/dir/structure

> > find -type d > ../dirs.txt

> > find -type f > ../files.txt

>

> I see your path correction, and raise you:

> * whitespace-safe folders

> * Automatic copy to remote system.

> * Automatic new file and folder creation

> * Using those pretty xargs parameters.

>

> cd /root/of/dir/structure

> find . -type d -print0 > ~/dirs.txt

> find . -type d -print0 > ~/files.txt

>

> scp dirs.txt files.txt remote.system:

>

> ssh remote.system <<ENDSSH

> cd /root/of/new/structure

> cat ~/dirs.txt|xargs -0 mkdir -p

> cat ~/files.txt|xargs -0 touch

> ENDSSH

>

>


Cool... except that your raise is invalid (should've used -type f in the 3rd line)...


;-)


Rgds,
 
Old 09-20-2012, 04:04 PM
Michael Mol
 
Default Recreating a directory structure and indicating a files presence

On Thu, Sep 20, 2012 at 11:54 AM, Pandu Poluan <pandu@poluan.info> wrote:
>
> On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:
>>
>> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk>
>> wrote:
>> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>> >
>> >> On the archive:
>> >>
>> >> find /root/of/dir/structure -type d > dirs.txt
>> >> find /root/of/dir/structure -type f > files.txt
>> >
>> > This will add '/root/of/dir/structure' to the start of each path. would
>> > it be better to do?
>> >
>> > cd /root/of/dir/structure
>> > find -type d > ../dirs.txt
>> > find -type f > ../files.txt
>>
>> I see your path correction, and raise you:
>> * whitespace-safe folders
>> * Automatic copy to remote system.
>> * Automatic new file and folder creation
>> * Using those pretty xargs parameters.
>>
>> cd /root/of/dir/structure
>> find . -type d -print0 > ~/dirs.txt
>> find . -type d -print0 > ~/files.txt
>>
>> scp dirs.txt files.txt remote.system:
>>
>> ssh remote.system <<ENDSSH
>> cd /root/of/new/structure
>> cat ~/dirs.txt|xargs -0 mkdir -p
>> cat ~/files.txt|xargs -0 touch
>> ENDSSH
>>
>>
>
> Cool... except that your raise is invalid (should've used -type f in the 3rd
> line)...
>
> ;-)
>

Heh. It's also performing a bunch of unnecessary mkdir commands. I
mean, "find -type d" isn't going to return a subpath of a folder until
it's first returned the folder, so -p is unnecessary. For a better
speed gain, you'd want to keep -p, but filter out any folder which has
a subfolder, to reduce the number of mkdir commands issued. It's
possible mkdir performs this optimization internally, though. And I
don't know how to quickly do that filter on the command line.

--
:wq
 
Old 09-20-2012, 04:05 PM
Michael Mol
 
Default Recreating a directory structure and indicating a files presence

On Thu, Sep 20, 2012 at 12:04 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Thu, Sep 20, 2012 at 11:54 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>
>> On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:
>>>
>>> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk>
>>> wrote:
>>> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>>> >
>>> >> On the archive:
>>> >>
>>> >> find /root/of/dir/structure -type d > dirs.txt
>>> >> find /root/of/dir/structure -type f > files.txt
>>> >
>>> > This will add '/root/of/dir/structure' to the start of each path. would
>>> > it be better to do?
>>> >
>>> > cd /root/of/dir/structure
>>> > find -type d > ../dirs.txt
>>> > find -type f > ../files.txt
>>>
>>> I see your path correction, and raise you:
>>> * whitespace-safe folders
>>> * Automatic copy to remote system.
>>> * Automatic new file and folder creation
>>> * Using those pretty xargs parameters.
>>>
>>> cd /root/of/dir/structure
>>> find . -type d -print0 > ~/dirs.txt
>>> find . -type d -print0 > ~/files.txt
>>>
>>> scp dirs.txt files.txt remote.system:
>>>
>>> ssh remote.system <<ENDSSH
>>> cd /root/of/new/structure
>>> cat ~/dirs.txt|xargs -0 mkdir -p
>>> cat ~/files.txt|xargs -0 touch
>>> ENDSSH
>>>
>>>
>>
>> Cool... except that your raise is invalid (should've used -type f in the 3rd
>> line)...
>>
>> ;-)
>>
>
> Heh. It's also performing a bunch of unnecessary mkdir commands. I
> mean, "find -type d" isn't going to return a subpath of a folder until
^^ s/subpath/subfolder/ ^^


--
:wq
 
Old 09-23-2012, 04:32 PM
Andrew Lowe
 
Default Recreating a directory structure and indicating a files presence

On 09/20/12 22:13, Alan McKinnon wrote:

On Thu, 20 Sep 2012 20:33:39 +0800
Andrew Lowe <agl@wht.com.au> wrote:


Hi all,
I have the situation where I have a large amount of data,
many TB's, made up of many, many files. This information has now been
archived but I've got people who want to be able to see what data
does/does not exist, filling in gaps where they may exist.

As this data used to be available/visible in a directory
structure, ie via a file browser, I thought the easiest way for them
to see if something existed would be to create a mirror of the
directory structure and then populate this dir structure with 1 - 5
byte files with the same name as the "real" data files that now
reside in the archive. I've seen some scripts on the interweb that
allow me to create the dir structure, but does anyone have any ideas
how to do the creation of the "marker" files in the active file
system?

Just buying more hard disks and keeping the data on line also
isn't an option.

Any thoughts, greatly appreciated,

Andrew



I don't understand why you specify 1-5 byte files. Those few bytes will
always be useless. Rather use 0-length files.

On the archive:

find /root/of/dir/structure -type d > dirs.txt
find /root/of/dir/structure -type f > files.txt

Copy those two files to the on-line system:

for I in `cat dirs.txt` ; do mkdir -p $I ; done
for I in `cat files.txt` ; do touch -p $I ; done

Do that in the appropriate top-level directory of course. You can
probably make it more efficient using decent options to xargs, but what
the hell, I'd do it as-is. It's a once off action and finding the xargs
man page will take longer than the mkdirs....



Gentlemen,
Thanks for the suggestions, worked like a charm. I did the above and
except for whitespaces, which shouldn't have been there in the first
place and "-", things went well. As to the reason for 1 - 5 bytes, I
forgot about zero length files.....


Andrew
 

Thread Tools




All times are GMT. The time now is 09:56 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org