Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Gentoo User (http://www.linux-archive.org/gentoo-user/)
-   -   Recreating a directory structure and indicating a files presence (http://www.linux-archive.org/gentoo-user/705773-recreating-directory-structure-indicating-files-presence.html)

Andrew Lowe 09-20-2012 12:33 PM

Recreating a directory structure and indicating a files presence
 
Hi all,
I have the situation where I have a large amount of data, many TB's,
made up of many, many files. This information has now been archived but
I've got people who want to be able to see what data does/does not
exist, filling in gaps where they may exist.


As this data used to be available/visible in a directory structure, ie
via a file browser, I thought the easiest way for them to see if
something existed would be to create a mirror of the directory structure
and then populate this dir structure with 1 - 5 byte files with the same
name as the "real" data files that now reside in the archive. I've seen
some scripts on the interweb that allow me to create the dir structure,
but does anyone have any ideas how to do the creation of the "marker"
files in the active file system?


Just buying more hard disks and keeping the data on line also isn't an
option.


Any thoughts, greatly appreciated,

Andrew

Alan McKinnon 09-20-2012 02:13 PM

Recreating a directory structure and indicating a files presence
 
On Thu, 20 Sep 2012 20:33:39 +0800
Andrew Lowe <agl@wht.com.au> wrote:

> Hi all,
> I have the situation where I have a large amount of data,
> many TB's, made up of many, many files. This information has now been
> archived but I've got people who want to be able to see what data
> does/does not exist, filling in gaps where they may exist.
>
> As this data used to be available/visible in a directory
> structure, ie via a file browser, I thought the easiest way for them
> to see if something existed would be to create a mirror of the
> directory structure and then populate this dir structure with 1 - 5
> byte files with the same name as the "real" data files that now
> reside in the archive. I've seen some scripts on the interweb that
> allow me to create the dir structure, but does anyone have any ideas
> how to do the creation of the "marker" files in the active file
> system?
>
> Just buying more hard disks and keeping the data on line also
> isn't an option.
>
> Any thoughts, greatly appreciated,
>
> Andrew
>

I don't understand why you specify 1-5 byte files. Those few bytes will
always be useless. Rather use 0-length files.

On the archive:

find /root/of/dir/structure -type d > dirs.txt
find /root/of/dir/structure -type f > files.txt

Copy those two files to the on-line system:

for I in `cat dirs.txt` ; do mkdir -p $I ; done
for I in `cat files.txt` ; do touch -p $I ; done

Do that in the appropriate top-level directory of course. You can
probably make it more efficient using decent options to xargs, but what
the hell, I'd do it as-is. It's a once off action and finding the xargs
man page will take longer than the mkdirs....

--
Alan McKinnon
alan.mckinnon@gmail.com

Neil Bothwick 09-20-2012 02:48 PM

Recreating a directory structure and indicating a files presence
 
On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:

> On the archive:
>
> find /root/of/dir/structure -type d > dirs.txt
> find /root/of/dir/structure -type f > files.txt

This will add '/root/of/dir/structure' to the start of each path. would
it be better to do?

cd /root/of/dir/structure
find -type d > ../dirs.txt
find -type f > ../files.txt


--
Neil Bothwick

All generalizations are false.

Michael Mol 09-20-2012 02:57 PM

Recreating a directory structure and indicating a files presence
 
On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>
>> On the archive:
>>
>> find /root/of/dir/structure -type d > dirs.txt
>> find /root/of/dir/structure -type f > files.txt
>
> This will add '/root/of/dir/structure' to the start of each path. would
> it be better to do?
>
> cd /root/of/dir/structure
> find -type d > ../dirs.txt
> find -type f > ../files.txt

I see your path correction, and raise you:
* whitespace-safe folders
* Automatic copy to remote system.
* Automatic new file and folder creation
* Using those pretty xargs parameters.

cd /root/of/dir/structure
find . -type d -print0 > ~/dirs.txt
find . -type d -print0 > ~/files.txt

scp dirs.txt files.txt remote.system:

ssh remote.system <<ENDSSH
cd /root/of/new/structure
cat ~/dirs.txt|xargs -0 mkdir -p
cat ~/files.txt|xargs -0 touch
ENDSSH


--
:wq

Pandu Poluan 09-20-2012 03:54 PM

Recreating a directory structure and indicating a files presence
 
On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:

>

> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk> wrote:

> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:

> >

> >> On the archive:

> >>

> >> find /root/of/dir/structure -type d > dirs.txt

> >> find /root/of/dir/structure -type f > files.txt

> >

> > This will add '/root/of/dir/structure' to the start of each path. would

> > it be better to do?

> >

> > cd /root/of/dir/structure

> > find -type d > ../dirs.txt

> > find -type f > ../files.txt

>

> I see your path correction, and raise you:

> * whitespace-safe folders

> * Automatic copy to remote system.

> * Automatic new file and folder creation

> * Using those pretty xargs parameters.

>

> cd /root/of/dir/structure

> find . -type d -print0 > ~/dirs.txt

> find . -type d -print0 > ~/files.txt

>

> scp dirs.txt files.txt remote.system:

>

> ssh remote.system <<ENDSSH

> cd /root/of/new/structure

> cat ~/dirs.txt|xargs -0 mkdir -p

> cat ~/files.txt|xargs -0 touch

> ENDSSH

>

>


Cool... except that your raise is invalid (should've used -type f in the 3rd line)...


;-)


Rgds,

Michael Mol 09-20-2012 04:04 PM

Recreating a directory structure and indicating a files presence
 
On Thu, Sep 20, 2012 at 11:54 AM, Pandu Poluan <pandu@poluan.info> wrote:
>
> On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:
>>
>> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk>
>> wrote:
>> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>> >
>> >> On the archive:
>> >>
>> >> find /root/of/dir/structure -type d > dirs.txt
>> >> find /root/of/dir/structure -type f > files.txt
>> >
>> > This will add '/root/of/dir/structure' to the start of each path. would
>> > it be better to do?
>> >
>> > cd /root/of/dir/structure
>> > find -type d > ../dirs.txt
>> > find -type f > ../files.txt
>>
>> I see your path correction, and raise you:
>> * whitespace-safe folders
>> * Automatic copy to remote system.
>> * Automatic new file and folder creation
>> * Using those pretty xargs parameters.
>>
>> cd /root/of/dir/structure
>> find . -type d -print0 > ~/dirs.txt
>> find . -type d -print0 > ~/files.txt
>>
>> scp dirs.txt files.txt remote.system:
>>
>> ssh remote.system <<ENDSSH
>> cd /root/of/new/structure
>> cat ~/dirs.txt|xargs -0 mkdir -p
>> cat ~/files.txt|xargs -0 touch
>> ENDSSH
>>
>>
>
> Cool... except that your raise is invalid (should've used -type f in the 3rd
> line)...
>
> ;-)
>

Heh. It's also performing a bunch of unnecessary mkdir commands. I
mean, "find -type d" isn't going to return a subpath of a folder until
it's first returned the folder, so -p is unnecessary. For a better
speed gain, you'd want to keep -p, but filter out any folder which has
a subfolder, to reduce the number of mkdir commands issued. It's
possible mkdir performs this optimization internally, though. And I
don't know how to quickly do that filter on the command line.

--
:wq

Michael Mol 09-20-2012 04:05 PM

Recreating a directory structure and indicating a files presence
 
On Thu, Sep 20, 2012 at 12:04 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Thu, Sep 20, 2012 at 11:54 AM, Pandu Poluan <pandu@poluan.info> wrote:
>>
>> On Sep 20, 2012 10:04 PM, "Michael Mol" <mikemol@gmail.com> wrote:
>>>
>>> On Thu, Sep 20, 2012 at 10:48 AM, Neil Bothwick <neil@digimed.co.uk>
>>> wrote:
>>> > On Thu, 20 Sep 2012 16:13:08 +0200, Alan McKinnon wrote:
>>> >
>>> >> On the archive:
>>> >>
>>> >> find /root/of/dir/structure -type d > dirs.txt
>>> >> find /root/of/dir/structure -type f > files.txt
>>> >
>>> > This will add '/root/of/dir/structure' to the start of each path. would
>>> > it be better to do?
>>> >
>>> > cd /root/of/dir/structure
>>> > find -type d > ../dirs.txt
>>> > find -type f > ../files.txt
>>>
>>> I see your path correction, and raise you:
>>> * whitespace-safe folders
>>> * Automatic copy to remote system.
>>> * Automatic new file and folder creation
>>> * Using those pretty xargs parameters.
>>>
>>> cd /root/of/dir/structure
>>> find . -type d -print0 > ~/dirs.txt
>>> find . -type d -print0 > ~/files.txt
>>>
>>> scp dirs.txt files.txt remote.system:
>>>
>>> ssh remote.system <<ENDSSH
>>> cd /root/of/new/structure
>>> cat ~/dirs.txt|xargs -0 mkdir -p
>>> cat ~/files.txt|xargs -0 touch
>>> ENDSSH
>>>
>>>
>>
>> Cool... except that your raise is invalid (should've used -type f in the 3rd
>> line)...
>>
>> ;-)
>>
>
> Heh. It's also performing a bunch of unnecessary mkdir commands. I
> mean, "find -type d" isn't going to return a subpath of a folder until
^^ s/subpath/subfolder/ ^^


--
:wq

Andrew Lowe 09-23-2012 04:32 PM

Recreating a directory structure and indicating a files presence
 
On 09/20/12 22:13, Alan McKinnon wrote:

On Thu, 20 Sep 2012 20:33:39 +0800
Andrew Lowe <agl@wht.com.au> wrote:


Hi all,
I have the situation where I have a large amount of data,
many TB's, made up of many, many files. This information has now been
archived but I've got people who want to be able to see what data
does/does not exist, filling in gaps where they may exist.

As this data used to be available/visible in a directory
structure, ie via a file browser, I thought the easiest way for them
to see if something existed would be to create a mirror of the
directory structure and then populate this dir structure with 1 - 5
byte files with the same name as the "real" data files that now
reside in the archive. I've seen some scripts on the interweb that
allow me to create the dir structure, but does anyone have any ideas
how to do the creation of the "marker" files in the active file
system?

Just buying more hard disks and keeping the data on line also
isn't an option.

Any thoughts, greatly appreciated,

Andrew



I don't understand why you specify 1-5 byte files. Those few bytes will
always be useless. Rather use 0-length files.

On the archive:

find /root/of/dir/structure -type d > dirs.txt
find /root/of/dir/structure -type f > files.txt

Copy those two files to the on-line system:

for I in `cat dirs.txt` ; do mkdir -p $I ; done
for I in `cat files.txt` ; do touch -p $I ; done

Do that in the appropriate top-level directory of course. You can
probably make it more efficient using decent options to xargs, but what
the hell, I'd do it as-is. It's a once off action and finding the xargs
man page will take longer than the mkdirs....



Gentlemen,
Thanks for the suggestions, worked like a charm. I did the above and
except for whitespaces, which shouldn't have been there in the first
place and "-", things went well. As to the reason for 1 - 5 bytes, I
forgot about zero length files.....


Andrew


All times are GMT. The time now is 07:03 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.