FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 01-29-2011, 12:54 PM
Etaoin Shrdlu
 
Default Cloning a directory hierarchy, but not the content

On Sat, 29 Jan 2011 14:58:13 +0100
Alex Schuster <wonko@wonkology.org> wrote:

> Hi there!
>
> I am currently putting extra backups to old hard drives I do no longer
> need for other purposes. After that I send the putput out ls -lR and du
> -m to my log directory so I can check what files are on which drive
> without having to attach the drive.
>
> Works, though a better method would be to clone the drive's root
> directory, but with all file sizes being zero. This way I can easily
> navigate the directory structure, instead of browsing through the ls-lR
> file. Is there a utility that does this? It would be even better if the
> files would be created as sparse files, faking the original size.
>
> I just wrote a little script that does this, but it does not do the
> sparse file thing yet, and would have problems with newline in file
> names. And I guess someone already wrote such a utility?

IIUC, try

find / -type d -exec sh 'mkdir -p target"$1"' - {} ;
 
Old 01-29-2011, 12:58 PM
Alex Schuster
 
Default Cloning a directory hierarchy, but not the content

Hi there!

I am currently putting extra backups to old hard drives I do no longer need
for other purposes. After that I send the putput out ls -lR and du -m to my
log directory so I can check what files are on which drive without having to
attach the drive.

Works, though a better method would be to clone the drive's root directory,
but with all file sizes being zero. This way I can easily navigate the
directory structure, instead of browsing through the ls-lR file. Is there a
utility that does this? It would be even better if the files would be
created as sparse files, faking the original size.

I just wrote a little script that does this, but it does not do the sparse
file thing yet, and would have problems with newline in file names. And I
guess someone already wrote such a utility?

Wonko
 
Old 01-29-2011, 01:27 PM
Alex Schuster
 
Default Cloning a directory hierarchy, but not the content

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 14:58:13 +0100
>
> Alex Schuster <wonko@wonkology.org> wrote:
> > Hi there!
> >
> > I am currently putting extra backups to old hard drives I do no longer
> > need for other purposes. After that I send the putput out ls -lR and du
> > -m to my log directory so I can check what files are on which drive
> > without having to attach the drive.
> >
> > Works, though a better method would be to clone the drive's root
> > directory, but with all file sizes being zero. This way I can easily
> > navigate the directory structure, instead of browsing through the ls-lR
> > file. Is there a utility that does this? It would be even better if the
> > files would be created as sparse files, faking the original size.
> >
> > I just wrote a little script that does this, but it does not do the
> > sparse file thing yet, and would have problems with newline in file
> > names. And I guess someone already wrote such a utility?
>
> IIUC, try
>
> find / -type d -exec sh 'mkdir -p target"$1"' - {} ;

Hmm, that does not really seem to work. It tries to execute the whole stuff
between single quotes as a command. And I don't really understand what it is
supposed to do, shouldn't this be something like mkdir -p
/destination/$1/{} ?

Anyway, this is what I already have. It duplicates the hierarchy with empty
files, but I have to add support for sparse files. That won't be too hard,
but maybe I'm re-inventing the wheel here.

#!/bin/bash

src=$1
dst=$2

cd "$src" || exit $?
IFS=$'
'
find . |
while read file
do
if [[ -d $file ]]
then
[[ -d "$dst/$file" ]] ||
mkdir -p "$dst/$file"
elif [[ -f $file ]]
then
[[ -d "$dst/${file%/*}" ]] ||
mkdir -p "$dst/${file%/*}"
touch "$dst/$file"
fi
done


Wonko
 
Old 01-29-2011, 01:36 PM
Etaoin Shrdlu
 
Default Cloning a directory hierarchy, but not the content

On Sat, 29 Jan 2011 15:27:59 +0100 Alex Schuster <wonko@wonkology.org>
wrote:

> > > I just wrote a little script that does this, but it does not do the
> > > sparse file thing yet, and would have problems with newline in file
> > > names. And I guess someone already wrote such a utility?
> >
> > IIUC, try
> >
> > find / -type d -exec sh 'mkdir -p target"$1"' - {} ;
>
> Hmm, that does not really seem to work. It tries to execute the whole
> stuff between single quotes as a command. And I don't really understand
> what it is supposed to do, shouldn't this be something like mkdir -p
> /destination/$1/{} ?

No. That recreates the full directory hierarchy based at / under /target/,
with no files in it. Just the directory hierarchy. I should have added
that, to do it safely, the target should reside higher than the source in
the hierarchy, or it should be on a different filesystem and in that case
-xdev should be specified to find (otherwise an recursive loop would
result).

A more sensible approach would probably be

cd /source && find . -type d -exec bash 'mkdir -p "${@/#//target/}"' - {} +

with -xdev if needed. But as I see now, this is not what you wanted, so
ignore it.

> Anyway, this is what I already have. It duplicates the hierarchy with
> empty files, but I have to add support for sparse files. That won't be
> too hard, but maybe I'm re-inventing the wheel here.
>
> #!/bin/bash
>
> src=$1
> dst=$2
>
> cd "$src" || exit $?
> IFS=$'
'
> find . |
> while read file
> do
> if [[ -d $file ]]
> then
> [[ -d "$dst/$file" ]] ||
> mkdir -p "$dst/$file"
> elif [[ -f $file ]]
> then
> [[ -d "$dst/${file%/*}" ]] ||
> mkdir -p "$dst/${file%/*}"
> touch "$dst/$file"
> fi
> done

Ok, I misunderstood. You also want the files but empty. Why do you need
support for sparse files? Do you need to manage other types of file
(symlinks, FIFOs, etc.)
 
Old 01-29-2011, 01:39 PM
Florian Philipp
 
Default Cloning a directory hierarchy, but not the content

Am 29.01.2011 14:58, schrieb Alex Schuster:
> Hi there!
>
> I am currently putting extra backups to old hard drives I do no longer need
> for other purposes. After that I send the putput out ls -lR and du -m to my
> log directory so I can check what files are on which drive without having to
> attach the drive.
>
> Works, though a better method would be to clone the drive's root directory,
> but with all file sizes being zero. This way I can easily navigate the
> directory structure, instead of browsing through the ls-lR file. Is there a
> utility that does this? It would be even better if the files would be
> created as sparse files, faking the original size.
>
> I just wrote a little script that does this, but it does not do the sparse
> file thing yet, and would have problems with newline in file names. And I
> guess someone already wrote such a utility?
>
> Wonko
>

Use `truncate -s <size> <file>`

It creates a sparse file if the specified file is smaller than the
specified size. It will also create a new file if it does not yet exist.

In order to avoid trouble with line breaks in names, I recommend
something like:
find . -type f -print0 |
while read -d $'' file; do
echo "File=$file"
done

Or use similar commands accepting or outputting 0-byte terminated
strings, for example xargs -0, du -0, grep -z.

For copying file attributes from one file to another you can use `cp
--attributes-only`.

Hope this helps,
Florian Philipp
 
Old 01-29-2011, 03:45 PM
Alex Schuster
 
Default Cloning a directory hierarchy, but not the content

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 15:27:59 +0100 Alex Schuster <wonko@wonkology.org>
> wrote:
> > > > I just wrote a little script that does this, but it does not do the
> > > > sparse file thing yet, and would have problems with newline in file
> > > > names. And I guess someone already wrote such a utility?
> > >
> > > IIUC, try
> > >
> > > find / -type d -exec sh 'mkdir -p target"$1"' - {} ;
> >
> > Hmm, that does not really seem to work. It tries to execute the whole
> > stuff between single quotes as a command. And I don't really understand
> > what it is supposed to do, shouldn't this be something like mkdir -p
> > /destination/$1/{} ?
>
> No. That recreates the full directory hierarchy based at / under
> /target/, with no files in it. Just the directory hierarchy.

Ah, now I get it. There's a -c missing after the sh command.

> I should
> have added that, to do it safely, the target should reside higher than
> the source in the hierarchy, or it should be on a different filesystem
> and in that case -xdev should be specified to find (otherwise an
> recursive loop would result).

Right, but not important in my case. I want to mount my backup drive to
/mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up
much space. Then I can remove the backup drive and I only have to mount it
again when I need a file's content, but not for finding out which files
there are and how much space they take. Well, the space already is in the
file created by du -m, but I'd like to directly navigate around.


> Ok, I misunderstood. You also want the files but empty. Why do you need
> support for sparse files? Do you need to manage other types of file
> (symlinks, FIFOs, etc.)

Yes, symlinks would ne nice, too, I forgot about them. The rest is
unimportant, as this would be data only, not root file systems. I backup
that with rdiff-backup to a 2nd drive, but there's much other stuff that I
would like to put on one of the old drives that lie around here.

Sparse files would be nice because then I do not only have the same logical
structure, the files also appear to have the same size as the originals,
instead of having a size of 0. I could navigate and explore the directory
structure with mc, and with du --apparent-size I could find out how much
space a subdirectory takes. Again, my du -m file already has this
information, but while navigating in the directory tree, being able to use
du would be nice.

Wonko
 
Old 01-29-2011, 05:59 PM
Etaoin Shrdlu
 
Default Cloning a directory hierarchy, but not the content

On Sat, 29 Jan 2011 17:45:30 +0100 Alex Schuster <wonko@wonkology.org>
wrote:

> Ah, now I get it. There's a -c missing after the sh command.

Right, thans for spotting it.


> > I should have added that, to do it safely, the target should reside
> > higher than the source in the hierarchy, or it should be on a different
> > filesystem and in that case -xdev should be specified to find
> > (otherwise an recursive loop would result).
>
> Right, but not important in my case. I want to mount my backup drive to
> /mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up
> much space. Then I can remove the backup drive and I only have to mount
> it again when I need a file's content, but not for finding out which
> files there are and how much space they take. Well, the space already is
> in the file created by du -m, but I'd like to directly navigate around.

Oh, I see now: you want the files to *look like* the real ones (eg when
doing ls -l etc.), but be sparse so they don't take up space?

> Sparse files would be nice because then I do not only have the same
> logical structure, the files also appear to have the same size as the
> originals, instead of having a size of 0. I could navigate and explore
> the directory structure with mc, and with du --apparent-size I could find
> out how much space a subdirectory takes. Again, my du -m file already has
> this information, but while navigating in the directory tree, being able
> to use du would be nice.

Ok, one way to create a sparse file of, say, 1 megabyte is using dd:

# dd if=/dev/null of=sparsefile bs=1 seek=1M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.5419e-05 s, 0.0 kB/s
# ls -l sparsefile
-rw-r--r-- 1 root root 1048576 Jan 29 11:57 sparsefile
# du -B1 sparsefile
0 sparsefile

Another way, already suggested, is by using truncate, eg

# truncate -s 1M sparsefile
 
Old 01-29-2011, 06:31 PM
Alex Schuster
 
Default Cloning a directory hierarchy, but not the content

Florian Philipp writes:

> Use `truncate -s <size> <file>`
>
> It creates a sparse file if the specified file is smaller than the
> specified size. It will also create a new file if it does not yet exist.

Nice one. First I did not see an improvement over using dd to create the
sparse file, but in combination with cp --attributes-only I can now
duplicate the file with its attributes but zero size, and then fake the
size.

> In order to avoid trouble with line breaks in names, I recommend
> something like:
> find . -type f -print0 |
> while read -d $'' file; do
> echo "File=$file"
> done

Thanks!


> For copying file attributes from one file to another you can use `cp
> --attributes-only`.

Oh my, another case of a (german) man page that does not show all the
possible arguments. Never heard about that, thanks!

Wonko
 
Old 01-29-2011, 08:14 PM
Florian Philipp
 
Default Cloning a directory hierarchy, but not the content

Am 29.01.2011 20:31, schrieb Alex Schuster:
> Florian Philipp writes:
[...]
>
>
>> For copying file attributes from one file to another you can use `cp
>> --attributes-only`.
>
> Oh my, another case of a (german) man page that does not show all the
> possible arguments. Never heard about that, thanks!
>
> Wonko
>

The German man-pages can be a real pain. Guess there are too few people
who keep them up-to-date. Oh well ... maybe some day when I have more time.

Until then I make it a point to never install them in the first place
when I can avoid it:
`LINGUAS='en' emerge -1 man-pages && emerge -C man-pages-de`

Or I just delete them with `rm -r /usr/share/man/de*`
 
Old 01-29-2011, 10:51 PM
Alex Schuster
 
Default Cloning a directory hierarchy, but not the content

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 17:45:30 +0100 Alex Schuster <wonko@wonkology.org>
> wrote:

>>> I should have added that, to do it safely, the target should reside
>>> higher than the source in the hierarchy, or it should be on a different
>>> filesystem and in that case -xdev should be specified to find
>>> (otherwise an recursive loop would result).
>>
>> Right, but not important in my case. I want to mount my backup drive to
>> /mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up
>> much space. Then I can remove the backup drive and I only have to mount
>> it again when I need a file's content, but not for finding out which
>> files there are and how much space they take. Well, the space already is
>> in the file created by du -m, but I'd like to directly navigate around.
>
> Oh, I see now: you want the files to *look like* the real ones (eg when
> doing ls -l etc.), but be sparse so they don't take up space?

Exactly. Sorry I did not make myself clearer.
It's working now, and I like it I added some more features, like
clipping files to a maximum size. So the clone can still be very small
compared to the original, with small files being intact and usable.


> Ok, one way to create a sparse file of, say, 1 megabyte is using dd:
>
> # dd if=/dev/null of=sparsefile bs=1 seek=1M
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 2.5419e-05 s, 0.0 kB/s
> # ls -l sparsefile
> -rw-r--r-- 1 root root 1048576 Jan 29 11:57 sparsefile
> # du -B1 sparsefile
> 0 sparsefile

That's how I wanted to do it first, too.

> Another way, already suggested, is by using truncate, eg
>
> # truncate -s 1M sparsefile

I used this, because so I can modify a file that I created empty with cp
--attributes-only. Keeping the attributes would have been a bit complicated.

In case anyone else is interested, the script is here:
http://www.wonkology.org/utils/clone0

wonko@weird ~ $ clone0 -h
clone0 version 2011-01-29
Duplicate a file / directory hierarchy. Files are
created as sparse files, not taking up real space.

Usage: clone0 [-dhSv0] [-s size] src... dst

Options:
-d clone directory structure only, not files
-h show this help
-s size copy files up to size as the are, and clip larger files
-S do not create sparse files
-v show directories being created
-vv show files being created
-vvv debug output
-0 clip files larger than size (option -s) to zero size

Arguments:
src... one or more directories to clone
dst destination directory (will be created)


Thanks for the input, guys!

Wonko
 

Thread Tools




All times are GMT. The time now is 06:38 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org