Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   CentOS (http://www.linux-archive.org/centos/)
-   -   wget and resulting filenames with mirror option (http://www.linux-archive.org/centos/405386-wget-resulting-filenames-mirror-option.html)

"Joseph L. Casale" 07-28-2010 08:26 PM

wget and resulting filenames with mirror option
 
I am trying to mirror a directory using wget and the resulting files as
expected are taking the name of the string in the url after the last "/".

Anyone know a way around this? I am using the mirror option so I don't
have to keep track of what to get making it simple.

Thanks!
jlc
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Todd Denniston 07-28-2010 08:44 PM

wget and resulting filenames with mirror option
 
Joseph L. Casale wrote, On 07/28/2010 04:26 PM:
> I am trying to mirror a directory using wget and the resulting files as
> expected are taking the name of the string in the url after the last "/".
>
> Anyone know a way around this? I am using the mirror option so I don't
> have to keep track of what to get making it simple.
>
> Thanks!
> jlc

I am having a bit of trouble parsing what you wrote, but I think you are saying you are issuing
something like:
wget --mirror http://example.com/subdir1/subdir2/fileiwant
and getting at your location
./fileiwant
which is a little confusing, because I would have expected you to get (with just the --mirror option):
./example.com/subdir1/subdir2/fileiwant

please give an example command and output received and desired.

to get ./fileiwant from my example url above I think you would have to do:
wget --mirror ---cut-dirs=2 -no-host-directories
http://example.com/subdir1/subdir2/fileiwant

--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

"Joseph L. Casale" 07-28-2010 08:51 PM

wget and resulting filenames with mirror option
 
>I am having a bit of trouble parsing what you wrote,

Ok, the exact use is the mirroring of vm directories on an esxi
server over to a centos storage server.

The vm's folder url is:
https://10.0.0.61/folder/vm-name?dcPath=ha-datacenter&dsName=Datastore_2
for example.

When it begins mirroring it, it see's urls like this:
https://10.0.0.61/folder/vm-name/Disc-1-flat.vmdk?dcPath=ha-datacenter&dsName=Datastore_2
so the resulting file downloaded is not Disc-1-flat.vmdk, but:
Disc-1-flat.vmdk?dcPath=ha-datacenter&dsName=Datastore_2
Thanks!
jlc

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

R P Herrold 07-28-2010 09:36 PM

wget and resulting filenames with mirror option
 
On Wed, 28 Jul 2010, Joseph L. Casale wrote:

> Anyone know a way around this? I am using the mirror option so I don't
> have to keep track of what to get making it simple.

my apology in advance for suggesting another tool, but, there
you are ...

use lftp for mirroring? wget can do mirroring but it is
clunky compared to lftp, and much harder to script. man pages
are worse, to my taste

Looking, it seems I have a bit over 1400 files driving lftp
that basically look like this on a local interior mirror

[root@xps400 ~]# wc -l *.conf *.conf_ | tail -1
1490 total

[root@xps400 ~]# cat lftp-ORC.conf
#
# inside at ftp.first, this is NOT automatically run
#
mirror -c -e
ftp://ftp.owlriver.com/pub/local/ORC
/var/ftp/pub/mirror/ORC
#

-------

One can invoke it as simply as:

lftp -f lftp-ORC.conf

and it will of course run non-root if one desires; it can be
wrappered up in a scriptlet (I use a flock scheme in the
wrappering so it does not overrun into a prior day's run;
mirroring runs continuously) ...

http://bugs.centos.org/view.php?id=4443

suggests a simple way to gather detail:

export YMD=`date +%Y%m%d`
cd $TOP
date > DIRECTORY_SIZES.txt
date > MANIFEST-${YMD}.txt
find . -type d -exec du -sh {} ; >> DIRECTORY_SIZES.txt
find . -type f ; >> MANIFEST-${YMD}.txt

and diff to taste, optionally automailing when more than the
top 'date' line changes

-- Russ herrold
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Keith Roberts 07-28-2010 09:55 PM

wget and resulting filenames with mirror option
 
On Wed, 28 Jul 2010, R P Herrold wrote:


To: CentOS mailing list <centos@centos.org>
From: R P Herrold <herrold@owlriver.com>
Subject: [CentOS] wget and resulting filenames with mirror option

On Wed, 28 Jul 2010, Joseph L. Casale wrote:


Anyone know a way around this? I am using the mirror option so I don't
have to keep track of what to get making it simple.


my apology in advance for suggesting another tool, but, there
you are ...


What about rsync ?

rsync(1)

NAME
rsync — a fast, versatile, remote (and local) file-copying tool

SYNOPSIS
Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...]
rsync://[USER@]HOST[:PORT]/SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC...
rsync://[USER@]HOST[:PORT]/DEST


Usages with just one SRC arg and no DEST arg will list the source
files instead of copying.

DESCRIPTION
Rsync is a fast and extraordinarily versatile file copying tool. It
can copy locally, to/from another host over any remote shell, or
to/from a remote rsync daemon. It offers a large number of options
that control every aspect of its behavior and permit very flexible
specification of the set of files to be copied. It is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files

...

Regards

Keith_____________________________________________ __
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

R P Herrold 07-28-2010 10:10 PM

wget and resulting filenames with mirror option
 
On Wed, 28 Jul 2010, Keith Roberts wrote:

>> my apology in advance for suggesting another tool, but, there
>> you are ...
>
> What about rsync ?

That requires co-operation from the remote end to set up an
anonymous rsync server .. doable, but as I understood the
scope of work, not within it

-- Russ herrold
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Whit Blauvelt 07-28-2010 10:16 PM

wget and resulting filenames with mirror option
 
On Wed, Jul 28, 2010 at 08:51:29PM +0000, Joseph L. Casale wrote:

> When it begins mirroring it, it see's urls like this:
> https://10.0.0.61/folder/vm-name/Disc-1-flat.vmdk?dcPath=ha-datacenter&dsName=Datastore_2
> so the resulting file downloaded is not Disc-1-flat.vmdk, but:
> Disc-1-flat.vmdk?dcPath=ha-datacenter&dsName=Datastore_2

Can't do it off the top of my head, but it would be a pretty simple Perl
script to simply go through the directory and rename the files, dropping the
? and everything after.

Whit
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

"Joseph L. Casale" 07-28-2010 10:22 PM

wget and resulting filenames with mirror option
 
>That requires co-operation from the remote end to set up an
>anonymous rsync server .. doable, but as I understood the
>scope of work, not within it

Well, doesn't need to be anon, but none the less I am SOL wrt
to ftp or rsync as none of these daemons run locally on the
host.

Someone pointed me to an existing script that accomplishes what
I need and is actively maintained so I am in luck.

Thanks guys, and I will look into lftp as I also don't like wget
for much more than fast one off's at the cli.
jlc
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

R P Herrold 07-28-2010 10:28 PM

wget and resulting filenames with mirror option
 
On Wed, 28 Jul 2010, Joseph L. Casale wrote:

>> That requires co-operation from the remote end to set up an
>> anonymous rsync server .. doable, but as I understood the
>> scope of work, not within it
>
> Well, doesn't need to be anon, but none the less I am SOL wrt
> to ftp or rsync as none of these daemons run locally on the
> host.

lftp does www (and through a SSL https) mirroring just fine
too ... I just did not mention it, as it did not occur to me
from the initial question. As wget is such a 'pill' my eyes
glazed over and I missed it in the follow on post you made

-- Russ herrold
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Les Mikesell 07-28-2010 10:33 PM

wget and resulting filenames with mirror option
 
On 7/28/2010 5:10 PM, R P Herrold wrote:
> On Wed, 28 Jul 2010, Keith Roberts wrote:
>
>>> my apology in advance for suggesting another tool, but, there
>>> you are ...
>>
>> What about rsync ?
>
> That requires co-operation from the remote end to set up an
> anonymous rsync server .. doable, but as I understood the
> scope of work, not within it

Or, if you have ssh access already it just works. But, I thought ESXi
was mentioned somewhere and while it is possible to enable ssh on it, I
don't think it has rsync.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


All times are GMT. The time now is 06:12 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.