FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 05-01-2012, 11:08 AM
Tuxoholic
 
Default W: best way to clone server data using rsync

Hi list

I'm about to clone 1TB of server data to a new file server I'm building.

Cloning will happen over the internal network, and it will take several days,
since I'll only run it while I'm awake/at home and checking up on the progress
every once in a while.

rsync seems like the right choice, but how will it handle job canceling when
I'm done for the day?

- will it resume files properly?
- will it run some sort of check sum to verify file integrity, or will I have
to run myself an integrity check like md5sum afterwards?


What's the right set of parameters to call rsync from a shell script? I'm not
used to sync such huge amount of files and directories, so I'm unclear about
how to call rsync in this case.

Here's what I got so far from google research:

rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive
--recursive --partial --partial-dir=rsync-part --progress --append
--files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log
root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool

LISTOFFILES.txt was created using:
rsync --list-only > /root/LISTOFFILES.txt


Can somebody comment on the parameters, e.g. how these will work fine with job
canceling and how this will handle file integrity?

Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task
the recommended/necessary way?


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BLU0-SMTP1800D3C7368BD6DDB7B248CD8290@phx.gbl">http://lists.debian.org/BLU0-SMTP1800D3C7368BD6DDB7B248CD8290@phx.gbl
 
Old 05-01-2012, 12:08 PM
"Lars Noodn"
 
Default W: best way to clone server data using rsync

On 5/1/12 2:08 PM, Tuxoholic wrote:
[snip]
> - will it resume files properly?
> - will it run some sort of check sum to verify file integrity, or will I have
> to run myself an integrity check like md5sum afterwards?

Yes it will resume and it will verify file integrity. Using the archive
option like you've chosen below, it will make sure that the target is
like the source.

[snip]
> Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task
> the recommended/necessary way?

It's safe to cancel, rsync will pick up what it has missed when it
starts again.

Regards,
/Lars


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F9FD24E.5000903@gmail.com">http://lists.debian.org/4F9FD24E.5000903@gmail.com
 
Old 05-01-2012, 04:38 PM
Indulekha
 
Default W: best way to clone server data using rsync

On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote:
> Hi list
>
> I'm about to clone 1TB of server data to a new file server I'm building.
>
> Cloning will happen over the internal network, and it will take several days,
> since I'll only run it while I'm awake/at home and checking up on the progress
> every once in a while.
>
> rsync seems like the right choice, but how will it handle job canceling when
> I'm done for the day?
>
> - will it resume files properly?

I use rsync for syncing and backups everyday, but I always do babysit it
or at least check it when it finishes. Not sure if it will "resume files
properly", but certainly it will update appropriately when used correctly.

> - will it run some sort of check sum to verify file integrity, or will I have
> to run myself an integrity check like md5sum afterwards?
>

Certainly wouldn't hurt, though I don't think it's necessary...

>
> What's the right set of parameters to call rsync from a shell script? I'm not
> used to sync such huge amount of files and directories, so I'm unclear about
> how to call rsync in this case.
>
> Here's what I got so far from google research:
>
> rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive
> --recursive --partial --partial-dir=rsync-part --progress --append
> --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log
> root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool
>
> LISTOFFILES.txt was created using:
> rsync --list-only > /root/LISTOFFILES.txt
>

Often I use something like:

rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete --progress --exclude-from '/home/indulekha/.rsync-exclude / /copy_of_system/

to duplicate a system, with the '~/.rsync-exclude' file containing
files and directories to ignore. Hopefully it's obvious that "copy_of_system"
is a path you determine.

> Can somebody comment on the parameters, e.g. how these will work fine with job
> canceling and how this will handle file integrity?

https://www.linux.com/news/enterprise/storage/8200-back-up-like-an-expert-with-rsynchttp://www.thegeekstuff.com/2010/09/rsync-command-examples/

> Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task
> the recommended/necessary way?
>

ctrl-c will work.

HTH!
--
❤ ♫ ❤ ♫ ❤ ♫ ❤
Indulekha


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120501163826.GB15438@radhesyama">http://lists.debian.org/20120501163826.GB15438@radhesyama


Tue May 1 20:30:01 2012
Return-path: <arch-general-bounces@archlinux.org>
Envelope-to: tom@linux-archive.org
Delivery-date: Tue, 01 May 2012 19:39:36 +0300
Received: from gerolde.archlinux.org ([66.211.214.132]:44310)
by s2.java-tips.org with esmtp (Exim 4.77)
(envelope-from <arch-general-bounces@archlinux.org>)
id 1SPG6x-0004Ux-Us
for tom@linux-archive.org; Tue, 01 May 2012 19:39:36 +0300
Received: from gudrun.archlinux.org (gudrun.archlinux.org [66.211.214.131])
by gerolde.archlinux.org (Postfix) with ESMTP id 6F4CD9009D;
Tue, 1 May 2012 12:39:14 -0400 (EDT)
Received: from gerolde.archlinux.org (gerolde.archlinux.org [66.211.214.132])
by gudrun.archlinux.org (Postfix) with ESMTP id 4B7C57004C
for <arch-general@archlinux.org>; Tue, 1 May 2012 12:39:13 -0400 (EDT)
Received-SPF: pass (gmail.com ... _spf.google.com: 209.85.214.44 is authorized
to use 'techlivezheng@gmail.com' in 'mfrom' identity (mechanism
'ip4:209.85.128.0/17' matched)) receiver=gerolde.archlinux.org;
identity=mailfrom; envelope-from="techlivezheng@gmail.com";
helo=mail-bk0-f44.google.com; client-ip 9.85.214.44
Received: from mail-bk0-f44.google.com (mail-bk0-f44.google.com
[209.85.214.44])
by gerolde.archlinux.org (Postfix) with ESMTPS id DE39B9009C
for <arch-general@archlinux.org>; Tue, 1 May 2012 12:39:12 -0400 (EDT)
Received: by bkuw5 with SMTP id w5so2540027bku.3
for <arch-general@archlinux.org>; Tue, 01 May 2012 09:39:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s 120113;
h=mime-version:in-reply-to:references:from:date:message-id:subject:to
:content-type; bh�RxBgPiwWXdJgCd1HomW1EoT/YVcyK1jEe5TIzub0g=;
b�WWSRitulrq3JgBaZsHBp7FSxL3HCrqDpMXFCHpEbpmL5K4 dqT2ghdWAtXNZqhVB
nsHQmcHN09lJapK/WVJ6LHLXY1C9/4XBlU+A/gLyNy1rMgR8T9rA2vOa7yNeMUeC6UVZ
u1yxXw7mQJsVG/whTHDGMYNmysvtfokRzGEHCtOtibmKvUaDdI+SC93ognnumfWI wqRC
vAmu2wIBoIP9lqvI2EktoxzJ8DuZZc7wax3jScw8Ux5Xjm2LTU 3/V74+J2O3y8kYhCH/
9StWQ97W7CaEIaTzS/Zg6rfqot2bBR0GIKeitS4c9r12Oy0p13mKmJAhZbr8lTqV0S5f
J19Q=Received: by 10.204.156.141 with SMTP id x13mr8240893bkw.50.1335890376264;
Tue, 01 May 2012 09:39:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.71.196 with HTTP; Tue, 1 May 2012 09:38:56 -0700 (PDT)
In-Reply-To: <CACNNBMpBnpYiUwOOtcR2SKokpDru85wX4bRzCz0rtg406J m-PQ@mail.gmail.com>
References: <CACNNBMpFCMvfz2PZyGCALJdX2Hg30b7ZkJUgYeH2m4J=T_TS MQ@mail.gmail.com>
<1333629931.23448.1.camel@localhost.localdomain>
<CACNNBMpBnpYiUwOOtcR2SKokpDru85wX4bRzCz0rtg406J m-PQ@mail.gmail.com>
From: =?UTF-8?B?6YOR5paH6L6JKFRlY2hsaXZlIFpoZW5nKQ==? <techlivezheng@gmail.com>
Date: Wed, 2 May 2012 00:38:56 +0800
Message-ID: <CAPYzjrR0joRxO02i2rt5nkaxj+UEx7eWDh-6hEFBU+R91itDEg@mail.gmail.com>
To: General Discussion about Arch Linux <arch-general@archlinux.org>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [arch-general] Installing in VirtualBox - Screen goes black
during installation
X-BeenThere: arch-general@archlinux.org
X-Mailman-Version: 2.1.14
Precedence: list
Reply-To: General Discussion about Arch Linux <arch-general@archlinux.org>
List-Id: General Discussion about Arch Linux <arch-general.archlinux.org>
List-Unsubscribe: <http://mailman.archlinux.org/mailman/options/arch-general>,
<mailto:arch-general-request@archlinux.org?subject=unsubscribe>
List-Archive: <http://mailman.archlinux.org/pipermail/arch-general>
List-Post: <mailto:arch-general@archlinux.org>
List-Help: <mailto:arch-general-request@archlinux.org?subject=help>
List-Subscribe: <http://mailman.archlinux.org/mailman/listinfo/arch-general>,
<mailto:arch-general-request@archlinux.org?subject=subscribe>
Errors-To: arch-general-bounces@archlinux.org
Sender: arch-general-bounces@archlinux.org

2012/4/5 Joshua Poehls <joshua@zduck.com>:
> Doh. I think that was it. Thank you...
>
> --
> Joshua Poehls
>
>
>
> On Thu, Apr 5, 2012 at 7:45 AM, Jesse Juhani Jaara <jesse.jaara@gmail.com>wrote:
>
>> to, 2012-04-05 kello 07:41 -0500, Joshua Poehls kirjoitti:
>> > I started installing Arch in VirtualBox this morning but twice now the
>> > screen goes black at some point during (or right after) the package
>> > download step and never comes back.
>> Well Linux in general has a "powersaving" or "screensaving" feature
>> don't know the exact meaning/use of it, but anyway in console, if you
>> don't touch the keyboard for a while the screen will blank itself.
>> Pressing any key will wake it up again. Don't know if this is the thing
>> you are experiencing...
>>
I have similar problmes here. After upgrade to the newest linux
kernel, the virtualbox became unresponsed, and output a screen of
dignostic informations.

Anyone knows what happend?
 
Old 05-01-2012, 04:43 PM
Dan Ritter
 
Default W: best way to clone server data using rsync

On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote:
> Here's what I got so far from google research:
>
> rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive
> --recursive --partial --partial-dir=rsync-part --progress --append
> --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log
> root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool

You don't need the sockopts. -a includes recursive. You probably
don't want partial or partial-dir. You probably don't want
append.

Having rsh installed is a bad security risk. Install ssh, make
sure you can ssh from this box to myserver as root. Change
-e rsh to -e ssh.

-dsr-

--
http://randomstring.org/~dsr/eula.html is hereby incorporated by reference.
You can't fight for freedom by taking away rights.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120501164330.GX11128@randomstring.org">http://lists.debian.org/20120501164330.GX11128@randomstring.org
 
Old 05-01-2012, 08:19 PM
Tuxoholic
 
Default W: best way to clone server data using rsync

I noticed rsh is not installed, it's linked to /etc/alternatives/rsh,
which is linked to /usr/bin/ssh.


Calling rsh instead of ssh should avoid file encryption during transfer,
at least that was the intention.


The socket options boost transfer speed quite a lot, I usually have
65355 buffers on my samba server, using rsync directly I can increase it
some more.


partial-dir does not work along with append, so I got rid of it.

I'll leave the rest as is minus the recursive option

Thanks for the inputs!


On 01.05.2012 18:43, Dan Ritter wrote:

On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote:

Here's what I got so far from google research:

rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive
--recursive --partial --partial-dir=rsync-part --progress --append
--files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log
root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool


You don't need the sockopts. -a includes recursive. You probably
don't want partial or partial-dir. You probably don't want
append.

Having rsh installed is a bad security risk. Install ssh, make
sure you can ssh from this box to myserver as root. Change
-e rsh to -e ssh.

-dsr-




--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: BLU0-SMTP363466E7FA2C0053691FDD4D8290@phx.gbl">http://lists.debian.org/BLU0-SMTP363466E7FA2C0053691FDD4D8290@phx.gbl
 
Old 05-02-2012, 06:16 AM
Chris Bannister
 
Default W: best way to clone server data using rsync

On Tue, May 01, 2012 at 11:38:26AM -0500, Indulekha wrote:
> Often I use something like:
>
> rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete ...
^^^^^^^^
> to duplicate a system, with the '~/.rsync-exclude' file containing
^^^^^^^^^^^^^
Is that a mistake? or am I musunderstanding the "--delete" option.

Note: I am not familiar with rsync, but I am familiar with the term
"duplicate"

--
"Religion is excellent stuff for keeping common people quiet."
-- Napoleon Bonaparte


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120502061658.GC13218@tal">http://lists.debian.org/20120502061658.GC13218@tal
 
Old 05-02-2012, 06:38 AM
Indulekha
 
Default W: best way to clone server data using rsync

On Wed, May 02, 2012 at 06:16:59PM +1200, Chris Bannister wrote:
> On Tue, May 01, 2012 at 11:38:26AM -0500, Indulekha wrote:
> > Often I use something like:
> >
> > rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete ...
> ^^^^^^^^
> > to duplicate a system, with the '~/.rsync-exclude' file containing
> ^^^^^^^^^^^^^
> Is that a mistake? or am I musunderstanding the "--delete" option.
>
> Note: I am not familiar with rsync, but I am familiar with the term
> "duplicate"
>

No mistake Chris, the "--delete" option deletes data from the
copy that doesn't exist in the source. The idea is to have only
the same data as the source you're copying, aside from the
stuff to ignore listed in the "~/.rsync-exclude" file (stuff
like /etc/fstab, if you're making a bootable copy of your system
on an external drive).

The links I gave will give a lot more detail, and rsync's
man page is quite good too, though perhaps a tad opaque in
places for the uninitiated.

--
❤ ♫ ❤ ♫ ❤ ♫ ❤
Indulekha


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120502063806.GB28630@radhesyama">http://lists.debian.org/20120502063806.GB28630@radhesyama
 
Old 05-02-2012, 09:14 AM
Jon Dowland
 
Default W: best way to clone server data using rsync

On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote:
> rsync seems like the right choice, but how will it handle job canceling when
> I'm done for the day?

Yes. But, for the initial sync, I tend to prefer a tarpipe which is a lot quicker.

( cd /srcdir && tar cf - . ) | ssh user@somehost '( cd /destdir && tar xf - )'

Add 'z' to the tar arguments (on both sides) if you have reasonable CPUs and
your data is not already compressed (don't bother for MPEGs, JPEGs, etc.; or
if one end of the transfer is an embedded system).

Once that has run, or if it was interrupted, I finish off with an rsync just
to be sure. (others have already commented on your rsync arguments.)


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120502091402.GG17757@debian">http://lists.debian.org/20120502091402.GG17757@debian
 
Old 05-02-2012, 01:57 PM
Dan Ritter
 
Default W: best way to clone server data using rsync

On Tue, May 01, 2012 at 10:19:42PM +0200, Tuxoholic wrote:
> I noticed rsh is not installed, it's linked to
> /etc/alternatives/rsh, which is linked to /usr/bin/ssh.
>
> Calling rsh instead of ssh should avoid file encryption during
> transfer, at least that was the intention.

That isn't what will happen. SSH will not allow you to turn off
encryption.

You may want to specify -O Ciphers=rc4 (or blowfish), both of
which are quite fast.

> The socket options boost transfer speed quite a lot, I usually have
> 65355 buffers on my samba server, using rsync directly I can
> increase it some more.

Have you tested this with ssh? It seems unlikely to me.

-dsr-

--
http://randomstring.org/~dsr/eula.html is hereby incorporated by reference.
You can't fight for freedom by taking away rights.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120502135739.GY11128@randomstring.org">http://lists.debian.org/20120502135739.GY11128@randomstring.org
 

Thread Tools




All times are GMT. The time now is 03:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org