recommendations for copying large filesystems
 

I need to copy over 100TB of data from one server to another via network. What is the best option to do this? I am planning to use rsync but is there a better tool or better way of doing this?

For example, I plan on doing

rsync -azv /largefs /targetfs

/targetfs is a NFS mounted filesystem.

Any thoughts?

TIA



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Mag Gam wrote:
I need to copy over 100TB of data from one server to another via
network. What is the best option to do this? I am planning to use
rsync but is there a better tool or better way of doing this?


For example, I plan on doing
rsync -azv /largefs /targetfs

/targetfs is a NFS mounted filesystem.

Any thoughts?


rsync would probably work better if you ran it in client server mode
rather than over NFS, especially if you have to restart it.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Mag Gam wrote:

I need to copy over 100TB of data from one server to another via network.
What is the best option to do this? I am planning to use rsync but is there
a better tool or better way of doing this?

For example, I plan on doing
rsync -azv /largefs /targetfs

/targetfs is a NFS mounted filesystem.


The only problem you are likely to have is that rsync reads the entire
directory contents into RAM before starting, then walks the list fixing
the differences. If you have a huge number of files and a small amount
of RAM, it may slow down due to swapping. 'cp -a ' can be faster if the
target doesn't already have any matching files. Also, the -v to display
the names can take longer than the file transfer on small files.
Running rsync over ssh instead of nfs has a tradeoff in that the remote
does part of the work but you lose some speed to ssh encryption. If the
filesystem is live, you might make an initial run copying the larger
directories with rsync or cp, then do whatever you can to stop the files
from changing and make another pass with 'rsync --av --delete' which
should go fairly quickly and and fix any remaining differences.


--
Les Mikesell
lesmikesell@gmail.com


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Am 21.06.2008 um 15:33 schrieb Mag Gam:

I need to copy over 100TB of data from one server to another via
network. What is the best option to do this? I am planning to use
rsync but is there a better tool or better way of doing this?


For example, I plan on doing
rsync -azv /largefs /targetfs

/targetfs is a NFS mounted filesystem.





What network link is there between these hosts?

Are these 1 or 2 million small files or bigger ones?

Does the data change a lot?

Is it a SAN or JBOD?


cheers,
Rainer
--
Rainer Duffner
CISSP, LPI, MCSE
rainer@ultra-secure.de


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Network is a 10/100
1 million large files
No SAN, JBOD



On Sat, Jun 21, 2008 at 1:19 PM, Rainer Duffner <rainer@ultra-secure.de> wrote:



Am 21.06.2008 um 15:33 schrieb Mag Gam:




I need to copy over 100TB of data from one server to another via network. What is the best option to do this? I am planning to use rsync but is there a better tool or better way of doing this?



For example, I plan on doing

rsync -azv /largefs /targetfs



/targetfs is a NFS mounted filesystem.










What network link is there between these hosts?



Are these 1 or 2 million small files or bigger ones?



Does the data change a lot?



Is it a SAN or JBOD?





cheers,

Rainer

--

Rainer Duffner

CISSP, LPI, MCSE

rainer@ultra-secure.de





_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Am 21.06.2008 um 21:51 schrieb Mag Gam:


Network is a 10/100




You're kidding?




1 million large files
No SAN, JBOD




Move the data by moving the storage itself.
It will take months to transfer 100 TB via FastEthernet.



cheers,
Rainer
--
Rainer Duffner
CISSP, LPI, MCSE
rainer@ultra-secure.de


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Mag Gam wrote:

Network is a 10/100
1 million large files
No SAN, JBOD


assuming 100baseT wire speed of about 10Mbyte/sec, moving 100TB will
take a minimum of 100TB/10MB/s = 10,000,000 seconds or 2900 hours, or
about 4 months. even on a gigE network, this would still take about 2
weeks or more.






_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Can add fiber network card to each server? fiber switch? if not try to
plugin to each server giga ethernet card.... put a crossover cable and
start rsync... i did that with 1tb of photos and takes a lot of
time....keep power supply working and cross the fingers....

I hope this can help

2008/6/21 John R Pierce <pierce@hogranch.com>:
> Mag Gam wrote:
>>
>> Network is a 10/100
>> 1 million large files
>> No SAN, JBOD
>
> assuming 100baseT wire speed of about 10Mbyte/sec, moving 100TB will take a
> minimum of 100TB/10MB/s = 10,000,000 seconds or 2900 hours, or about 4
> months. even on a gigE network, this would still take about 2 weeks or
> more.
>
>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

On Sat, Jun 21, 2008 at 5:12 PM, nightduke <nightduke2005@gmail.com> wrote:

Can add fiber network card to each server? fiber switch? if not try to

plugin to each server giga ethernet card.... put a crossover cable and

start rsync... i did that with 1tb of photos and takes a lot of

time....keep power supply working and cross the fingers....



I hope this can help



2008/6/21 John R Pierce <pierce@hogranch.com>:

> Mag Gam wrote:

>>

>> Network is a 10/100

>> 1 million large files

>> No SAN, JBOD

>

> assuming 100baseT wire speed of about 10Mbyte/sec, moving 100TB will take a

> minimum of 100TB/10MB/s = 10,000,000 seconds or 2900 hours, or about 4

> months. * even on a gigE network, this would still take about 2 weeks or

> more.
Then if you get the network sorted out, the fastest & most reliable way I know to copy lots of files is

star --copy

You can get star with

yum install star


--Matt


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

recommendations for copying large filesystems
 

Am 21.06.2008 um 23:44 schrieb Matt Morgan:


O
Then if you get the network sorted out, the fastest & most reliable
way I know to copy lots of files is


star --copy

You can get star with

yum install star




Now that I know the details - I don' think this is going to work. Not
with 100 TB of data. It kind-of-works with 1 TB.

Can anybody comment on the feasibility of rsync on 1 million files?
Maybe DRBD would be a solution.
If you can retrofit DRDB to an existing setup...

If not it's faster to move the drives physically - believe me, this
will create far less problems.
In a SAN, you would have the possibility of synching the data outside
of the filesystem, during normal operations.



100 TB is a lot of data.
How do you back that up, BTW?
What is your estimated time to restore it from the medium you back
it up to?




cheers,
Rainer
--
Rainer Duffner
CISSP, LPI, MCSE
rainer@ultra-secure.de


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos