FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 08-08-2012, 07:14 AM
lina
 
Default which one is faster?

Hi,

It's a bit big data to transfer, around 1.1 T,

from one server to another server.

I checked that rsync is faster than scp,
but in my situations rsync has elapsed for 1 hour, I guess the network
is also a problem,

Here I wish to know are there some tools (better default) can use for
fast transferring, regardless the security reason, my data is just
some data, no need special security care.

Thanks with best regards,


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/CAG9cJmnfi=Jo7aPQZ4hk-QEs=wN_KCdkoJkNWkv=Xisc3TGeHg@mail.gmail.com
 
Old 08-08-2012, 07:33 AM
Raffaele Morelli
 
Default which one is faster?

2012/8/8 lina <lina.lastname@gmail.com>

Hi,



It's a bit big data to transfer, around 1.1 T,



from one server to another server.



I checked that rsync is faster than scp,

but in my situations rsync has elapsed for 1 hour, I guess the network

is also a problem

split (man split) your data into little chuncks then and/or use ftp

-r


--

L'unica speranza di catarsi, ammesso che ne esista una,
resta affidata all'istinto di ribellione, alla rivolta non isterilita
in progetti, alla protesta violenta e viscerale. (V. Evangelisti)
 
Old 08-08-2012, 07:35 AM
Jochen Spieker
 
Default which one is faster?

lina:
>
> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.

Either use rsync without encryption (= not tunneled over SSH), or pipe
tar through netcat. The latter does not support resuming.

If you can tell us a bit more, we might be able to help better. Do you
have many small files or just a fewer big files? How fast and reliable
is the connection between the two computers?

J.
--
When driving at night I find the headlights of oncoming vehicles very
attractive.
[Agree] [Disagree]
<http://www.slowlydownward.com/NODATA/data_enter2.html>
 
Old 08-08-2012, 07:50 AM
Kelly Clowers
 
Default which one is faster?

On Wed, Aug 8, 2012 at 12:14 AM, lina <lina.lastname@gmail.com> wrote:
> Hi,
>
> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.
>
> I checked that rsync is faster than scp,

Well, sometimes. Certainly if you have some of the data in both
places and need to sync it, rsync will just do the delta.

But if the destination does not have any of the data yet, *at best*
it will be similar.

And rsync, depending on version and settings, will do things
like make a full file list and/or checksums up front. That can
be good, but it can also take along time on large data.

You should carefully check out what options you might want
or *not* want: --append, --inplace, --size-only, -c, -z, etc

And since rsync typically goes over ssh, the encryption adds
to the overhead as well.

Or just use scp or ftp or something in those cases.

Cheers,
Kelly Clowers


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/CAFoWM=8NVwdBX1RZfABFNT3POoa0vtP79wi+tYd-i2EWnEVEZg@mail.gmail.com
 
Old 08-08-2012, 07:58 AM
Johann Spies
 
Default which one is faster?

Hallo Lina,


> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.

You can also use netcat (man nc - see under examples). It is probably
the fastest method.

I have used that in the past in combination with rsync: copy it using
netcat and check the result with rsync.

Regards
Johann

--
Johann Spies Telefoon: 021-808 4699
Databestuurder / Data manager

Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie
Centre for Research on Evaluation, Science and Technology
Universiteit Stellenbosch.

"Ye lust, and have not: ye kill, and desire to have,
and cannot obtain: ye fight and war, yet ye have not,
because ye ask not." James 4:2
E-pos vrywaringsklousule

Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie.

E-mail disclaimer

This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120808075852.GA21238@sun.ac.za">http://lists.debian.org/20120808075852.GA21238@sun.ac.za
 
Old 08-08-2012, 08:19 AM
Tóth Tibor Péter
 
Default which one is faster?

I user rsync for everything.

rsync -vaP /location/ user@remote-host:/location/

I've found it best sofar.

-----Original Message-----
From: Johann Spies [mailto:jspies@sun.ac.za]
Sent: Wednesday, August 08, 2012 9:59 AM
To: debian-user@lists.debian.org
Subject: Re: which one is faster?

Hallo Lina,


> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.

You can also use netcat (man nc - see under examples). It is probably the fastest method.

I have used that in the past in combination with rsync: copy it using netcat and check the result with rsync.

Regards
Johann

--
Johann Spies Telefoon: 021-808 4699
Databestuurder / Data manager

Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie Centre for Research on Evaluation, Science and Technology Universiteit Stellenbosch.

"Ye lust, and have not: ye kill, and desire to have,
and cannot obtain: ye fight and war, yet ye have not,
because ye ask not." James 4:2
E-pos vrywaringsklousule

Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige l??s aangeheg by hierdie e-pos nie.

E-mail disclaimer

This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120808075852.GA21238@sun.ac.za



Ezen üzenet és annak bármely csatolt anyaga bizalmas, jogi védelem alatt áll, a nyilvános közléstől védett. Az üzenetet kizárólag a c*mzett, illetve az általa meghatalmazottak használhatják fel. Ha Ön nem az üzenet c*mzettje, úgy kérjük, hogy telefonon, vagy e-mail-ben értes*tse erről az üzenet küldőjét és törölje az üzenetet, valamint annak összes csatolt mellékletét a rendszeréből. Ha Ön nem az üzenet c*mzettje, abban az esetben tilos az üzenetet vagy annak bármely csatolt mellékletét lemásolnia, elmentenie, az üzenet tartalmát bárkivel közölnie vagy azzal visszaélnie.

This message and any attachment are confidential and are legally privileged. It is intended solely for the use of the individual or entity to whom it is addressed and others authorised to receive it. If you are not the intended recipient, please phone or email the sender and delete this message and any attachment of it from your system. Please note that any dissemination, distribution, copying or use of or reliance upon the information contained in and transmitted by this e-mail or to anyone other than the recipient designated above by the sender is unauthorised and strictly prohibited.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 7AF9F7955A605547AB5BCD2B303F4B7D01024248DDA9@BUDAK ZPMBX02.mak.allamkincstar.gov.hu">http://lists.debian.org/7AF9F7955A605547AB5BCD2B303F4B7D01024248DDA9@BUDAK ZPMBX02.mak.allamkincstar.gov.hu
 
Old 08-08-2012, 08:43 AM
Johannes Wiedersich
 
Default which one is faster?

On 08/08/12 09:14, lina wrote:
> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.
>
> I checked that rsync is faster than scp,
> but in my situations rsync has elapsed for 1 hour, I guess the network
> is also a problem,
>
> Here I wish to know are there some tools (better default) can use for
> fast transferring, regardless the security reason, my data is just
> some data, no need special security care.

IIUC, the question is not just, which is the fastest tool. If you have
network problems (ie. intermittent connections) or fear thereof, you
need a fast *and* a reliable tool.

I suggest you stick with rsync. IMHO it is the best tool for your task.

With the -c option, eg. you could check, whether all files transferred
correctly, without much demand on the network.

Cheers,

Johannes


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 502226A7.9030109@aktendiener.de">http://lists.debian.org/502226A7.9030109@aktendiener.de
 
Old 08-08-2012, 12:10 PM
Darac Marjal
 
Default which one is faster?

On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote:
> Hi,
>
> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.
>
> I checked that rsync is faster than scp,
> but in my situations rsync has elapsed for 1 hour, I guess the network
> is also a problem,
>
> Here I wish to know are there some tools (better default) can use for
> fast transferring, regardless the security reason, my data is just
> some data, no need special security care.
>

In addition to the suggestions mentioned by other people, consider the
compressability of your data. I don't believe it's possible to
definitely predict the trade-offs here (it depends on how well the data
compresses, basically, but also on how your CPUs compare to the network
bandwidth, but you may find that spending some time compressing the data
reduces the overall time.

In that case try, the following. On the receiver:

$ nc -l -p 12345 | $COMP -d | pv > outfile

and on the sender:

$ pv infile | $COMP | nc receiver 12345

where $COMP is your preferred streaming compressor (gzip, bzip2 and xz
should all work nicely here). By the way, "pv" (package: pv) is a useful
pipeline-viewer and will show you progress (on the sending side) as well
as throughput levels.
 
Old 08-08-2012, 01:00 PM
lina
 
Default which one is faster?

On Wed, Aug 8, 2012 at 3:35 PM, Jochen Spieker <ml@well-adjusted.de> wrote:
> lina:
>>
>> It's a bit big data to transfer, around 1.1 T,
>>
>> from one server to another server.
>
> Either use rsync without encryption (= not tunneled over SSH), or pipe
> tar through netcat. The latter does not support resuming.

Thanks. I failed to find the mute the encryption options in rsync.

BTW, How to set the port for netcat?

The remote one has the following ports open:

Not shown: 987 closed ports
PORT STATE SERVICE
21/tcp open ftp
22/tcp open ssh
80/tcp open http
111/tcp open rpcbind
711/tcp open cisco-tdp
1027/tcp open IIS
1029/tcp open ms-lsa
3001/tcp open nessus
3389/tcp open ms-wbt-server
8649/tcp open unknown
8651/tcp open unknown
8652/tcp open unknown
15004/tcp open unknown

$ nc badapple.net 21
220 (vsFTPd 2.0.5)

it chocked there.

another one has:
Not shown: 995 filtered ports
PORT STATE SERVICE
80/tcp open http
443/tcp open https
3128/tcp open squid-http
8080/tcp open http-proxy
8888/tcp open sun-answerbook

>
> If you can tell us a bit more, we might be able to help better. Do you
> have many small files or just a fewer big files? How fast and reliable
> is the connection between the two computers?
It's 8000 files, half 400M, half 5M.

I don't know the reliable of the connection between the two servers, I
guess it's okay.

But from my side, the wireless is not stable.
I don't know how to let it stable. I mean, not login every 10~15 minutes.
(btw, Is big wind affects the wireless signal, kinda of silly to ask,
I prefer the fresh air outside, so move laptop outside)

>
> J.
> --
> When driving at night I find the headlights of oncoming vehicles very
> attractive.
> [Agree] [Disagree]
> <http://www.slowlydownward.com/NODATA/data_enter2.html>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAG9cJmnaJXHb9McpbdUMs5OftYNHyOwiqXPHUJhkHQ1pTmO5g g@mail.gmail.com">http://lists.debian.org/CAG9cJmnaJXHb9McpbdUMs5OftYNHyOwiqXPHUJhkHQ1pTmO5g g@mail.gmail.com
 
Old 08-08-2012, 01:24 PM
lina
 
Default which one is faster?

On Wed, Aug 8, 2012 at 3:58 PM, Johann Spies <jspies@sun.ac.za> wrote:
> Hallo Lina,
>
>
>> It's a bit big data to transfer, around 1.1 T,
>>
>> from one server to another server.
>
> You can also use netcat (man nc - see under examples). It is probably
> the fastest method.
>
> I have used that in the past in combination with rsync: copy it using
> netcat and check the result with rsync.

Thanks, this is smart.
but I still have not figured it out how to use nc.
I will try,

Best regards,
>
> Regards
> Johann
>
> --
> Johann Spies Telefoon: 021-808 4699
> Databestuurder / Data manager
>
> Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie
> Centre for Research on Evaluation, Science and Technology
> Universiteit Stellenbosch.
>
> "Ye lust, and have not: ye kill, and desire to have,
> and cannot obtain: ye fight and war, yet ye have not,
> because ye ask not." James 4:2
> E-pos vrywaringsklousule
>
> Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige ls aangeheg by hierdie e-pos nie.
>
> E-mail disclaimer
>
> This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/20120808075852.GA21238@sun.ac.za
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAG9cJmkOe5GTF98WPJKg60gaAsA5WFBLJ4k9XUKb2qPw_WeN2 g@mail.gmail.com">http://lists.debian.org/CAG9cJmkOe5GTF98WPJKg60gaAsA5WFBLJ4k9XUKb2qPw_WeN2 g@mail.gmail.com
 

Thread Tools




All times are GMT. The time now is 11:17 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org