FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 08-08-2012, 01:33 PM
lina
 
Default which one is faster?

On Wed, Aug 8, 2012 at 4:43 PM, Johannes Wiedersich
<debian@aktendiener.de> wrote:
> On 08/08/12 09:14, lina wrote:
>> It's a bit big data to transfer, around 1.1 T,
>>
>> from one server to another server.
>>
>> I checked that rsync is faster than scp,
>> but in my situations rsync has elapsed for 1 hour, I guess the network
>> is also a problem,
>>
>> Here I wish to know are there some tools (better default) can use for
>> fast transferring, regardless the security reason, my data is just
>> some data, no need special security care.
>
> IIUC, the question is not just, which is the fastest tool. If you have
> network problems (ie. intermittent connections) or fear thereof, you
> need a fast *and* a reliable tool.
>
> I suggest you stick with rsync. IMHO it is the best tool for your task.
>
> With the -c option, eg. you could check, whether all files transferred
> correctly, without much demand on the network.

Thanks, at present an email had been sent to the administrator to
hopefully get 2TB space for data handling.
/dev/gpfs1 117T 43T 74T 37% /scratch
/dev/gpfs3 30T 74G 30T 1% /userbackup

Seems lots of free space, hope won't be refused.

But here I still wish to hear the suggestions, very nice, at least I
started to know the nc now,

Best regards,
>
> Cheers,
>
> Johannes
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/502226A7.9030109@aktendiener.de
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/CAG9cJmk63_YLsjXZWyZMnrQ=V8VOV4OaHn6=ZnA9rUzfGafaG g@mail.gmail.com
 
Old 08-08-2012, 01:47 PM
lina
 
Default which one is faster?

On Wed, Aug 8, 2012 at 8:10 PM, Darac Marjal <mailinglist@darac.org.uk> wrote:
> On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote:
>> Hi,
>>
>> It's a bit big data to transfer, around 1.1 T,
>>
>> from one server to another server.
>>
>> I checked that rsync is faster than scp,
>> but in my situations rsync has elapsed for 1 hour, I guess the network
>> is also a problem,
>>
>> Here I wish to know are there some tools (better default) can use for
>> fast transferring, regardless the security reason, my data is just
>> some data, no need special security care.
>>
>
> In addition to the suggestions mentioned by other people, consider the
> compressability of your data. I don't believe it's possible to
> definitely predict the trade-offs here (it depends on how well the data
> compresses, basically, but also on how your CPUs compare to the network
> bandwidth, but you may find that spending some time compressing the data
> reduces the overall time.
>
> In that case try, the following. On the receiver:
>
> $ nc -l -p 12345 | $COMP -d | pv > outfile
>
> and on the sender:
>
> $ pv infile | $COMP | nc receiver 12345
>
> where $COMP is your preferred streaming compressor (gzip, bzip2 and xz
> should all work nicely here). By the way, "pv" (package: pv) is a useful
> pipeline-viewer and will show you progress (on the sending side) as well
> as throughput levels.

Thank you, neither receiver nor sender has the pv installed.

I will spend sometime to figure the nc out first.

Best regards,
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQIcBAEBCgAGBQJQIlc9AAoJEKB7YbRsd8TGRkkP/iQLhUKprzjOhVAPv8A4rRoI
> QYktoGPfRedswG4ivs0EaUVySUuK4sjmFuSj84vLh3WRLmsQUt DpiFbOUnlJS1Uj
> BrtKGEe73uMZQHrO6l/HtnOvUj/X5hxKPLI5ZFId3AvDIiiDOtZxVbLgjVEfrFnV
> LeV5YKVwks0O9uGxTOwUItXjSF2LUG2W96gKDqVAOvRLSKweG6 mKtWALjDs4QYTw
> PPfpuvXxdEr0CscqUehph2iY/27MwPW6yNbC91o0uxRAj6/fcqnnJr3bR9+Vyxxk
> HtP6G+niroSoF8EFV4ETtZQXubIZuMkp7FhaxRjynLkVFbskJ4 pRIXgL789qOX38
> LJxHhPDNpn3F009rYaawz7t1ySv+sobicbI38hEQynCv9hZ9Bi X0nIriTB2DXJ/A
> 8RLkpKGSlUXbbCA8UJ35wBwVBdaVEhLGACOXsE3zLbjcjsEDRt iHJIwdeHfpJbK4
> xV1seAxkgKljPvTf+k+GEuRyh60CIduYx9yLG5icUV/HFVz/21azpqELBH2kWdpq
> WKREKD5ymmzofFLQtS8QI8I5UH6ZerHfptj/r5E5ff4BcFNmQ7ys7q52vkV3HPeI
> QBJ3GMuey/iNruQJXs7xU6P+BdXq+TyG9XK0LF4/4nk6mPSMot/Vqc5ShWsmeYkW
> tHddx04MGvI0OHc04vOo
> =a8qg
> -----END PGP SIGNATURE-----
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/CAG9cJm=Cpg0uxfU38AtmHKT7Ku-5RfKVfwBgvB4WSj0h45z01w@mail.gmail.com
 
Old 08-08-2012, 02:22 PM
Johann Spies
 
Default which one is faster?

On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote:
>
> BTW, How to set the port for netcat?

> The remote one has the following ports open:

Choose one not from that list above 1024 and make sure any firewalls
between the two computers allow that.

See the following examples:
http://www.screenage.de/blog/2007/12/30/using-netcat-and-tar-for-network-file-transfer/
http://g33kinfo.com/info/archives/1713
http://www.g-loaded.eu/2006/11/06/netcat-a-couple-of-useful-examples/

Regards
Johann
--
Johann Spies Telefoon: 021-808 4699
Databestuurder / Data manager

Sentrum vir Navorsing oor Evaluasie, Wetenskap en Tegnologie
Centre for Research on Evaluation, Science and Technology
Universiteit Stellenbosch.

"Ye lust, and have not: ye kill, and desire to have,
and cannot obtain: ye fight and war, yet ye have not,
because ye ask not." James 4:2
E-pos vrywaringsklousule

Hierdie e-pos mag vertroulike inligting bevat en mag regtens geprivilegeerd wees en is slegs bedoel vir die persoon aan wie dit geadresseer is. Indien u nie die bedoelde ontvanger is nie, word u hiermee in kennis gestel dat u hierdie dokument geensins mag gebruik, versprei of kopieer nie. Stel ook asseblief die sender onmiddellik per telefoon in kennis en vee die e-pos uit. Die Universiteit aanvaar nie aanspreeklikheid vir enige skade, verlies of uitgawe wat voortspruit uit hierdie e-pos en/of die oopmaak van enige lês aangeheg by hierdie e-pos nie.

E-mail disclaimer

This e-mail may contain confidential information and may be legally privileged and is intended only for the person to whom it is addressed. If you are not the intended recipient, you are notified that you may not use, distribute or copy this document in any manner whatsoever. Kindly also notify the sender immediately by telephone, and delete the e-mail. The University does not accept liability for any damage, loss or expense arising from this e-mail and/or accessing any files attached to this e-mail.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120808142242.GC11234@sun.ac.za">http://lists.debian.org/20120808142242.GC11234@sun.ac.za
 
Old 08-08-2012, 02:35 PM
Camaleón
 
Default which one is faster?

On Wed, 08 Aug 2012 15:14:50 +0800, lina wrote:

> It's a bit big data to transfer, around 1.1 T,
>
> from one server to another server.

Are both hosts remote (over Internet) or local (LAN)?

> I checked that rsync is faster than scp, but in my situations rsync has
> elapsed for 1 hour, I guess the network is also a problem,

That can be because rsync performs better for small chunks of data
instead bigger ones.

> Here I wish to know are there some tools (better default) can use for
> fast transferring, regardless the security reason, my data is just some
> data, no need special security care.

I would run a bunch of tests to transfer one file (~1 GiB) with different
tools (scp, ftp, http, rsync...) and choose whichever gives the fastest
results :-)

Note: adding a secure channel (ftps/sftp/scp/https...) will delay the
transfer.

Greetings,

--
Camaleón


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/jvttfc$kgo$8@dough.gmane.org
 
Old 08-09-2012, 03:05 PM
Chris Bannister
 
Default which one is faster?

On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote:
> I don't know the reliable of the connection between the two servers, I
> guess it's okay.
>
> But from my side, the wireless is not stable.
> I don't know how to let it stable. I mean, not login every 10~15 minutes.
> (btw, Is big wind affects the wireless signal, kinda of silly to ask,
> I prefer the fresh air outside, so move laptop outside)

Wind won't affect the signal, but if you are outside you may be in a
spot where the signal is weak. Can you see what the signal strength is?

--
"If you're not careful, the newspapers will have you hating the people
who are being oppressed, and loving the people who are doing the
oppressing." --- Malcolm X


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120809150547.GK2687@tal
 
Old 08-09-2012, 04:00 PM
lina
 
Default which one is faster?

On 9 Aug, 2012, at 23:05, Chris Bannister <cbannister@slingshot.co.nz> wrote:

> On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote:
>> I don't know the reliable of the connection between the two servers, I
>> guess it's okay.
>>
>> But from my side, the wireless is not stable.
>> I don't know how to let it stable. I mean, not login every 10~15 minutes.
>> (btw, Is big wind affects the wireless signal, kinda of silly to ask,
>> I prefer the fresh air outside, so move laptop outside)
>
> Wind won't affect the signal, but if you are outside you may be in a
> spot where the signal is weak. Can you see what the signal strength is?
Wind won't affect the signal?!
Thanks.

I don't know how to check the signal strength except seeing the icon of the network manager.

>
> --
> "If you're not careful, the newspapers will have you hating the people
> who are being oppressed, and loving the people who are doing the
> oppressing." --- Malcolm X
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/20120809150547.GK2687@tal
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: D9EAA49A-3915-4DEA-8565-8EF3850FD2C8@gmail.com">http://lists.debian.org/D9EAA49A-3915-4DEA-8565-8EF3850FD2C8@gmail.com
 
Old 08-09-2012, 05:51 PM
Chris Bannister
 
Default which one is faster?

On Fri, Aug 10, 2012 at 12:00:12AM +0800, lina wrote:
>
> I don't know how to check the signal strength except seeing the icon of the network manager.

Sorry, I don't use "network manager", but you could check by going
inside to test. The reason the transfer speeds could be slow, is because
of a flaky connection where it has to do retransmits every so often.

Compare speeds of large file transfer, if same inside as outside, then
at least you can sit outside and try the other suggestions from this
thread.

--
"If you're not careful, the newspapers will have you hating the people
who are being oppressed, and loving the people who are doing the
oppressing." --- Malcolm X


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20120809175158.GA5072@tal
 
Old 08-10-2012, 04:52 AM
Kelly Clowers
 
Default which one is faster?

On Thu, Aug 9, 2012 at 9:00 AM, lina <lina.lastname@gmail.com> wrote:
>
> On 9 Aug, 2012, at 23:05, Chris Bannister <cbannister@slingshot.co.nz> wrote:
>
>> On Wed, Aug 08, 2012 at 09:00:18PM +0800, lina wrote:
>>> I don't know the reliable of the connection between the two servers, I
>>> guess it's okay.
>>>
>>> But from my side, the wireless is not stable.
>>> I don't know how to let it stable. I mean, not login every 10~15 minutes.
>>> (btw, Is big wind affects the wireless signal, kinda of silly to ask,
>>> I prefer the fresh air outside, so move laptop outside)
>>
>> Wind won't affect the signal, but if you are outside you may be in a
>> spot where the signal is weak. Can you see what the signal strength is?
>
> Wind won't affect the signal?!
>Thanks.

Nope. The air itself absorbs some of the electromagnetic spectrum
(which is why gamma ray and x ray telescopes are *all* in space).
Movement of the air does not really affect it much though. Density
changes in the air (from pressure or thermal differences) can affect
EM radiation, which is one reason the big optical telescopes have
computer controlled micro-adjustments or are in space (Hubble).

However, that is visible light, which has a much shorter wavelength
than radio. Being longer, radio is much less affected by density
differences, and on the scale of wifi it is not worth thinking about.
And wind is basically a non-effect even for light (unless it carries
dust or snow or something, but that is a different matter). I will
say that in a heavy Montana blizzard, satellite TV signal can fade.
But that comes down to the amount of water (frozen) in the air,
as water tends to be a pretty good absorber of EM radiation in
general.

Cheers,
Kelly Clowers


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/CAFoWM=94wkDLe9NEbO9Zke4N2T5y5ujJ17PFn460vmpvg9uWa g@mail.gmail.com
 
Old 08-10-2012, 09:03 AM
Martin Steigerwald
 
Default which one is faster?

Am Mittwoch, 8. August 2012 schrieb Johannes Wiedersich:
> On 08/08/12 09:14, lina wrote:
> > It's a bit big data to transfer, around 1.1 T,
> >
> > from one server to another server.
> >
> > I checked that rsync is faster than scp,
> > but in my situations rsync has elapsed for 1 hour, I guess the network
> > is also a problem,
> >
> > Here I wish to know are there some tools (better default) can use for
> > fast transferring, regardless the security reason, my data is just
> > some data, no need special security care.
>
> IIUC, the question is not just, which is the fastest tool. If you have
> network problems (ie. intermittent connections) or fear thereof, you
> need a fast *and* a reliable tool.

Yes.

It likely would also be a good idea to fix these network problems

> I suggest you stick with rsync. IMHO it is the best tool for your task.
>
> With the -c option, eg. you could check, whether all files transferred
> correctly, without much demand on the network.

I only use rsync for these kinds of stuff. Its just reliable.

With BTRFS I will investigate btrfs send/receive, but in the first time
I will – as recommended – make sure I run rsync -c after it.

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201208101103.19086.Martin@lichtvoll.de">http://lists.debian.org/201208101103.19086.Martin@lichtvoll.de
 
Old 08-10-2012, 09:05 AM
Martin Steigerwald
 
Default which one is faster?

Am Mittwoch, 8. August 2012 schrieb Darac Marjal:
> On Wed, Aug 08, 2012 at 03:14:50PM +0800, lina wrote:
> > Hi,
> >
> > It's a bit big data to transfer, around 1.1 T,
> >
> > from one server to another server.
> >
> > I checked that rsync is faster than scp,
> > but in my situations rsync has elapsed for 1 hour, I guess the network
> > is also a problem,
> >
> > Here I wish to know are there some tools (better default) can use for
> > fast transferring, regardless the security reason, my data is just
> > some data, no need special security care.
> >
>
> In addition to the suggestions mentioned by other people, consider the
> compressability of your data. I don't believe it's possible to

rsync -z and possibly --compress-level might some in handy as well. It
should be more efficient than using compression on SSH level.

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201208101105.26784.Martin@lichtvoll.de">http://lists.debian.org/201208101105.26784.Martin@lichtvoll.de
 

Thread Tools




All times are GMT. The time now is 04:46 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org