On 11/21/10 9:02 AM, Michael D. Berger wrote:
> On Sun, 21 Nov 2010 06:47:04 -0500, Nico Kadel-Garcia wrote:
>
>> On Sat, Nov 20, 2010 at 10:28 PM, Michael D. Berger
>> <m_d_berger_1900@yahoo.com> wrote:
> [...]
>>
>> > From decades of experience in many environments, I can tell you that
>> reliable transfer of large files with protocols that require
>> uninterrupted transfer is awkward. The larger the file, the larger the
>> chance that any interruption at any point between the repository and the
>> client will break things, and with a lot of ISP's over-subscribing their
>> available bandwidth, such large transfers are, by their nature,
>> unreliable.
>>
>> Consider fragmenting the large file: Bittorrent transfers do this
>> automatically: the old "shar" and "split" tools also work well, and
>> tools like "rsync" and the lftp "mirror" utility are very good at
>> mirroring directories of such split up contents quite efficiently.
>
> What, then, is the largest file size that you would consider
> appropriate?

There's no particular limit with rsync since if you use the -P option it will be
able to restart a failed transfer with just a little extra time to verify it
with a block-checksum transfer. With methods that don't restart, an appropriate
size would depend on the reliability and speed of the connections since it
relates to the odds of a connection problem during the time it takes to complete
the transfer.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

On Sun, Nov 21, 2010 at 11:51 AM, Les Mikesell <lesmikesell@gmail.com> wrote:
> On 11/21/10 9:02 AM, Michael D. Berger wrote:
>> On Sun, 21 Nov 2010 06:47:04 -0500, Nico Kadel-Garcia wrote:
>>
>>> On Sat, Nov 20, 2010 at 10:28 PM, Michael D. Berger
>>> <m_d_berger_1900@yahoo.com> *wrote:
>> [...]
>>>
>>> > From decades of experience in many environments, I can tell you that
>>> reliable transfer of large files with protocols that require
>>> uninterrupted transfer is awkward. The larger the file, the larger the
>>> chance that any interruption at any point between the repository and the
>>> client will break things, and with a lot of ISP's over-subscribing their
>>> available bandwidth, such large transfers are, by their nature,
>>> unreliable.
>>>
>>> Consider fragmenting the large file: Bittorrent transfers do this
>>> automatically: the old "shar" and "split" tools also work well, and
>>> tools like "rsync" and the lftp "mirror" utility are very good at
>>> mirroring directories of such split up contents quite efficiently.
>>
>> What, then, is the largest file size that you would consider
>> appropriate?
>
> There's no particular limit with rsync since if you use the -P option it will be
> able to restart a failed transfer with just a little extra time to verify it
> with a block-checksum transfer. *With methods that don't restart, an appropriate
> size would depend on the reliability and speed of the connections since it
> relates to the odds of a connection problem during the time it takes to complete
> the transfer.

Rsync is wonderful, but not supported by a lot typical web browsers
and a lot of file managers that can speak FTP and HTTP. I like rsync
because it comprehends symlinks, hardlinks, has good scripting, and
allows sophistated exclude options without getting overwhelmed.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

On Sun, 21 Nov 2010 11:49:29 -0500, Nico Kadel-Garcia wrote:

[...]
>
> Good question. I don't have a hard rule of thumb, but I'd estimate that
> any one file that takes more than 10 minutes to transfer is too big. So
> transferring CD images over a high bandwidth local connection at 1
> MByte/second, sure, no problem! But for DSL that may have only 80
> KB/second, 80 KB/second * 60 seconds/minute * 10 minutes = 48 Meg. So
> splitting a CD down to lumps of of, say, 50 Megs seems reasonable.
>
[...]

The file I was having trouble with was a tar file of a complex
directory tree containing mostly jpg files under 15M in size.
So instead I did rsync -rv on the unpacked directory tree, and it
worked just fine. PROBLEM SOLVED.

Thanks,
Mike.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

On Sun, Nov 21, 2010 at 10:13 PM, Michael D. Berger
<m_d_berger_1900@yahoo.com> wrote:
> On Sun, 21 Nov 2010 11:49:29 -0500, Nico Kadel-Garcia wrote:
>
> [...]
>>
>> Good question. I don't have a hard rule of thumb, but I'd estimate that
>> any one file that takes more than 10 minutes to transfer is too big. So
>> transferring CD images over a high bandwidth local connection at 1
>> MByte/second, sure, no problem! But for DSL that may have only 80
>> KB/second, 80 KB/second * 60 seconds/minute * 10 minutes = 48 Meg. So
>> splitting a CD down to lumps of of, say, 50 Megs seems reasonable.
>>
> [...]
>
> The file I was having trouble with was a tar file of a complex
> directory tree containing mostly jpg files under 15M in size.
> So instead I did rsync -rv on the unpacked directory tree, and it
> worked just fine. *PROBLEM SOLVED.

Good for you. Next time, use "rsync -avH". "-H" preserves hardlinks,
"-a" preserves lots of other useful characteristics, such as symlinks
and full ownership and permissions.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos