Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   CentOS (http://www.linux-archive.org/centos/)
-   -   BackupPC and rsync (http://www.linux-archive.org/centos/707411-backuppc-rsync.html)

Emmett Culley 09-26-2012 03:33 AM

BackupPC and rsync
 
I have been using BackupPC via rsync daemons on the target machines and all has been working well. At least until about the time rsync went to version 3.0.6.

Since then backups have been failing for some targets. The failure is a TCP ZeroWindow issue. It looks like the server rsync process cannot empty it's buffer soon enough and eventually the process fails.

I really do not know if it is rsync version related, because I never looked at the packet stream until I started having failures.

BackupPC was running on a VM host and the failing machine was one of the guest VMs. I tried upgrading rsync to 3.0.9 and that didn't help.

So I moved the BackupPC set up to a different server (no VM). Not all backups are working. though I still see some ZeroWindow packets along with a few retransmision packets. But the backup completes on all machines, including the VM that was failing before.

I've googled rsync zerowindow and found some references, but none seemed to apply to my situation.

Anybody else having a similar experience?

Emmett
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Les Mikesell 09-26-2012 06:05 PM

BackupPC and rsync
 
On Tue, Sep 25, 2012 at 10:33 PM, Emmett Culley <emmett@webengineer.com> wrote:
> I have been using BackupPC via rsync daemons on the target machines and all has been working well. At least until about the time rsync went to version 3.0.6.
>
> Since then backups have been failing for some targets. The failure is a TCP ZeroWindow issue. It looks like the server rsync process cannot empty it's buffer soon enough and eventually the process fails.
>
> I really do not know if it is rsync version related, because I never looked at the packet stream until I started having failures.
>
> BackupPC was running on a VM host and the failing machine was one of the guest VMs. I tried upgrading rsync to 3.0.9 and that didn't help.
>
> So I moved the BackupPC set up to a different server (no VM). Not all backups are working. though I still see some ZeroWindow packets along with a few retransmision packets. But the backup completes on all machines, including the VM that was failing before.
>
> I've googled rsync zerowindow and found some references, but none seemed to apply to my situation.
>
> Anybody else having a similar experience?

I found an old mention of turning off TCP window scaling solving a
problem with rsync (not involving backuppc). Seems unlikely but
might be worth a try: sysctl -w net.ipv4.tcp_window_scaling=0.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Emmett Culley 09-27-2012 01:35 AM

BackupPC and rsync
 
On 09/26/2012 11:05 AM, Les Mikesell wrote:
> On Tue, Sep 25, 2012 at 10:33 PM, Emmett Culley <emmett@webengineer.com> wrote:
>> I have been using BackupPC via rsync daemons on the target machines and all has been working well. At least until about the time rsync went to version 3.0.6.
>>
>> Since then backups have been failing for some targets. The failure is a TCP ZeroWindow issue. It looks like the server rsync process cannot empty it's buffer soon enough and eventually the process fails.
>>
>> I really do not know if it is rsync version related, because I never looked at the packet stream until I started having failures.
>>
>> BackupPC was running on a VM host and the failing machine was one of the guest VMs. I tried upgrading rsync to 3.0.9 and that didn't help.
>>
>> So I moved the BackupPC set up to a different server (no VM). Not all backups are working. though I still see some ZeroWindow packets along with a few retransmision packets. But the backup completes on all machines, including the VM that was failing before.
>>
>> I've googled rsync zerowindow and found some references, but none seemed to apply to my situation.
>>
>> Anybody else having a similar experience?
>
> I found an old mention of turning off TCP window scaling solving a
> problem with rsync (not involving backuppc). Seems unlikely but
> might be worth a try: sysctl -w net.ipv4.tcp_window_scaling=0.
>

I tried that on the target VM and it didn't make any difference. Then I tried it on the BackupPC server and that didn't make any difference either.

Thanks for the suggestion anyway.

Emmett

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Gordon Messmer 09-27-2012 02:25 AM

BackupPC and rsync
 
On 09/25/2012 08:33 PM, Emmett Culley wrote:
> Since then backups have been failing for some targets. The failure
> is a TCP ZeroWindow issue. It looks like the server rsync process
> cannot empty it's buffer soon enough and eventually the process
> fails.

If the rsync server sends a TCP ZeroWindow packet to a client, that
probably indicates that the server is either hung or looped. Either
way, it has stopped reading data from the socket, and the kernel sends
ZeroWindow to the client's host to indicate that the client should stop
sending data.

You may have found a bug in rsync that the developers need to resolve.
Have you tried attaching strace to the rsync server process when it hangs?
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


All times are GMT. The time now is 05:16 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.