Performance problems with squeeze kernel
 

Hello there,

I think I found a performance issue in the kernel (2.6.32/amd64) of the
squeeze release.

I tested from different servers and each time made a wget of a 16 MB
file from a remote co-location which is connected with gigabit (but the
issue showed also on a 32MBit/s "dial-up" cable connection).

The squeeze system starts the transfer significantly slower and takes
longer to get faster over time (on a 2GB file the maximum was reached
after 12 seconds). The lenny system reaches this maxium very fast so the
overall time drops from 2.4s to 0.4s. It could be, this is related to
changes in TCP/IP behaviour. I could not see this behaviour in the same
subnet, it shows up, as soon as routing is involved in the connection.

I made two graphs with tcpdump and wireshark from the remote host during
the wget from the servers which stand side by side in the rack on the
same switch. You can see them under

http://www.youimage.com/a/TPaszvOgtrM$/

On a lenny system with 2.6.26 everything works fine. All our the new
servers with squeeze (and the 2.6.32) show the issue. If I substitute
the standard kernel from squeeze with the current backports 2.6.39-bpo.2
kernel everthing is working again. So I strongly believe the issue is
related to the kernel.

I diffed a "sysctl -a" from the lenny system and the not properly
working squeeze systems with nearly no differences at all, but set all
the values as they were on the lenny system - just to be sure with no luck.

If I can help any further please let me know, I will try as good as I
can. If someone can confirm this, I am of course able to file a bugreport.

regards
Thorsten


--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E5E0D62.7090305@anw.de">http://lists.debian.org/4E5E0D62.7090305@anw.de

Performance problems with squeeze kernel
 

On Wed, 2011-08-31 at 12:30 +0200, Thorsten Giese wrote:
> Hello there,
>
> I think I found a performance issue in the kernel (2.6.32/amd64) of the
> squeeze release.
>
> I tested from different servers and each time made a wget of a 16 MB
> file from a remote co-location which is connected with gigabit (but the
> issue showed also on a 32MBit/s "dial-up" cable connection).
>
> The squeeze system starts the transfer significantly slower and takes
> longer to get faster over time (on a 2GB file the maximum was reached
> after 12 seconds). The lenny system reaches this maxium very fast so the
> overall time drops from 2.4s to 0.4s. It could be, this is related to
> changes in TCP/IP behaviour. I could not see this behaviour in the same
> subnet, it shows up, as soon as routing is involved in the connection.
[...]

Which network driver are you using?

Is there any packet loss? Run 'netstat -s | grep retransmited' to check
this.

Ben.

Performance problems with squeeze kernel
 

Am 31.08.2011 14:09, schrieb Ben Hutchings:
>> I think I found a performance issue in the kernel (2.6.32/amd64) of the
>> squeeze release.
>>
>> I tested from different servers and each time made a wget of a 16 MB
>> file from a remote co-location which is connected with gigabit (but the
>> issue showed also on a 32MBit/s "dial-up" cable connection).
>>
>> The squeeze system starts the transfer significantly slower and takes
>> longer to get faster over time (on a 2GB file the maximum was reached
>> after 12 seconds). The lenny system reaches this maxium very fast so the
>> overall time drops from 2.4s to 0.4s. It could be, this is related to
>> changes in TCP/IP behaviour. I could not see this behaviour in the same
>> subnet, it shows up, as soon as routing is involved in the connection.
> [...]
>
> Which network driver are you using?
>
> Is there any packet loss? Run 'netstat -s | grep retransmited' to
> check this.


I checked and I am using the e1000e on all affected systems. I checked
on a squeeze system with the r8169 driver and the problem seems not to
appear there.

Your netstat command says: "4096 segments retransmited" before the wget
and the same after it.

I checked with mtr and ping multiple times that there was no packet
loss, between and during the requests as part of my research in the past.

Thorsten


--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E5E2AC7.2060306@anw.de">http://lists.debian.org/4E5E2AC7.2060306@anw.de

Performance problems with squeeze kernel
 

On Wed, 2011-08-31 at 14:36 +0200, Thorsten Giese wrote:
> Am 31.08.2011 14:09, schrieb Ben Hutchings:
> >> I think I found a performance issue in the kernel (2.6.32/amd64) of the
> >> squeeze release.
> >>
> >> I tested from different servers and each time made a wget of a 16 MB
> >> file from a remote co-location which is connected with gigabit (but the
> >> issue showed also on a 32MBit/s "dial-up" cable connection).
> >>
> >> The squeeze system starts the transfer significantly slower and takes
> >> longer to get faster over time (on a 2GB file the maximum was reached
> >> after 12 seconds). The lenny system reaches this maxium very fast so the
> >> overall time drops from 2.4s to 0.4s. It could be, this is related to
> >> changes in TCP/IP behaviour. I could not see this behaviour in the same
> >> subnet, it shows up, as soon as routing is involved in the connection.
> > [...]
> >
> > Which network driver are you using?
> >
> > Is there any packet loss? Run 'netstat -s | grep retransmited' to
> > check this.
>
>
> I checked and I am using the e1000e on all affected systems. I checked
> on a squeeze system with the r8169 driver and the problem seems not to
> appear there.
[...]

OK, so that narrows this down a bit.

We're about to update e1000e in stable to add hardware support, but it
may help with this problem as well. Can you test the appropriate
package from <http://people.debian.org/~benh/packages/>? (The changes
file contains checksums for all files and is signed by me.)

Ben.

Performance problems with squeeze kernel
 

Am 31.08.2011 14:48, schrieb Ben Hutchings:
> On Wed, 2011-08-31 at 14:36 +0200, Thorsten Giese wrote:
>> Am 31.08.2011 14:09, schrieb Ben Hutchings:
>>>> I think I found a performance issue in the kernel (2.6.32/amd64) of the
>>>> squeeze release.
>>>>
>>>> I tested from different servers and each time made a wget of a 16 MB
>>>> file from a remote co-location which is connected with gigabit (but the
>>>> issue showed also on a 32MBit/s "dial-up" cable connection).
>>>>
>>>> The squeeze system starts the transfer significantly slower and takes
>>>> longer to get faster over time (on a 2GB file the maximum was reached
>>>> after 12 seconds). The lenny system reaches this maxium very fast so the
>>>> overall time drops from 2.4s to 0.4s. It could be, this is related to
>>>> changes in TCP/IP behaviour. I could not see this behaviour in the same
>>>> subnet, it shows up, as soon as routing is involved in the connection.
>>> [...]
>>>
>>> Which network driver are you using?
>>>
>>> Is there any packet loss? Run 'netstat -s | grep retransmited' to
>>> check this.
>>
>>
>> I checked and I am using the e1000e on all affected systems. I checked
>> on a squeeze system with the r8169 driver and the problem seems not to
>> appear there.
> [...]
>
> We're about to update e1000e in stable to add hardware support, but it
> may help with this problem as well. Can you test the appropriate
> package from <http://people.debian.org/~benh/packages/>? (The changes
> file contains checksums for all files and is signed by me.)

I installed your kernel an the problem persists. No luck here, sorry.

When the server is powered up (e.g. for the new kernel, there is a short
time, where it seems to work, but then after about 4-5 requests (2-3
minutes) it stops working correctly and shows the problems as described
above.

Thorsten
--
ANW GmbH & Co. KG
Holunderweg 2
66894 Bechhofen

Telefon 06372 8033190
Telefax 06372 8033191
E-Mail kontakt@anw.de
Internet www.anw.de

Sitz der Gesellschaft/Registered Office:
66894 Bechhofen, AG Zweibrücken HRA 30241
Persönlich haftende Gesellschafterin/
Partner with unlimited liability:
ANW Management GmbH, AG Zweibrücken HRB 30449
Geschäftsführung: Oliver Gab, Thorsten Giese
USt-ID/VAT-ID: DE241553575


--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E5E80BA.8010300@anw.de">http://lists.debian.org/4E5E80BA.8010300@anw.de

Performance problems with squeeze kernel
 

On Wed, 2011-08-31 at 20:43 +0200, Thorsten Giese wrote:
> Am 31.08.2011 14:48, schrieb Ben Hutchings:
> > On Wed, 2011-08-31 at 14:36 +0200, Thorsten Giese wrote:
> >> Am 31.08.2011 14:09, schrieb Ben Hutchings:
> >>>> I think I found a performance issue in the kernel (2.6.32/amd64) of the
> >>>> squeeze release.
> >>>>
> >>>> I tested from different servers and each time made a wget of a 16 MB
> >>>> file from a remote co-location which is connected with gigabit (but the
> >>>> issue showed also on a 32MBit/s "dial-up" cable connection).
> >>>>
> >>>> The squeeze system starts the transfer significantly slower and takes
> >>>> longer to get faster over time (on a 2GB file the maximum was reached
> >>>> after 12 seconds). The lenny system reaches this maxium very fast so the
> >>>> overall time drops from 2.4s to 0.4s. It could be, this is related to
> >>>> changes in TCP/IP behaviour. I could not see this behaviour in the same
> >>>> subnet, it shows up, as soon as routing is involved in the connection.
> >>> [...]
> >>>
> >>> Which network driver are you using?
> >>>
> >>> Is there any packet loss? Run 'netstat -s | grep retransmited' to
> >>> check this.
> >>
> >>
> >> I checked and I am using the e1000e on all affected systems. I checked
> >> on a squeeze system with the r8169 driver and the problem seems not to
> >> appear there.
> > [...]
> >
> > We're about to update e1000e in stable to add hardware support, but it
> > may help with this problem as well. Can you test the appropriate
> > package from <http://people.debian.org/~benh/packages/>? (The changes
> > file contains checksums for all files and is signed by me.)
>
> I installed your kernel an the problem persists. No luck here, sorry.
>
> When the server is powered up (e.g. for the new kernel, there is a short
> time, where it seems to work, but then after about 4-5 requests (2-3
> minutes) it stops working correctly and shows the problems as described
> above.

Please use the 'reportbug' program to submit a bug on the kernel
package. This should automatically include various useful system
information.

Ben.

Performance problems with squeeze kernel
 

On Wed, 2011-08-31 at 20:43 +0200, Thorsten Giese wrote:
> Am 31.08.2011 14:48, schrieb Ben Hutchings:
> > On Wed, 2011-08-31 at 14:36 +0200, Thorsten Giese wrote:
> >> Am 31.08.2011 14:09, schrieb Ben Hutchings:
> >>>> I think I found a performance issue in the kernel (2.6.32/amd64) of the
> >>>> squeeze release.
> >>>>
> >>>> I tested from different servers and each time made a wget of a 16 MB
> >>>> file from a remote co-location which is connected with gigabit (but the
> >>>> issue showed also on a 32MBit/s "dial-up" cable connection).
> >>>>
> >>>> The squeeze system starts the transfer significantly slower and takes
> >>>> longer to get faster over time (on a 2GB file the maximum was reached
> >>>> after 12 seconds). The lenny system reaches this maxium very fast so the
> >>>> overall time drops from 2.4s to 0.4s. It could be, this is related to
> >>>> changes in TCP/IP behaviour. I could not see this behaviour in the same
> >>>> subnet, it shows up, as soon as routing is involved in the connection.
> >>> [...]
> >>>
> >>> Which network driver are you using?
> >>>
> >>> Is there any packet loss? Run 'netstat -s | grep retransmited' to
> >>> check this.
> >>
> >>
> >> I checked and I am using the e1000e on all affected systems. I checked
> >> on a squeeze system with the r8169 driver and the problem seems not to
> >> appear there.
> > [...]
> >
> > We're about to update e1000e in stable to add hardware support, but it
> > may help with this problem as well. Can you test the appropriate
> > package from <http://people.debian.org/~benh/packages/>? (The changes
> > file contains checksums for all files and is signed by me.)
>
> I installed your kernel an the problem persists. No luck here, sorry.
>
> When the server is powered up (e.g. for the new kernel, there is a short
> time, where it seems to work, but then after about 4-5 requests (2-3
> minutes) it stops working correctly and shows the problems as described
> above.

Please use the 'reportbug' program to submit a bug on the kernel
package. This should automatically include various useful system
information.

Ben.