Bug#651914: linux-image-2.6.32-5-xen-amd64: Poor IPv6 performance from Xen guest; GSO-related?
Package: linux-2.6
Version: 2.6.32-38
Severity: important
Tags: ipv6
I have three squeeze servers running:
ii linux-image-2.6.32-5-xen-amd64 2.6.32-38 Linux 2.6.32 for 64-bit PCs, Xen dom0 support
All three servers have Intel gigabit NICs, but one server uses the
e1000e driver and the other two use the igb driver.
They've been in production for around 6 months now and it seems like
somewhat embarrassingly we've only just now discovered a problem
with IPv6 performance on the two servers with the igb driver.
The problem manifests itself as awful TCP performance to a Xen domU,
on the order of 15-30KB/sec data transfer. Doing the same data
transfer from the server dom0 itself does not show the same issue,
and the expected tens of MB/sec data transfer is achieved.
Here's an example tcpdump from the dom0 host when the problem is
occurring:
2a00:801:0:11::2 is speedtest.tele2.net which helpfully hosts files
like http://speedtest.tele2.net/100MB.zip for testing purposes. The
above is the result of me using wget to download that file from a
domU on this server. The domU is at 2001:db8:1f1:f240::2 and the
dom0 is at 2001:db8:0:1f1::8.
What I'm noticing is the occasional incorrect checksum and "ICMPv6
packet too big" messages seen above around 23:59:00.672905 and
23:59:01.240946 after a packet of length 2856. These do not occur
on the server with the e1000e driver, where all the packets top out
at 1428. They always occur on the two servers with the igb driver
where the poor throughput is observed.
This also does not occur when the IPv6 traffic is sourced on the dom0.
It's only when it's coming from a Xen domU.
I'm wondering if I am hitting something like this:
I have played with disabling and enabling GSO and checksums on every
interface I can (using ethtool), both in dom0 and domUs, and that makes
no difference.
Looking at linux-source-2.6.32 on squeeze, it does not have this
patch:
although I notice that this commit also touches e1000e where I am
not currently having any problems, even without this commit.
("severity: important" seemed correct because this renders IPv6
effectively unusable for large transfers to/from domUs)
-- Package-specific info:
** Version:
Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-34squeeze1) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Thu May 19 01:16:47 UTC 2011
00:1f.2 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1 [8086:3a20] (prog-if 8f [Master SecP SecO PriP PriO])
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at cc00 [size=8]
Region 1: I/O ports at c880 [size=4]
Region 2: I/O ports at c800 [size=8]
Region 3: I/O ports at c480 [size=4]
Region 4: I/O ports at c400 [size=16]
Region 5: I/O ports at c080 [size=16]
Capabilities: <access denied>
Kernel driver in use: ata_piix
00:1f.3 SMBus [0c05]: Intel Corporation 82801JI (ICH10 Family) SMBus Controller [8086:3a30]
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin C routed to IRQ 18
Region 0: Memory at fbcfa000 (64-bit, non-prefetchable) [size=256]
Region 4: I/O ports at 0400 [size=32]
Kernel driver in use: i801_smbus
00:1f.5 IDE interface [0101]: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2 [8086:3a26] (prog-if 85 [Master SecO PriO])
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at bc00 [size=8]
Region 1: I/O ports at b880 [size=4]
Region 2: I/O ports at b800 [size=8]
Region 3: I/O ports at b480 [size=4]
Region 4: I/O ports at b400 [size=16]
Region 5: I/O ports at b080 [size=16]
Capabilities: <access denied>
Kernel driver in use: ata_piix
01:01.0 VGA compatible controller [0300]: Matrox Graphics, Inc. MGA G200eW WPCM450 [102b:0532] (rev 0a) (prog-if 00 [VGA controller])
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (4000ns min, 8000ns max), Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7000000 (32-bit, prefetchable) [size=16M]
Region 1: Memory at faffc000 (32-bit, non-prefetchable) [size=16K]
Region 2: Memory at fb000000 (32-bit, non-prefetchable) [size=8M]
Expansion ROM at <unassigned> [disabled]
Capabilities: <access denied>
06:00.0 RAID bus controller [0104]: 3ware Inc 9650SE SATA-II RAID PCIe [13c1:1004] (rev 01)
Subsystem: 3ware Inc 9650SE SATA-II RAID PCIe [13c1:1004]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 26
Region 0: Memory at f8000000 (64-bit, prefetchable) [size=32M]
Region 2: Memory at fbddf000 (64-bit, non-prefetchable) [size=4K]
Region 4: I/O ports at d800 [size=256]
Expansion ROM at fbde0000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: 3w-9xxx
08:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 28
Region 0: Memory at fbe60000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at fbe40000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at e880 [size=32]
Region 3: Memory at fbe1c000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at fbe20000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: igb
08:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
Subsystem: Super Micro Computer Inc Device [15d9:060f]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin B routed to IRQ 40
Region 0: Memory at fbee0000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at fbec0000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at ec00 [size=32]
Region 3: Memory at fbe9c000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at fbea0000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: igb
** USB devices:
Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 002: ID 0557:2221 ATEN International Co., Ltd
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-2.6.32-5-xen-amd64 depends on:
ii debconf [debconf-2.0] 1.5.36.1 Debian configuration management sy
ii initramfs-tools 0.98.8 tools for generating an initramfs
ii linux-base 2.6.32-38 Linux image base package
ii module-init-tools 3.12-1 tools for managing Linux kernel mo
Versions of packages linux-image-2.6.32-5-xen-amd64 recommends:
ii firmware-linux-free 2.6.32-38 Binary firmware for various driver
Versions of packages linux-image-2.6.32-5-xen-amd64 suggests:
pn grub <none> (no description available)
pn linux-doc-2.6.32 <none> (no description available)
Versions of packages linux-image-2.6.32-5-xen-amd64 is related to:
pn firmware-bnx2 <none> (no description available)
pn firmware-bnx2x <none> (no description available)
pn firmware-ipw2x00 <none> (no description available)
pn firmware-ivtv <none> (no description available)
pn firmware-iwlwifi <none> (no description available)
pn firmware-linux <none> (no description available)
pn firmware-linux-nonfree <none> (no description available)
pn firmware-qlogic <none> (no description available)
pn firmware-ralink <none> (no description available)
ii xen-hypervisor-4.0-amd64 [xen 4.0.1-2 The Xen Hypervisor on AMD64
--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20111213052323.25157.90833.reportbug@president.bit folk.com">http://lists.debian.org/20111213052323.25157.90833.reportbug@president.bit folk.com
12-13-2011, 05:05 AM
Andy Smith
Bug#651914: linux-image-2.6.32-5-xen-amd64: Poor IPv6 performance from Xen guest; GSO-related?
On Tue, Dec 13, 2011 at 05:23:23AM +0000, Andy Smith wrote:
> What I'm noticing is the occasional incorrect checksum and "ICMPv6
> packet too big" messages seen above around 23:59:00.672905 and
> 23:59:01.240946 after a packet of length 2856. These do not occur
> on the server with the e1000e driver, where all the packets top out
> at 1428. They always occur on the two servers with the igb driver
> where the poor throughput is observed.
Ah, could this be a duplicate of #630730? That was fixed in
2.6.32-39 while the machine in question is running 2.6.32-34.
#630730 seems to suggest that the bug would also have affected
e1000e, though. Another machine running 2.6.32-34 and using driver
e1000e is fine..
Thanks,
Andy
--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20111213060547.GO4221@bitfolk.com">http://lists.debian.org/20111213060547.GO4221@bitfolk.com