Bug#496917: BUG: soft lockup - CPU#1 stuck for 4096s! and crash (kvm guest)

08-28-2008, 03:41 PM

On Thu, Aug 28, 2008 at 02:54:56PM +0000, Lukas Kolbe wrote:
> Package: linux-image-2.6.26-1-amd64
> Version: 2.6.26-3
> Severity: important
>
>
> Using kvm 72, the guest is started with:
> kvm -smp 2
> -net nic,macaddr=DE:AD:BE:EF:21:71,model=virtio
> -net tap,ifname=tap02,script=/etc/kvm/kvm-ifup
> -net nic,macaddr=DE:AD:BE:EF:21:72,model=virtio
> -net tap,ifname=tap12,script=/etc/kvm/kvm-ifup-web
> -drive if=virtio,boot=on,file=/dev/vg0/web-root
> -drive if=virtio,file=/dev/vg0/web-log
> -drive if=virtio,file=/dev/vg0/web-srv
> -drive if=virtio,file=/dev/vg0/web-swap
> -m 3192
> -kernel /boot/vmlinuz-2.6.26-1-amd64
> -initrd /boot/initrd.img-2.6.26-1-amd64
> -append "root=/dev/vda ro"
> -curses
>
> On the guest, I run the latest JDK from Sun 1.6.0_07, and the latest JBoss from
> Redhat 4.2.3.GA, default configuration (except that it's started via runit:
> exec chpst -u jboss:jboss -e ./env -o 65536 /opt/jboss/bin/run.sh -b <host>
>

should be fixed in 2.6.26-4, should be available tomorrow in unstable.
otherwise find sid snapshots http://wiki.debian.org/DebianKernel

thanks

--
maks

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

08-29-2008, 12:20 PM

maximilian attems wrote:

>should be fixed in 2.6.26-4, should be available tomorrow in unstable.
>otherwise find sid snapshots http://wiki.debian.org/DebianKernel
>

Sorry, but it crashed on me again - this time stuck in swapper.

[36037.786125] BUG: soft lockup - CPU#1 stuck for
4097s! [swapper:0]
[36037.786125] Modules linked in: ipv6 snd_pcm snd_timer snd soundcore
snd_page_alloc parport_pc parport serio_raw psmouse pcspkr i2c_piix4
i2c_core button evdev joydev dm_mirror dm_log dm_snapshot dm_mod
ide_cd_mod cdrom ata_generic floppy piix ide_pci_generic thermal fan
virtio_balloon virtio_pci virtio_ring virtio_rng rng_core virtio_net
virtio_blk virtio freq_table processor thermal_sys raid1 raid0 md_mod
atiixp ahci sata_nv sata_sil sata_via libata dock via82cxxx ide_core
3w_9xxx 3w_xxxx scsi_mod xfs ext3 jbd ext2 mbcache reiserfs
[36037.786125] CPU 1:
[36037.786125] Modules linked in: ipv6 snd_pcm snd_timer snd soundcore
snd_page_alloc parport_pc parport serio_raw psmouse pcspkr i2c_piix4
i2c_core button evdev joydev dm_mirror dm_log dm_snapshot dm_mod
ide_cd_mod cdrom ata_generic floppy piix ide_pci_generic thermal fan
virtio_balloon virtio_pci virtio_ring virtio_rng rng_core virtio_net
virtio_blk virtio freq_table processor thermal_sys raid1 raid0 md_mod
atiixp ahci sata_nv sata_sil sata_via libata dock via82cxxx ide_core
3w_9xxx 3w_xxxx scsi_mod xfs ext3 jbd ext2 mbcache reiserfs
[36037.786125] Pid: 0, comm: swapper Not tainted 2.6.26-1-amd64 #1
[36037.786125] RIP: 0010:[<ffffffff8021eb20>] [<ffffffff8021eb20>]
native_safe_halt+0x2/0x3
[36037.786125] RSP: 0018:ffff8100c6ea5f38 EFLAGS: 00000246
[36037.786125] RAX: ffff8100c6ea5fd8 RBX: 0000000000000000 RCX:
0000000000000000
[36037.786125] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffffffff804fadf0
[36037.786125] RBP: 0000000000ce8b7c R08: ffff8100010276a0 R09:
ffff8100c55b4b10
[36037.786125] R10: ffff8100c45dbbc8 R11: ffff8100c485f160 R12:
ffff8100c6ea5ed8
[36037.786125] R13: 0000000000000000 R14: ffffffff8023cd02 R15:
0000196c2157187c
[36037.786125] FS: 00000000414e8960(0000) GS:ffff8100c6e4a0c0(0000)
knlGS:0000000000000000
[36037.786125] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[36037.786125] CR2: 00007f4a0d0c4000 CR3: 00000000c548c000 CR4:
00000000000006e0
[36037.786125] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[36037.786125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[36037.786125]
[36037.786125] Call Trace:
[36037.786125] [<ffffffff8020b0cd>] ? default_idle+0x2a/0x49
[36037.786125] [<ffffffff8020ac79>] ? cpu_idle+0x89/0xb3
[36037.786125]
[40435.828242] BUG: soft lockup - CPU#0 stuck for 8193s! [cron:2082]
[9223956581.178656] Modules linked in: ipv6 snd_pcm snd_timer snd
soundcore snd_page_alloc parport_pc parport serio_raw psmouse pcspkr
i2c_piix4 i2c_core button evdev joydev dm_mirror dm_log dm_snapshot
dm_mod ide_cd_mod cdrom ata_generic floppy piix ide_pci_generic thermal
fan virtio_balloon virtio_pci virtio_ring virtio_rng rng_core virtio_net
virtio_blk virtio freq_table processor thermal_sys raid1 raid0 md_mod
atiixp ahci sata_nv sata_sil sata_via libata dock via82cxxx ide_core
3w_9xxx 3w_xxxx scsi_mod xfs ext3 jbd ext2 mbcache reiserfs
[9223956581.178656] CPU 0:
[9223956581.178656] Modules linked in: ipv6 snd_pcm snd_timer snd
soundcore snd_page_alloc parport_pc parport serio_raw psmouse pcspkr
i2c_piix4 i2c_core button evdev joydev dm_mirror dm_log dm_snapshot
dm_mod ide_cd_mod cdrom ata_generic floppy piix ide_pci_generic thermal
fan virtio_balloon virtio_pci virtio_ring virtio_rng rng_core virtio_net
virtio_blk virtio freq_table processor thermal_sys raid1 raid0 md_mod
atiixp ahci sata_nv sata_sil sata_via libata dock via82cxxx ide_core
3w_9xxx 3w_xxxx scsi_mod xfs ext3 jbd ext2 mbcache reiserfs
[9223956581.178656] Pid: 2082, comm: cron Not tainted 2.6.26-1-amd64 #1
[9223956581.178656] RIP: 0010:[<ffffffff8024aa86>] [<ffffffff8024aa86>]
getnstimeofday+0x9/0x98
[9223956581.178656] RSP: 0018:ffff8100c545df18 EFLAGS: 00000202
[9223956581.178656] RAX: 0000000000ce8b7d RBX: 0000000000ce8b7d RCX:
00000000000007d8
[9223956581.178656] RDX: 0000000000000002 RSI: 0000000000000000 RDI:
ffff8100c545df38
[9223956581.178656] RBP: 0000000000000008 R08: 0000000000000003 R09:
00000000000007d8
[9223956581.178656] R10: 00000000000007d8 R11: 0000000000000246 R12:
0000000000001000
[9223956581.178656] R13: 00000000000001c8 R14: 0000000000000000 R15:
0000000000000000
[9223956581.178656] FS: 00007fcf5ad0e6d0(0000)
GS:ffffffff8053b000(0000) knlGS:0000000000000000
[9223956581.178656] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[9223956581.178656] CR2: 0000000000418238 CR3: 00000000c41ac000 CR4:
00000000000006e0
[9223956581.178656] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[9223956581.178656] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[9223956581.178656]
[9223956581.178656] Call Trace:
[9223956581.178656] [<ffffffff8024aab6>] ? getnstimeofday+0x39/0x98
[9223956581.178656] [<ffffffff8024ab25>] ? do_gettimeofday+0x10/0x32
[9223956581.178656] [<ffffffff80238c3b>] ? sys_gettimeofday+0x1b/0x61
[9223956581.178656] [<ffffffff8020be9a>] ?
system_call_after_swapgs+0x8a/0x8f
[9223956581.178656]

>thanks
>
>--
>maks

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

08-29-2008, 12:36 PM

Sorry, my previous answer didn't make it through my mail setup. I was
using 2.6.26-4snapshot.12144 when the crash happened. I'll try it with
the current snapshot again, though the changelog doesn't say anything
about actual changes

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

08-29-2008, 12:55 PM

On Fri, Aug 29, 2008 at 02:36:00PM +0200, Lukas Kolbe wrote:
>
> Sorry, my previous answer didn't make it through my mail setup. I was
> using 2.6.26-4snapshot.12144 when the crash happened. I'll try it with
> the current snapshot again, though the changelog doesn't say anything
> about actual changes

seeing if it is fixed in 2.6.27-rc5 might be more interesting.
thanks

--
maks

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

08-29-2008, 07:03 PM

maximilian attems wrote:

>On Fri, Aug 29, 2008 at 02:36:00PM +0200, Lukas Kolbe wrote:
>>
>> Sorry, my previous answer didn't make it through my mail setup. I was
>> using 2.6.26-4snapshot.12144 when the crash happened. I'll try it with
>> the current snapshot again, though the changelog doesn't say anything
>> about actual changes

>
>seeing if it is fixed in 2.6.27-rc5 might be more interesting.
>thanks

2.6.27-rc5 has now been running fine in the guest for more than four
hours (and me restarting jboss every now and then). I'll report back tomorrow evening, that would be the timeframe
the bug should've triggered.

>--
>maks

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

09-01-2008, 01:56 PM

Hi!

> >seeing if it is fixed in 2.6.27-rc5 might be more interesting.
> >thanks
>
> 2.6.27-rc5 has now been running fine in the guest for more than four
> hours (and me restarting jboss every now and then). I'll report back tomorrow evening, that would be the timeframe
> the bug should've triggered.

So far the 2.6.27-rc5 seems to be stable, at least it hasn't crashed on
me. What can I do to help to get the needed fix to testing? (I know that
this kernel won't make it and that's a good thing, but I don't really
know how to identify what's needed to fix this).

> >--
> >maks

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

12-14-2008, 09:50 PM

On Mon, Sep 01, 2008 at 03:56:49PM +0200, Lukas Kolbe wrote:
> Hi!
>
> > >seeing if it is fixed in 2.6.27-rc5 might be more interesting.
> > >thanks
> >
> > 2.6.27-rc5 has now been running fine in the guest for more than four
> > hours (and me restarting jboss every now and then). I'll report back tomorrow evening, that would be the timeframe
> > the bug should've triggered.
>
> So far the 2.6.27-rc5 seems to be stable, at least it hasn't crashed on
> me. What can I do to help to get the needed fix to testing? (I know that
> this kernel won't make it and that's a good thing, but I don't really
> know how to identify what's needed to fix this).

Does this bug still persist with the current Lenny kernel?

Cheers,
Moritz

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

12-15-2008, 08:04 PM

Hi,

> > So far the 2.6.27-rc5 seems to be stable, at least it hasn't crashed on
> > me. What can I do to help to get the needed fix to testing? (I know that
> > this kernel won't make it and that's a good thing, but I don't really
> > know how to identify what's needed to fix this).
>
> Does this bug still persist with the current Lenny kernel?

Sorry, I have no machine to test this at the moment. Perhaps I find one
in the next days ...

> Cheers,
> Moritz

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

01-06-2009, 03:56 PM

Am Sonntag, den 14.12.2008, 23:50 +0100 schrieb Moritz Muehlenhoff:
> On Mon, Sep 01, 2008 at 03:56:49PM +0200, Lukas Kolbe wrote:
> > Hi!
> >
> > > >seeing if it is fixed in 2.6.27-rc5 might be more interesting.
> > > >thanks
> > >
> > > 2.6.27-rc5 has now been running fine in the guest for more than four
> > > hours (and me restarting jboss every now and then). I'll report back tomorrow evening, that would be the timeframe
> > > the bug should've triggered.
> >
> > So far the 2.6.27-rc5 seems to be stable, at least it hasn't crashed on
> > me. What can I do to help to get the needed fix to testing? (I know that
> > this kernel won't make it and that's a good thing, but I don't really
> > know how to identify what's needed to fix this).
>
> Does this bug still persist with the current Lenny kernel?

I am sorry to report that I cannot reproduce this bug anymore, but
that's because the system locks up hard (I have no serial console to it
and it is remotely accessible only).

When I do an rsync of ca. 100GB of data on a guest from a remote
location (both guest and host with 2.6.26-1-amd64 2.6.26-12, but it also
happened with -9), after around 10-25GB the host (!) locks up. Ping does
work, but the shells are dead and no further access (via ssh) is
possible at all. After a forced reboot, the logs show up completely
clueless (as in: syslog marks, but nothing else until the new bootup
messages).

When I do the rsync in the host system (chroot'ed to the guest system
for easyness), I can transfer all 100GB without a hitch.

I am using virtio by the way, as in:

#!/bin/bash

KERNEL=2.6.26-1-amd64
NAME=myname

kvm -smp 2
-drive if=virtio,file=/dev/vg0/${NAME}-root,cache=on,boot=on
-drive if=virtio,file=/dev/vg0/${NAME}-log,cache=on,boot=off
-m 512
-nographic
-daemonize
-name ${NAME}
-kernel /boot/kvm/${NAME}/vmlinuz-${KERNEL}
-initrd /boot/kvm/${NAME}/initrd.img-${KERNEL}
-append "root=/dev/vda ro console=ttyS0,115200"
-serial mon:unix:/etc/kvm/consoles/${NAME}.sock,server,nowait
-net nic,macaddr=DE:AD:BE:EF:21:75,model=virtio
-net tap,ifname=tap04,script=/etc/kvm/kvm-ifup
-net nic,macaddr=DE:AD:BE:EF:21:76,model=virtio
-net tap,ifname=tap14,script=/etc/kvm/kvm-ifup-${NAME}

I'm sorry as I can't at the moment reproduce the problem that lead to
open this bug. Maybe I should open a new bug for this?

> Cheers,
> Moritz

All the best and a happy new year,
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

01-07-2009, 01:12 PM

Hi Moritz,

> Does this bug still persist with the current Lenny kernel?

Lucky me (sort of) - we stumbled upon this bug again on another server
(8 cores, 8 GB Ram) using 2.6.26-1-amd64_2.6.26-11, and fortune wants it
that it will be under heavy load tomorrow so that we can try the -12
kernel. I'll report back on it.

> Cheers,
> Moritz

--
Lukas

--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org