FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Kernel

 
 
LinkBack Thread Tools
 
Old 07-22-2010, 12:33 PM
Ben Hutchings
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

On Tue, 2010-07-20 at 22:51 +0100, Ian Jackson wrote:
> Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> > Please try 2.6.34 from experimental.
>
> I've now replicated the problem on my coffee table with a temporary
> (intermittent) loan of an identical machine from Jump networks.
> I can get 2.6.26-21lenny4 and 2.6.26-24 to crash on demand very
> easily. In none of the crashes do I get any kind of kernel log
> messages (eg, soft lockup warnings).
[...]

But I do see a deadlock on the barrier between resync and normal I/O, as
you suspected. Every process is blocked in wait_barrier(),
raise_barrier() or a filesystem lock.

The only explanations I can think of are:
1. The regular I/O requests were miscounted so that the resync process
will wait forever for them to complete in raise_barrier(), while the
other processes wait forever for the barrier to be lowered. (It seems
like the directions of this barrier are inverted. Maybe it is really a
bollard.)
2. The wait condition of one of the waiters was met, but it wasn't
woken.
3. A thread which raised the barrier failed to lower the barrier.
4. A thread which raised the barrier caused paging.

Unfortunately I can't see any evidence for any of these.

The following patch should give us some chance of detecting cases 1 and
3. The instructions at
<http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official> explain how to rebuild an official kernel package with a patch applied.

Ben.

--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -657,6 +657,7 @@

/* block any new IO from starting */
conf->barrier++;
+ WARN_ON(conf->barrier != 1);

/* No wait for all pending IO to complete */
wait_event_lock_irq(conf->wait_barrier,
@@ -671,6 +672,7 @@
{
unsigned long flags;
spin_lock_irqsave(&conf->resync_lock, flags);
+ WARN_ON(conf->barrier == 0);
conf->barrier--;
spin_unlock_irqrestore(&conf->resync_lock, flags);
wake_up(&conf->wait_barrier);
@@ -694,6 +696,7 @@
{
unsigned long flags;
spin_lock_irqsave(&conf->resync_lock, flags);
+ WARN_ON(conf->nr_pending == 0);
conf->nr_pending--;
spin_unlock_irqrestore(&conf->resync_lock, flags);
wake_up(&conf->wait_barrier);
--- END ----

Ben.

--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
 
Old 07-23-2010, 08:47 PM
Ian Jackson
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

Ben Hutchings writes ("Re: Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> The following patch should give us some chance of detecting cases 1 and
> 3. The instructions at
> <http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official> explain how to rebuild an official kernel package with a patch applied.

Thanks. I did this, and the result was that this warning:

> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -657,6 +657,7 @@
>
> /* block any new IO from starting */
> conf->barrier++;
> + WARN_ON(conf->barrier != 1);

triggered an enormous number of times during boot and afterwards. It
was a bit hard to tell but it seemed to be correlated with IO done by
the boot process. It was spinning past at 115200bps at top whack,
mostly.

Eventually the system did boot. I was able to log in and perform my
usual "get it to crash" test, with it spewing these warnings
continuously all the while. I did have a bit of difficulty at first,
but I did succeed. Between my last successful bulk disk access and my
being sure that the system had crashed, it stopped spewing the
warnings and became entirely silent. Now it sits there locked up, and
I can still do IO on the console, but nothing is spewing out.

I'll attach the output for the booting, and the last bit before the
lockup. I'm going to compile the kernel again without that particular
warning (and with kgdb support) and see if I can dig out anything
interesting.

Ian.

GNU GRUB version 0.97 (628K lower / 3135076K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.  Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem  Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686  Debian GNU/Linux, kernel 2.6.26-2-686 (recovery mode)         The highlighted entry will be booted automatically in 10 seconds.   Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem  Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.35-rc5-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686  Debian GNU/Linux, kernel 2.6.26-2-686  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem (recovery mode)  Debian GNU/Linux, kernel 2.6.26-2-686-bigmem  Booting 'Debian GNU/Linux, kernel 2.6.26-2-686-bigmem'

root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /vmlinuz-2.6.26-2-686-bigmem ro root=/dev/vg-main/lv-root console=tty0
console=ttyS0,115200n8 TERM=dumb reboot=bios,smp,warm
[Linux-bzImage, setup=0x3000, size=0x1772d0]
initrd /initrd.img-2.6.26-2-686-bigmem
[Linux-initrd @ 0x3788e000, 0x761b98 bytes]

[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 2.6.26-2-686-bigmem (Debian 2.6.26-24) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Jul 22 21:32:17 BST 2010
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
[ 0.000000] BIOS-e820: 0000000000100000 - 00000000bf699000 (usable)
[ 0.000000] BIOS-e820: 00000000bf699000 - 00000000bf6af000 (reserved)
[ 0.000000] BIOS-e820: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
[ 0.000000] BIOS-e820: 00000000bf6ce000 - 00000000c0000000 (reserved)
[ 0.000000] BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
[ 0.000000] BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
[ 0.000000] 4224MB HIGHMEM available.
[ 0.000000] 896MB LOWMEM available.
[ 0.000000] found SMP MP-table at [c00fe710] 000fe710
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0 -> 4096
[ 0.000000] Normal 4096 -> 229376
[ 0.000000] HighMem 229376 -> 1310720
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[1] active PFN ranges
[ 0.000000] 0: 0 -> 1310720
[ 0.000000] DMI 2.6 present.
[ 0.000000] Using APIC driver default
[ 0.000000] ACPI: RSDP 000F0B40, 0024 (r2 DELL )
[ 0.000000] ACPI: XSDT 000F0C40, 0094 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: FACP BF6C3BB4, 00F4 (r3 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: DSDT BF6AF000, 398C (r1 DELL PE_SC3 1 INTL 20050624)
[ 0.000000] ACPI: FACS BF6C6000, 0040
[ 0.000000] ACPI: APIC BF6C3478, 0152 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: SPCR BF6C35CC, 0050 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: HPET BF6C3620, 0038 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: DM__ BF6C365C, 00A8 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: MCFG BF6C3850, 003C (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: WD__ BF6C3890, 0134 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: SLIC BF6C39C8, 0024 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: ERST BF6B2B0C, 0270 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: HEST BF6B2D7C, 03A8 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: BERT BF6B298C, 0030 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: EINJ BF6B29BC, 0150 (r1 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: TCPA BF6C3B4C, 0064 (r2 DELL PE_SC3 1 DELL 1)
[ 0.000000] ACPI: SSDT BF6C7000, 2164 (r1 INTEL PPM RCM 80000001 INTL 20061109)
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x28] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x29] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2b] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2c] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2d] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x2f] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled)
[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] Enabling APIC mode: Flat. Using 1 I/O APICs
[ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
[ 0.000000] PM: Registered nosave memory: 000000000009d000 - 0000000000100000
[ 0.000000] SMP: Allowing 32 CPUs, 24 hotplug CPUs
[ 0.000000] PERCPU: Allocating 42184 bytes of per cpu data
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1300480
[ 0.000000] Kernel command line: ro root=/dev/vg-main/lv-root console=tty0 console=ttyS0,115200n8 TERM=dumb reboot=bios,smp,warm
[ 0.000000] Enabling fast FPU save and restore... done.
[ 0.000000] Enabling unmasked SIMD FPU exception support... done.
[ 0.000000] Initializing CPU#0
[ 0.000000] PID hash table entries: 4096 (order: 12, 16384 bytes)
[ 0.000000] Extended CMOS year: 2000
[ 0.000000] Detected 1862.039 MHz processor.
[ 0.004000] Console: colour VGA+ 80x25
[ 0.004000] console [tty0] enabled
[ 0.004000] console [ttyS0] enabled
[ 0.004000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.004000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.004000] Memory: 4129272k/5242880k available (1836k kernel code, 54204k reserved, 790k data, 256k init, 3267172k highmem)
[ 0.004000] virtual kernel memory layout:
[ 0.004000] fixmap : 0xffe13000 - 0xfffff000 (1968 kB)
[ 0.004000] pkmap : 0xffa00000 - 0xffc00000 (2048 kB)
[ 0.004000] vmalloc : 0xf8800000 - 0xff9fe000 ( 113 MB)
[ 0.004000] lowmem : 0xc0000000 - 0xf8000000 ( 896 MB)
[ 0.004000] .init : 0xc0399000 - 0xc03d9000 ( 256 kB)
[ 0.004000] .data : 0xc02cb1cd - 0xc0390ac0 ( 790 kB)
[ 0.004000] .text : 0xc0100000 - 0xc02cb1cd (1836 kB)
[ 0.004000] Checking if this processor honours the WP bit even in supervisor mode...Ok.
[ 0.083864] Calibrating delay using timer specific routine.. 3727.31 BogoMIPS (lpj=7454621)
[ 0.090895] Security Framework initialized
[ 0.095052] SELinux: Disabled at boot.
[ 0.098953] Capability LSM initialized
[ 0.102782] Mount-cache hash table entries: 512
[ 0.107490] Initializing cgroup subsys ns
[ 0.111567] Initializing cgroup subsys cpuacct
[ 0.116007] Initializing cgroup subsys devices
[ 0.120007] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.124007] CPU: L2 cache: 256K
[ 0.127209] CPU: L3 cache: 8192K
[ 0.130497] CPU: Physical Processor ID: 0
[ 0.134573] CPU: Processor Core ID: 0
[ 0.138303] Intel machine check architecture supported.
[ 0.143593] Intel machine check reporting enabled on CPU#0.
[ 0.148009] using mwait in idle threads.
[ 0.151997] Checking 'hlt' instruction... OK.
[ 0.172010] ACPI: Core revision 20080321
[ 0.180011] ENABLING IO-APIC IRQs
[ 0.183571] ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.227660] CPU0: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.236014] Booting processor 1/2 ip 6000
[ 0.248015] Initializing CPU#1
[ 0.248015] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447619)
[ 0.248015] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.248015] CPU: L2 cache: 256K
[ 0.248015] CPU: L3 cache: 8192K
[ 0.248015] CPU: Physical Processor ID: 0
[ 0.248015] CPU: Processor Core ID: 1
[ 0.248015] Intel machine check architecture supported.
[ 0.248015] Intel machine check reporting enabled on CPU#1.
[ 0.328020] CPU1: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.337468] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[ 0.344021] Booting processor 2/4 ip 6000
[ 0.360022] Initializing CPU#2
[ 0.360022] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447627)
[ 0.360022] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.360022] CPU: L2 cache: 256K
[ 0.360022] CPU: L3 cache: 8192K
[ 0.360022] CPU: Physical Processor ID: 0
[ 0.360022] CPU: Processor Core ID: 2
[ 0.360022] Intel machine check architecture supported.
[ 0.360022] Intel machine check reporting enabled on CPU#2.
[ 0.436558] CPU2: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.448028] checking TSC synchronization [CPU#0 -> CPU#2]: passed.
[ 0.452028] Booting processor 3/6 ip 6000
[ 0.468029] Initializing CPU#3
[ 0.468029] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447625)
[ 0.468029] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.468029] CPU: L2 cache: 256K
[ 0.468029] CPU: L3 cache: 8192K
[ 0.468029] CPU: Physical Processor ID: 0
[ 0.468029] CPU: Processor Core ID: 3
[ 0.468029] Intel machine check architecture supported.
[ 0.468029] Intel machine check reporting enabled on CPU#3.
[ 0.544619] CPU3: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.556032] checking TSC synchronization [CPU#0 -> CPU#3]: passed.
[ 0.560032] Booting processor 4/1 ip 6000
[ 0.576036] Initializing CPU#4
[ 0.576036] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447605)
[ 0.576036] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.576036] CPU: L2 cache: 256K
[ 0.576036] CPU: L3 cache: 8192K
[ 0.576036] CPU: Physical Processor ID: 0
[ 0.576036] CPU: Processor Core ID: 0
[ 0.576036] Intel machine check architecture supported.
[ 0.576036] Intel machine check reporting enabled on CPU#4.
[ 0.652650] CPU4: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.664036] checking TSC synchronization [CPU#0 -> CPU#4]: passed.
[ 0.668036] Booting processor 5/3 ip 6000
[ 0.684042] Initializing CPU#5
[ 0.684042] Calibrating delay using timer specific routine.. 3723.80 BogoMIPS (lpj=7447618)
[ 0.684042] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.684042] CPU: L2 cache: 256K
[ 0.684042] CPU: L3 cache: 8192K
[ 0.684042] CPU: Physical Processor ID: 0
[ 0.684042] CPU: Processor Core ID: 1
[ 0.684042] Intel machine check architecture supported.
[ 0.684042] Intel machine check reporting enabled on CPU#5.
[ 0.760708] CPU5: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.772042] checking TSC synchronization [CPU#0 -> CPU#5]: passed.
[ 0.776043] Booting processor 6/5 ip 6000
[ 0.788049] Initializing CPU#6
[ 0.788049] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447621)
[ 0.788049] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.788049] CPU: L2 cache: 256K
[ 0.788049] CPU: L3 cache: 8192K
[ 0.788049] CPU: Physical Processor ID: 0
[ 0.788049] CPU: Processor Core ID: 2
[ 0.788049] Intel machine check architecture supported.
[ 0.788049] Intel machine check reporting enabled on CPU#6.
[ 0.868204] CPU6: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.880020] checking TSC synchronization [CPU#0 -> CPU#6]: passed.
[ 0.884043] Booting processor 7/7 ip 6000
[ 0.896056] Initializing CPU#7
[ 0.896056] Calibrating delay using timer specific routine.. 3723.81 BogoMIPS (lpj=7447620)
[ 0.896056] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.896056] CPU: L2 cache: 256K
[ 0.896056] CPU: L3 cache: 8192K
[ 0.896056] CPU: Physical Processor ID: 0
[ 0.896056] CPU: Processor Core ID: 3
[ 0.896056] Intel machine check architecture supported.
[ 0.896056] Intel machine check reporting enabled on CPU#7.
[ 0.976727] CPU7: Intel(R) Xeon(R) CPU L3426 @ 1.87GHz stepping 05
[ 0.988042] checking TSC synchronization [CPU#0 -> CPU#7]: passed.
[ 0.992042] Brought up 8 CPUs
[ 0.996024] Total of 8 processors activated (29793.97 BogoMIPS).
[ 1.000997] net_namespace: 660 bytes
[ 1.004640] Booting paravirtualized kernel on bare hardware
[ 1.008997] NET: Registered protocol family 16
[ 1.012997] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[ 1.019721] ACPI: bus type pci registered
[ 1.023721] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[ 1.027721] PCI: MCFG area at e0000000 reserved in E820
[ 1.034779] PCI: Using MMCONFIG for extended config space
[ 1.038779] PCI: Using configuration type 1 for base access
[ 1.042779] Setting up standard PCI resources
[ 1.050819] ACPI: BIOS _OSI(Linux) query ignored
[ 1.055495] ACPI: DMI System Vendor: Dell Inc.
[ 1.059992] ACPI: DMI Product Name: PowerEdge R210
[ 1.064838] ACPI: DMI Product Version:
[ 1.068729] ACPI: DMI Board Name: 0M877N
[ 1.072710] ACPI: DMI BIOS Vendor: Dell Inc.
[ 1.076974] ACPI: DMI BIOS Date: 01/28/2010
[ 1.080974] ACPI: Please send DMI info above to linux-acpi@vger.kernel.org
[ 1.086735] ACPI: If "acpi_osi=Linux" works better, please notify linux-acpi@vger.kernel.org
[ 1.090735] ACPI: Interpreter enabled
[ 1.094737] ACPI: (supports S0 S4 S5)
[ 1.099730] ACPI: Using IOAPIC for interrupt routing
[ 1.108079] ACPI: PCI Root Bridge [PCI0] (0000:00)
[ 1.113170] PCI: Transparent bridge - 0000:00:1e.0
[ 1.120794] ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 *15)
[ 1.124794] ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 *10 11 14 15)
[ 1.128794] ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[ 1.137190] ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 *6 7 10 11 14 15)
[ 1.144891] ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 *11 14 15)
[ 1.152587] ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[ 1.160794] ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 *14 15)
[ 1.168911] ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
[ 1.177834] Linux Plug and Play Support v0.97 (c) Adam Belay
[ 1.181834] pnp: PnP ACPI init
[ 1.185479] ACPI: bus type pnp registered
[ 1.196246] pnp: PnP ACPI: found 13 devices
[ 1.200246] ACPI: ACPI bus type pnp unregistered
[ 1.204246] PnPBIOS: Disabled by ACPI PNP
[ 1.208247] PCI: Using ACPI for IRQ routing
[ 1.220247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[ 1.227050] hpet0: 8 64-bit timers, 14318180 Hz
[ 1.232683] ACPI: RTC can wake from S4
[ 1.258734] system 00:07: ioport range 0x800-0x87f has been reserved
[ 1.265143] system 00:07: ioport range 0x880-0x8ff has been reserved
[ 1.273026] system 00:07: ioport range 0x900-0x91f has been reserved
[ 1.279436] system 00:07: ioport range 0x920-0x923 has been reserved
[ 1.285868] system 00:07: ioport range 0x924-0x924 has been reserved
[ 1.292291] system 00:07: ioport range 0xca0-0xca7 has been reserved
[ 1.298703] system 00:07: ioport range 0xca9-0xcab has been reserved
[ 1.305134] system 00:07: ioport range 0xcad-0xcaf has been reserved
[ 1.311547] system 00:08: ioport range 0xca8-0xca8 has been reserved
[ 1.317978] system 00:08: ioport range 0xcac-0xcac has been reserved
[ 1.324406] system 00:0a: iomem range 0xe0000000-0xefffffff could not be reserved
[ 1.331963] system 00:0c: iomem range 0xfed50000-0xfed53fff could not be reserved
[ 1.370778] PCI: Bridge: 0000:00:03.0
[ 1.374773] IO window: f000-ffff
[ 1.378781] MEM window: 0xdf100000-0xdf1fffff
[ 1.383367] PREFETCH window: disabled.
[ 1.387356] PCI: Bridge: 0000:00:1c.0
[ 1.391081] IO window: disabled.
[ 1.394782] MEM window: 0xda000000-0xddffffff
[ 1.399376] PREFETCH window: disabled.
[ 1.403366] PCI: Bridge: 0000:00:1e.0
[ 1.407088] IO window: disabled.
[ 1.410781] MEM window: 0xde000000-0xdeffffff
[ 1.415370] PREFETCH window: 0x00000000d9800000-0x00000000d9ffffff
[ 1.421167] ACPI: PCI Interrupt 0000:00:03.0[A] -> GSI 16 (level, low) -> IRQ 16
[ 1.428350] NET: Registered protocol family 2
[ 1.450143] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 1.454656] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[ 1.462810] TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
[ 1.469822] TCP: Hash tables configured (established 131072 bind 65536)
[ 1.476810] TCP reno registered
[ 1.490368] NET: Registered protocol family 1
[ 1.494850] checking if image is initramfs... it is
[ 2.155659] Freeing initrd memory: 7558k freed
[ 2.163663] audit: initializing netlink socket (disabled)
[ 2.275640] type=2000 audit(1279924502.004:1): initialized
[ 2.281857] highmem bounce pool size: 64 pages
[ 2.572644] Total HugeTLB memory allocated, 0
[ 2.581869] VFS: Disk quotas dquot_6.5.1
[ 2.581891] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 2.590252] msgmni has been set to 1700
[ 2.598154] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[ 2.614156] io scheduler noop registered
[ 2.618155] io scheduler anticipatory registered
[ 2.622248] io scheduler deadline registered
[ 2.626589] io scheduler cfq registered (default)
[ 2.648060] assign_interrupt_mode Found MSI capability
[ 2.655515] assign_interrupt_mode Found MSI capability
[ 2.665231] isapnp: Scanning for PnP cards...
[ 3.022514] isapnp: No Plug & Play device found
[ 3.032771] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
[ 3.038315] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 3.050941] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 3.057118] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 3.063028] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 3.069223] brd: module loaded
[ 3.189669] PNP: No PS/2 controller found. Probing ports directly.
[ 3.194528] i8042.c: No controller found.
[ 3.200944] mice: PS/2 mouse device common for all mice
[ 3.206276] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
[ 3.211615] rtc0: alarms up to one day, y3k
[ 3.218083] cpuidle: using governor ladder
[ 3.222085] cpuidle: using governor menu
[ 3.226064] No iBFT detected.
[ 3.229286] TCP cubic registered
[ 3.232572] NET: Registered protocol family 17
[ 3.236888] Using IPI No-Shortcut mode
[ 3.241211] registered taskstats version 1
[ 3.245450] rtc_cmos 00:04: setting system clock to 2010-07-23 22:35:04 UTC (1279924504)
[ 3.253504] Freeing unused kernel memory: 256k freed
Loading, please wait...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... [ 3.938926] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[ 3.943258] ACPI: ACPI0007:00 is registered as cooling_device0
[ 4.863816] ------------[ cut here ]------------
[ 4.867480] ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])
[ 4.867480] ACPI: ACPI0007:01 is registered as cooling_device1
[ 4.883473] WARNING: at arch/x86/kernel/hpet.c:299 hpet_legacy_next_event+0x38/0x5a()
[ 4.883473] Modules linked in: processor(+) fan thermal_sys
[ 4.883473] Pid: 0, comm: swapper Not tainted 2.6.26-2-686-bigmem #1
[ 4.883473] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 4.883473] [<c01e7836>] __next_cpu+0x12/0x21
[ 4.883473] [<c0120239>] find_busiest_group+0x21e/0x623
[ 4.883473] [<c013cfa0>] getnstimeofday+0x37/0xbc
[ 4.883473] [<c01192bb>] hpet_legacy_next_event+0x38/0x5a
[ 4.883473] [<c013f6c8>] clockevents_program_event+0xc4/0xd2
[ 4.883473] [<c01403ca>] tick_dev_program_event+0x1e/0x82
[ 4.883473] [<c013fe50>] tick_broadcast_set_event+0xf/0x11
[ 4.883473] [<c01400da>] tick_broadcast_oneshot_control+0x8a/0xda
[ 4.883473] [<c013fb24>] tick_notify+0x1cd/0x2e3
[ 4.883473] [<c013ace6>] enqueue_hrtimer+0xc9/0xd4
[ 4.883473] [<c013be0c>] notifier_call_chain+0x2a/0x47
[ 4.883473] [<c013be50>] raw_notifier_call_chain+0x9/0xc
[ 4.883473] [<c013f5c8>] clockevents_notify+0x19/0x55
[ 4.883473] [<f8848f1b>] acpi_idle_enter_bm+0x12f/0x315 [processor]
[ 4.883473] [<c025b4f9>] cpuidle_idle_call+0x5b/0x86
[ 4.883473] [<c025b49e>] cpuidle_idle_call+0x0/0x86
[ 4.883473] [<c01075d3>] cpu_idle+0xb0/0xd0
[ 4.883473] =======================
[ 4.883473] ---[ end trace 66948f2233c04087 ]---
[ 5.007472] Marking TSC unstable due to: TSC halts in idle.
[ 5.147488] ACPI: CPU2 (power states: C1[C1] C2[C2] C3[C3])
[ 5.147488] ACPI: ACPI0007:02 is registered as cooling_device2
[ 5.159525] ACPI: CPU3 (power states: C1[C1] C2[C2] C3[C3])
[ 5.175510] ACPI: ACPI0007:03 is registered as cooling_device3
[ 5.343505] ACPI: CPU4 (power states: C1[C1] C2[C2] C3[C3])
[ 5.789408] ACPI: ACPI0007:04 is registered as cooling_device4
[ 5.795299] ACPI: CPU5 (power states: C1[C1] C2[C2] C3[C3])
[ 5.803608] ACPI: ACPI0007:05 is registered as cooling_device5
[ 5.809538] ACPI: CPU6 (power states: C1[C1] C2[C2] C3[C3])
[ 5.826296] ACPI: ACPI0007:06 is registered as cooling_device6
[ 5.832181] ACPI: CPU7 (power states: C1[C1] C2[C2] C3[C3])
[ 5.873327] ACPI: ACPI0007:07 is registered as cooling_device7
[ 6.013722] usbcore: registered new interface driver usbfs
[ 6.019313] usbcore: registered new interface driver hub
[ 6.031034] usbcore: registered new device driver usb
[ 6.032670] e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
[ 6.037893] e1000e: Copyright (c) 1999-2008 Intel Corporation.
[ 6.047075] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[ 6.063206] No dock devices found.
[ 6.072740] SCSI subsystem initialized
[ 6.173171] eth0: (PCI Express:2.5GB/s:Width x1) 00:1b:21:6b:5a:d0
[ 6.179686] eth0: Intel(R) PRO/1000 Network Connection
[ 6.184910] eth0: MAC: 3, PHY: 8, PBA No: e46981-003
[ 6.191507] ACPI: PCI Interrupt 0000:00:1a.0[A] -> <6>udev: renamed network interface eth0 to eth3
[ 6.203205] GSI 22 (level, low) -> IRQ 22
[ 6.207288] ehci_hcd 0000:00:1a.0: EHCI Host Controller
[ 6.212634] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
[ 6.224179] ehci_hcd 0000:00:1a.0: debug port 2
[ 6.232062] ehci_hcd 0000:00:1a.0: irq 22, io mem 0xdf0fa000
[ 6.250606] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[ 6.254705] usb usb1: configuration #1 chosen from 1 choice
[ 6.264296] hub 1-0:1.0: USB hub found
[ 6.272278] hub 1-0:1.0: 2 ports detected
[ 6.378630] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 6.383032] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 6.390355] usb usb1: Product: EHCI Host Controller
[ 6.395319] usb usb1: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[ 6.402337] usb usb1: SerialNumber: 0000:00:1a.0
[ 6.409887] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 22 (level, low) -> IRQ 22
[ 6.414823] ehci_hcd 0000:00:1d.0: EHCI Host Controller
[ 6.422695] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[ 6.438065] ehci_hcd 0000:00:1d.0: debug port 2
[ 6.442705] ehci_hcd 0000:00:1d.0: irq 22, io mem 0xdf0fc000
[ 6.462483] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[ 6.470090] usb usb2: configuration #1 chosen from 1 choice
[ 6.474068] hub 2-0:1.0: USB hub found
[ 6.482051] hub 2-0:1.0: 2 ports detected
[ 6.589036] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[ 6.591983] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 6.599985] usb usb2: Product: EHCI Host Controller
[ 6.605015] usb usb2: Manufacturer: Linux 2.6.26-2-686-bigmem ehci_hcd
[ 6.612353] usb usb2: SerialNumber: 0000:00:1d.0
[ 6.617034] Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.7.5 (April 29, 2008)
[ 6.617034] ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 20 (level, low) -> IRQ 20
[ 6.620611] usb 1-1: new high speed USB device using ehci_hcd and address 2
[ 6.642800] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[ 6.652158] firmware: requesting bnx2-09-4.0.5.fw
[ 6.665878] eth0: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem da000000, IRQ 16, node addr b8:ac:6f:87:77:89
[ 6.695737] ACPI: PCI Interrupt 0000:02:00.1[b] -> GSI 17 (level, low) -> IRQ 17
[ 6.704703] firmware: requesting bnx2-09-4.0.5.fw
[ 6.715935] eth1: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 17, node addr b8:ac:6f:87:77:8a
[ 6.739893] udev: renamed network interface eth1 to eth2
[ 6.790342] udev: renamed network interface eth0_rename to eth1
[ 6.791754] usb 1-1: configuration #1 chosen from 1 choice
[ 6.791914] hub 1-1:1.0: USB hub found
[ 6.792005] hub 1-1:1.0: 6 ports detected
[ 6.899618] usb 1-1: New USB device found, idVendor=8087, idProduct=0020
[ 6.904523] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 7.193415] usb 2-1: new high speed USB device using ehci_hcd and address 2
[ 7.331178] usb 2-1: configuration #1 chosen from 1 choice
[ 7.355242] hub 2-1:1.0: USB hub found
[ 7.360394] hub 2-1:1.0: 8 ports detected
[ 7.488156] usb 2-1: New USB device found, idVendor=8087, idProduct=0020
[ 7.493867] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 7.699771] usb 2-1.1: new low speed USB device using ehci_hcd and address 3
[ 7.741535] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[ 7.749852] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pmp pio slum part
[ 7.758594] scsi0 : ahci
[ 7.758594] scsi1 : ahci
[ 7.778883] scsi2 : ahci
[ 7.782658] scsi3 : ahci
[ 7.790221] scsi4 : ahci
[ 7.790221] scsi5 : ahci
[ 7.790221] ata1: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe100 irq 218
[ 7.790221] ata2: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe180 irq 218
[ 7.790221] ata3: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe200 irq 218
[ 7.790221] ata4: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe280 irq 218
[ 7.790221] ata5: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe300 irq 218
[ 7.827018] usb 2-1.1: configuration #1 chosen from 1 choice
[ 7.827185] usb 2-1.1: New USB device found, idVendor=0a81, idProduct=0101
[ 7.827185] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 7.827185] usb 2-1.1: Product: USB Keyboard
[ 7.827185] usb 2-1.1: Manufacturer: CHESEN
[ 7.945282] ata6: SATA max UDMA/133 abar m2048@0xdf0fe000 port 0xdf0fe380 irq 218
[ 8.122697] usb 2-1.5: new high speed USB device using ehci_hcd and address 4
[ 8.220329] usb 2-1.5: configuration #1 chosen from 1 choice
[ 8.225119] hub 2-1.5:1.0: USB hub found
[ 8.231311] hub 2-1.5:1.0: 4 ports detected
[ 8.346824] usb 2-1.5: New USB device found, idVendor=0424, idProduct=2514
[ 8.351725] usb 2-1.5: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 8.362258] usbcore: registered new interface driver hiddev
[ 8.374754] input: CHESEN USB Keyboard as /class/input/input0
[ 8.386158] input,hidraw0: USB HID v1.10 Keyboard [CHESEN USB Keyboard] on usb-0000:00:1d.0-1.1
[ 8.409581] input: CHESEN USB Keyboard as /class/input/input1
[ 8.416083] input,hidraw1: USB HID v1.10 Device [CHESEN USB Keyboard] on usb-0000:00:1d.0-1.1
[ 8.424940] usbcore: registered new interface driver usbhid
[ 8.432929] usbhid: v2.6:USB HID core driver
[ 8.463900] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 8.482198] ata1.00: ATA-8: WDC WD1602ABKS-18N8A0, 02.03B04, max UDMA/133
[ 8.489167] ata1.00: 312500000 sectors, multi 0: LBA48 NCQ (depth 31/32)
[ 8.497295] ata1.00: configured for UDMA/133
[ 8.996655] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 9.003969] ata2.00: ATA-8: WDC WD1602ABKS-18N8A0, 02.03B04, max UDMA/133
[ 9.010969] ata2.00: 312500000 sectors, multi 0: LBA48 NCQ (depth 31/32)
[ 9.020017] ata2.00: configured for UDMA/133
[ 9.359143] ata3: SATA link down (SStatus 0 SControl 300)
[ 9.862534] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 9.878187] ata4.00: ATAPI: TEAC DVD-ROM DV-28SW, R.2A, max UDMA/100, ATAPI AN
[ 9.902001] ata4.00: configured for UDMA/100
[ 10.242612] ata5: SATA link down (SStatus 0 SControl 300)
[ 10.582428] ata6: SATA link down (SStatus 0 SControl 300)
[ 10.709539] scsi 0:0:0:0: Direct-Access ATA WDC WD1602ABKS-1 02.0 PQ: 0 ANSI: 5
[ 10.709539] scsi 1:0:0:0: Direct-Access ATA WDC WD1602ABKS-1 02.0 PQ: 0 ANSI: 5
[ 10.849809] scsi 3:0:0:0: CD-ROM TEAC DVD-ROM DV-28SW R.2A PQ: 0 ANSI: 5
[ 10.892024] Driver 'sr' needs updating - please use bus_type methods
[ 10.913465] Driver 'sd' needs updating - please use bus_type methods
[ 10.924140] sd 0:0:0:0: [sda] 312500000 512-byte hardware sectors (160000 MB)
[ 10.930641] sd 0:0:0:0: [sda] Write Protect is off
[ 10.936887] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 10.938056] sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
[ 10.938061] Uniform CD-ROM driver Revision: 3.20
[ 10.972205] sd 0:0:0:0: [sda] 312500000 512-byte hardware sectors (160000 MB)
[ 10.979552] sd 0:0:0:0: [sda] Write Protect is off
[ 10.984492] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 10.993864] sda:<5>sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 11.009149] sda1 sda2
[ 11.011450] scsi 1:0:0:0: Attached scsi generic sg1 type 0
[ 11.013753] sd 0:0:0:0: [sda] Attached SCSI disk
[ 11.013786] sd 1:0:0:0: [sdb] 312500000 512-byte hardware sectors (160000 MB)
[ 11.017184] sd 1:0:0:0: [sdb] Write Protect is off
[ 11.017208] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 11.017225] sd 1:0:0:0: [sdb] 312500000 512-byte hardware sectors (160000 MB)
[ 11.013815] sd 1:0:0:0: [sdb] Write Protect is off
[ 11.013815] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 11.013815] sdb: sdb1 sdb2
[ 11.076682] sr 3:0:0:0: Attached scsi generic sg2 type 5
[ 11.076682] sd 1:0:0:0: [sdb] Attached SCSI disk
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... Begin: Loading[ 11.337836] device-mapper: uevent: version 1.0.3
[ 11.355285] device-mapper: ioctl: 4.13.0-ioctl (2007-10-18) initialised: dm-devel@redhat.com
Success: loaded module dm-mod.
Success: loaded module dm-snapshot.
[ 11.388754] md: raid0 personality registered for level 0
Success: loaded module raid0.
[ 11.401037] md: raid1 personality registered for level 1
Success: loaded module raid1.
done.
+ lvm vgchange -a y chiark-early-a
2 logical volume(s) in volume group "chiark-early-a" now active
+ lvm vgchange -a y chiark-early-b
2 logical volume(s) in volume group "chiark-early-b" now active
mdadm: cannot open /dev/md/chiark-mirror: No [ 11.529335] md: md127 stopped.
such file or directory
+ /sbin/mdadm --assemble --no-degraded --auto=md /dev/md/chiark-mirror
[ 11.573007] md: bind<dm-2>
[ 11.576085] md: bind<dm-0>
[ 11.578882] md: md127: raid array is not clean -- starting background reconstruction
[ 11.624950] raid1: raid set md127 active with 2 out of 2 mirrors
mdadm: /dev/md/chiark-mirror has been started with 2 drives.
Reading all physical volumes. This may take a while...
Found volume group "vg-main" using metadata type lvm2
Found volume group "chiark-early-b" using metadata type lvm2
Found volume group "chiark-early-a" using metadata type lvm2
9 logical volume(s) in volume group "vg-main" now active
EARLY ROOT LVM OK
Begin: Loadingdone.
Begin: Waiting for udev to process events ... done.
done.
Begin: Running /scripts/local-premount ... done.
[ 12.243768] kjournald starting. Commit interval 5 seconds
Begin: Running /[ 12.243768] EXT3-fs: mounted filesystem with ordered data mode.
scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.

INIT: version 2.86 booting

Setting hostname to 'rescue'...done.
Files under mount point '/lib/init/rw' will be hidden. (warning).
Starting the hotplug events dispatcher: udevd[ 13.196885] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[ 13.619201] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
[ 13.648270] input: PC Speaker as /class/input/input2
[ 13.769390] input: Power Button (FF) as /class/input/input3
[ 13.817584] ACPI: Power Button (FF) [PWRF]
done.
RAID status not OK. Exiting. failed!
Setting the system clock.
System Clock set to: Fri Jul 23 22:35:12 UTC 2010.
Not activating swap on logical volume. (warning).
Will now check root file system:fsck 1.41.3 (12-Oct-2008)
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/mapper/vg--main-lv--root
/dev/mapper/vg--main-lv--root: clean, 16743/98304 files, 298478/393216 blocks
.
[ 14.976504] md: resync of RAID array md127
[ 14.981175] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 14.986643] EXT3 FS on dm-4, internal journal
[ 14.994599] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 15.004184] md: using 128k window, over a total of 75400320 blocks.
[ 15.011625] md: resuming resync of md127 from checkpoint.
[ 15.020348] ------------[ cut here ]------------
[ 15.023576] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 15.031350] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 15.059078] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 15.059078] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 15.059078] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 15.059078] [<c01db25d>] elv_insert+0x102/0x196
[ 15.059078] [<c01dda71>] __make_request+0x2fe/0x348
[ 15.059078] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 15.059078] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 15.059078] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 15.059078] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 15.059078] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 15.059078] [<c012126f>] __wake_up+0x29/0x39
[ 15.059078] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 15.059078] [<c02c8f8d>] schedule+0x63b/0x66d
[ 15.059078] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.059078] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 15.059078] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.059078] [<c013876b>] kthread+0x38/0x5d
[ 15.059078] [<c0138733>] kthread+0x0/0x5d
[ 15.059078] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 15.059078] =======================
[ 15.059078] ---[ end trace 66948f2233c04087 ]---
[ 15.208006] ------------[ cut here ]------------
[ 15.208006] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
Setting the syst[ 15.208006] Modules linked in:em clock.
button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 15.251058] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 15.251058] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 15.251058] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 15.251058] [<c01db25d>] elv_insert+0x102/0x196
[ 15.251058] [<c01dda71>] __make_request+0x2fe/0x348
[ 15.251058] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 15.251058] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 15.251058] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 15.251058] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 15.251058] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 15.251058] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 15.251058] [<c02c8f8d>] schedule+0x63b/0x66d
[ 15.251058] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.251058] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 15.251058] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.251058] [<c013876b>] kthread+0x38/0x5d
[ 15.251058] [<c0138733>] kthread+0x0/0x5d
[ 15.251058] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 15.251058] =======================
[ 15.251058] ---[ end trace 66948f2233c04087 ]---
[ 15.377706] ------------[ cut here ]------------
System Clock set[ 15.382407] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
to: Fri Jul 23 [ 15.394056] Modules linked in:22:35:13 UTC 201 button0.
pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 15.426049] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 15.426049] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 15.426049] [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[ 15.426049] [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[ 15.426049] [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[ 15.426049] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 15.426049] [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[ 15.426049] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 15.426049] [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[ 15.426049] [<c013cfa0>] getnstimeofday+0x37/0xbc
[ 15.426049] [<c013b418>] ktime_get_ts+0x22/0x49
[ 15.426049] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 15.426049] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 15.426049] [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[ 15.426049] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 15.426049] [<c02c8f8d>] schedule+0x63b/0x66d
[ 15.426049] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.426049] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 15.426049] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.426049] [<c013876b>] kthread+0x38/0x5d
[ 15.426049] [<c0138733>] kthread+0x0/0x5d
[ 15.426049] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 15.426049] =======================
[ 15.426049] ---[ end trace 66948f2233c04087 ]---
[ 15.565176] ------------[ cut here ]------------
[ 15.569881] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 15.577155] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 15.605927] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 15.605927] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 15.605927] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 15.605927] [<c01db25d>] elv_insert+0x102/0x196
[ 15.605927] [<c01dda71>] __make_request+0x2fe/0x348
[ 15.605927] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 15.605927] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 15.605927] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 15.605927] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 15.605927] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 15.605927] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 15.605927] [<c02c8f8d>] schedule+0x63b/0x66d
[ 15.605927] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.605927] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 15.605927] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.605927] [<c013876b>] kthread+0x38/0x5d
[ 15.605927] [<c0138733>] kthread+0x0/0x5d
[ 15.605927] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 15.605927] =======================
[ 15.605927] ---[ end trace 66948f2233c04087 ]---
[ 15.716679] ------------[ cut here ]------------
[ 15.721394] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 15.728630] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 15.753489] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 15.753489] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 15.753489] [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[ 15.753489] [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[ 15.753489] [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[ 15.753489] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 15.753489] [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[ 15.753489] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 15.753489] [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[ 15.753489] [<c013cfa0>] getnstimeofday+0x37/0xbc
[ 15.753489] [<c013b418>] ktime_get_ts+0x22/0x49
[ 15.753489] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 15.753489] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 15.753489] [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[ 15.753489] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 15.753489] [<c02c8f8d>] schedule+0x63b/0x66d
[ 15.753489] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.753489] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 15.753489] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 15.753489] [<c013876b>] kthread+0x38/0x5d
[ 15.753489] [<c0138733>] kthread+0x0/0x5d
[ 15.753489] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 15.753489] =======================
[ 15.753489] ---[ end trace 66948f2233c04087 ]---
[ 15.972618] ------------[ cut here ]------------
[ 15.977362] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 15.984597] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 16.009364] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 16.009364] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 16.009364] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 16.009364] [<c01db25d>] elv_insert+0x102/0x196
[ 16.009364] [<c01dda71>] __make_request+0x2fe/0x348
[ 16.009364] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 16.009364] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 16.009364] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 16.009364] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 16.009364] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 16.009364] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 16.009364] [<c02c8f8d>] schedule+0x63b/0x66d
[ 16.009364] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.009364] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 16.009364] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.009364] [<c013876b>] kthread+0x38/0x5d
[ 16.009364] [<c0138733>] kthread+0x0/0x5d
[ 16.009364] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 16.009364] =======================
[ 16.009364] ---[ end trace 66948f2233c04087 ]---
[ 16.133608] ------------[ cut here ]------------
[ 16.136222] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 16.142045] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 16.171563] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 16.171563] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 16.171563] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 16.171563] [<c01db25d>] elv_insert+0x102/0x196
[ 16.171563] [<c01dda71>] __make_request+0x2fe/0x348
[ 16.171563] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 16.171563] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 16.171563] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 16.171563] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 16.171563] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 16.171563] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 16.171563] [<c02c8f8d>] schedule+0x63b/0x66d
[ 16.171563] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.171563] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 16.171563] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.171564] [<c013876b>] kthread+0x38/0x5d
[ 16.171564] [<c0138733>] kthread+0x0/0x5d
[ 16.171564] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 16.171564] =======================
[ 16.171564] ---[ end trace 66948f2233c04087 ]---
[ 16.303096] ------------[ cut here ]------------
[ 16.303096] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 16.303096] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 16.349056] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 16.349056] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 16.349056] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 16.349056] [<c01db25d>] elv_insert+0x102/0x196
[ 16.349056] [<c01dda71>] __make_request+0x2fe/0x348
[ 16.349056] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 16.349056] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 16.349056] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 16.349056] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 16.349056] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 16.349056] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 16.349056] [<c02c8f8d>] schedule+0x63b/0x66d
[ 16.349056] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.349056] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 16.349056] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.349056] [<c013876b>] kthread+0x38/0x5d
[ 16.349056] [<c0138733>] kthread+0x0/0x5d
[ 16.349056] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 16.349056] =======================
[ 16.349056] ---[ end trace 66948f2233c04087 ]---
[ 16.657662] ------------[ cut here ]------------
[ 16.657662] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 16.665653] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 16.697660] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 16.697660] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 16.697660] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 16.697660] [<c01db25d>] elv_insert+0x102/0x196
[ 16.697660] [<c01dda71>] __make_request+0x2fe/0x348
[ 16.697660] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 16.697660] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 16.697660] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 16.697660] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 16.697660] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 16.697660] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 16.697660] [<c02c8f8d>] schedule+0x63b/0x66d
[ 16.697660] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.697660] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 16.697660] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.697660] [<c013876b>] kthread+0x38/0x5d
[ 16.697660] [<c0138733>] kthread+0x0/0x5d
[ 16.697660] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 16.697660] =======================
[ 16.697660] ---[ end trace 66948f2233c04087 ]---
[ 16.835780] ------------[ cut here ]------------
[ 16.835780] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
Cleaning up ifup[ 16.835780] Modules linked in:down... button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 16.871876] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 16.871876] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 16.871876] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 16.871876] [<c01db25d>] elv_insert+0x102/0x196
[ 16.871876] [<c01dda71>] __make_request+0x2fe/0x348
[ 16.871876] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 16.871876] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 16.871876] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 16.871876] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 16.871876] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 16.871876] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 16.871876] [<c02c8f8d>] schedule+0x63b/0x66d
[ 16.871876] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.871876] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 16.871876] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 16.871876] [<c013876b>] kthread+0x38/0x5d
[ 16.871876] [<c0138733>] kthread+0x0/0x5d
[ 16.871876] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 16.871876] =======================
[ 16.871876] ---[ end trace 66948f2233c04087 ]---
[ 17.056673] ------------[ cut here ]------------
[ 17.056673] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 17.056673] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 17.097066] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 17.097066] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 17.097066] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 17.097066] [<c01db25d>] elv_insert+0x102/0x196
[ 17.097066] [<c01dda71>] __make_request+0x2fe/0x348
[ 17.097066] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 17.097066] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 17.097066] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 17.097066] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 17.097066] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 17.097066] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 17.097066] [<c02c8f8d>] schedule+0x63b/0x66d
[ 17.097066] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 17.097066] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 17.097066] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 17.097066] [<c013876b>] kthread+0x38/0x5d
[ 17.097066] [<c0138733>] kthread+0x0/0x5d
[ 17.097066] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 17.097066] =======================
[ 17.097066] ---[ end trace 66948f2233c04087 ]---
[ 17.246032] ------------[ cut here ]------------
.
[ 17.246032] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 17.246032] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 17.300851] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 17.300851] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 17.300851] [<f8873c82>] ahci_qc_prep+0x7d/0x123 [ahci]
[ 17.300851] [<f8873cdf>] ahci_qc_prep+0xda/0x123 [ahci]
[ 17.300851] [<f8930e05>] ata_qc_issue+0x161/0x1b6 [libata]
[ 17.300851] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 17.300851] [<f89352ba>] ata_scsi_translate+0xd2/0xf7 [libata]
[ 17.300851] [<f890e426>] scsi_done+0x0/0x16 [scsi_mod]
[ 17.300851] [<f89374af>] ata_scsi_queuecmd+0x195/0x1b1 [libata]
[ 17.300851] [<c0114acb>] apic_wait_icr_idle+0xe/0x15
[ 17.300851] [<c0113f77>] native_smp_send_reschedule+0x33/0x34
[ 17.300851] [<c011fc26>] __resched_task+0x5c/0x5f
[ 17.300851] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 17.300851] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 17.300851] [<f89e5b51>] md_wakeup_thread+0x1e/0x20 [md_mod]
[ 17.300851] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 17.300851] [<c02c8f8d>] schedule+0x63b/0x66d
[ 17.300851] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 17.300851] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 17.300851] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 17.300851] [<c013876b>] kthread+0x38/0x5d
[ 17.300851] [<c0138733>] kthread+0x0/0x5d
[ 17.300851] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 17.300851] =======================
[ 17.300851] ---[ end trace 66948f2233c04087 ]---
[ 18.260106] ------------[ cut here ]------------
[ 18.264845] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 18.272085] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 18.300856] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 18.300856] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 18.300856] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 18.300856] [<c01db25d>] elv_insert+0x102/0x196
[ 18.300856] [<c01dda71>] __make_request+0x2fe/0x348
[ 18.300856] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 18.300856] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 18.300856] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 18.300856] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 18.300856] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 18.300856] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 18.300856] [<c02c8f8d>] schedule+0x63b/0x66d
[ 18.300856] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.300856] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 18.300856] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.300856] [<c013876b>] kthread+0x38/0x5d
[ 18.300856] [<c0138733>] kthread+0x0/0x5d
[ 18.300856] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 18.300856] =======================
[ 18.300856] ---[ end trace 66948f2233c04087 ]---
[ 18.435184] ------------[ cut here ]------------
[ 18.435184] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 18.435184] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 18.474951] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 18.474951] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 18.474951] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 18.474951] [<c01db25d>] elv_insert+0x102/0x196
[ 18.474951] [<c01dda71>] __make_request+0x2fe/0x348
[ 18.474951] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 18.474951] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 18.474951] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 18.474951] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 18.474951] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 18.474951] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 18.474951] [<c02c8f8d>] schedule+0x63b/0x66d
[ 18.474951] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.474951] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 18.474951] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.474951] [<c013876b>] kthread+0x38/0x5d
[ 18.474951] [<c0138733>] kthread+0x0/0x5d
[ 18.474951] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 18.474951] =======================
[ 18.474951] ---[ end trace 66948f2233c04087 ]---
[ 18.624832] ------------[ cut here ]------------
[ 18.624832] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 18.624832] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 18.662446] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 18.662446] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 18.662446] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 18.662447] [<c01db25d>] elv_insert+0x102/0x196
[ 18.662447] [<c01dda71>] __make_request+0x2fe/0x348
[ 18.662447] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 18.662447] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 18.662447] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 18.662447] [<c019e0a3>] bio_add_page+0x25/0x2e
[ 18.662447] [<f89c232d>] sync_request+0x1dd/0x59a [raid1]
[ 18.662447] [<f89ec9f3>] md_do_sync+0x616/0xa9f [md_mod]
[ 18.662447] [<c02c8f8d>] schedule+0x63b/0x66d
[ 18.662447] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.662447] [<f89ec3c7>] md_thread+0xb7/0xcd [md_mod]
[ 18.662447] [<f89ec310>] md_thread+0x0/0xcd [md_mod]
[ 18.662447] [<c013876b>] kthread+0x38/0x5d
[ 18.662447] [<c0138733>] kthread+0x0/0x5d
[ 18.662447] [<c01094f7>] kernel_thread_helper+0x7/0x10
[ 18.662447] =======================
[ 18.662447] ---[ end trace 66948f2233c04087 ]---
[ 18.792884] ------------[ cut here ]------------
[ 18.792884] WARNING: at drivers/md/raid1.c:660 raise_barrier+0xa8/0x146 [raid1]()
[ 18.804882] Modules linked in: button pcspkr dcdbas evdev ext3 jbd mbcache dm_mirror dm_log raid1 raid0 md_mod dm_snapshot dm_mod sg sd_mod sr_mod cdrom usbhid hid ff_memless ahci libata scsi_mod dock bnx2 ehci_hcd firmware_class e1000e usbcore thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[ 18.832882] Pid: 2287, comm: md127_resync Tainted: G W 2.6.26-2-686-bigmem #1
[ 18.832882] [<c0129183>] warn_on_slowpath+0x40/0x66
[ 18.832882] [<c01e5f68>] cfq_insert_request+0x34/0x39e
[ 18.832882] [<c01db25d>] elv_insert+0x102/0x196
[ 18.832882] [<c01dda71>] __make_request+0x2fe/0x348
[ 18.832882] [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
[ 18.832882] [<c01dc690>] generic_make_request+0x34d/0x37b
[ 18.832882] [<f89c048e>] raise_barrier+0xa8/0x146 [raid1]
[ 18.832882] [<c
 
Old 07-23-2010, 10:58 PM
Ian Jackson
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

I wrote:
> ... I'm going to compile the kernel again without that particular
> warning (and with kgdb support) and see if I can dig out anything
> interesting.

My attempt to compile the kernel with kgdb support failed. Something
in the Debian kernel packaging thingy complained like this:
ABI has changed! Refusing to continue.
Added symbols:
kgdb_breakpoint module: vmlinux,
version: 0x01e1a8de, export: EXPORT_SYMBOL_GPL
...
Changed symbols:
__audit_inode_child module: vmlinux, version: 0xead6af60 -> 0xe3835d2d, export: EXPORT_SYMBOL_GPL
__fatal_signal_pending module: vmlinux, version: 0xb5428735 -> 0xee8d0603, export: EXPORT_SYMBOL
[lots more]

I've run out of time to do more on this now.

Ian.



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19530.7824.326265.929499@chiark.greenend.org.uk">h ttp://lists.debian.org/19530.7824.326265.929499@chiark.greenend.org.uk
 
Old 07-24-2010, 12:10 AM
Ben Hutchings
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

On Fri, 2010-07-23 at 23:58 +0100, Ian Jackson wrote:
> I wrote:
> > ... I'm going to compile the kernel again without that particular
> > warning (and with kgdb support) and see if I can dig out anything
> > interesting.
>
> My attempt to compile the kernel with kgdb support failed. Something
> in the Debian kernel packaging thingy complained like this:
> ABI has changed! Refusing to continue.
> Added symbols:
> kgdb_breakpoint module: vmlinux,
> version: 0x01e1a8de, export: EXPORT_SYMBOL_GPL
> ...
> Changed symbols:
> __audit_inode_child module: vmlinux, version: 0xead6af60 -> 0xe3835d2d, export: EXPORT_SYMBOL_GPL
> __fatal_signal_pending module: vmlinux, version: 0xb5428735 -> 0xee8d0603, export: EXPORT_SYMBOL
> [lots more]
>
> I've run out of time to do more on this now.

If you remove the 'debian/abi' directory this check will be skipped.

Ben.

--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
 
Old 01-17-2011, 11:56 AM
"Mark"
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

Related:

https://bugzilla.kernel.org/show_bug.cgi?id=12905





--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 7ebf2d89f525c3b724aca5d197887aea.squirrel@mstier.d e">http://lists.debian.org/7ebf2d89f525c3b724aca5d197887aea.squirrel@mstier.d e
 
Old 04-01-2012, 08:13 PM
Ben Hutchings
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

There is a change in Linux 3.3, also intended to go into Linux 3.2.14,
which looks like a fix for bug #584881.

I'm attaching a backported version of this bug fix for Debian 6.0
'squeeze', which you may wish to test. You can build a kernel package
with this patch by following the instructions at
<http://kernel-handbook.alioth.debian.org/ch-common-tasks.html#s-common-official>.

Note that no further bug fixes will be made in Debian 5.0 'lenny'.

Ben.

--
Ben Hutchings
I'm always amazed by the number of people who take up solipsism because
they heard someone else explain it. - E*Borg on alt.fan.pratchett
 
Old 04-02-2012, 10:29 AM
Ian Jackson
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

Ben Hutchings writes ("Re: Lockups under heavy disk IO; md (RAID) resync/check implicated"):
> There is a change in Linux 3.3, also intended to go into Linux 3.2.14,
> which looks like a fix for bug #584881.

Thanks. I haven't experienced the bug in production in squeeze.
I will try to set up a test environment to repro and hopefully confirm
the fix.

Ian.



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20345.32678.837487.661064@chiark.greenend.org.uk"> http://lists.debian.org/20345.32678.837487.661064@chiark.greenend.org.uk
 
Old 04-02-2012, 11:09 AM
Tim Small
 
Default Bug#584881: Lockups under heavy disk IO; md (RAID) resync/check implicated

FWIW, I was only able to reproduce the problem which I was seeing on
lenny+openvz (running the same workload on lenny+chroot, or
squeeze+openvz didn't trigger it).

The fix you attached does sound like a plausible fix for the issue I was
seeing (having spent a day or two peering at the code and sprinkling
printks all over it, but not ultimately getting anywhere), but I don't
think I still have my test environment archived anywhere. If I remember
correctly, things always ended up deadlocking with 1 request sitting in
md's queue.

Tim.

--
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309




--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4F7988E6.3060506@seoss.co.uk">http://lists.debian.org/4F7988E6.3060506@seoss.co.uk
 

Thread Tools




All times are GMT. The time now is 08:15 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org