FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 03-28-2012, 09:07 AM
Lars Skovlund
 
Default Pinpointing faulty kernel driver?

Hi all

When upgrading kernel series 3.1.0 to series 3.2.0, suddenly my system
won't boot. It hangs with the message

Waiting for /dev to be fully populated

I read the manual and found out how to drop into the initramfs debug
shell, but I am unsure how to proceed from here. I guess the problem
could be a faulty driver that I need to disable, but I know of no
(efficient) way to find out which one. Ideally, I would load one
driver at a time, and see when the problem occurs.

Can anyone help? Or am I completely off track with this problem?

I am running the latest packages in both kernel series, and the 3.1
series does not have the problem.

Regards

Lars


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120328090715.GA4262@starfish.lan">http://lists.debian.org/20120328090715.GA4262@starfish.lan
 
Old 03-28-2012, 04:14 PM
Camaleón
 
Default Pinpointing faulty kernel driver?

On Wed, 28 Mar 2012 11:07:15 +0200, Lars Skovlund wrote:

> When upgrading kernel series 3.1.0 to series 3.2.0, suddenly my system
> won't boot. It hangs with the message
>
> Waiting for /dev to be fully populated

But the old working kernel is still available for you boot with, right?

> I read the manual and found out how to drop into the initramfs debug
> shell, but I am unsure how to proceed from here. I guess the problem
> could be a faulty driver that I need to disable, but I know of no
> (efficient) way to find out which one. Ideally, I would load one driver
> at a time, and see when the problem occurs.
>
> Can anyone help? Or am I completely off track with this problem?

Can you take a snapshot of the screen and upload somewhere? Maybe there
is additional info alongside the "Waiting for /dev to be fully populated"
message that can provide any insightful information about the source of
the problem.

> I am running the latest packages in both kernel series, and the 3.1
> series does not have the problem.

Are you using any special kernel module that may require to be re-
compiled (as for example, the VGA driver) or any special module for the
hard disk controller?

Some hints for debugging this can be found here:

http://wiki.debian.org/InitramfsDebug

Greetings,

--
Camaleón


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: jkvdcd$hqr$13@dough.gmane.org">http://lists.debian.org/jkvdcd$hqr$13@dough.gmane.org
 
Old 03-29-2012, 10:08 PM
Arnt Karlsen
 
Default Pinpointing faulty kernel driver?

On Wed, 28 Mar 2012 11:07:15 +0200, Lars wrote in message
<20120328090715.GA4262@starfish.lan>:

> Hi all
>
> When upgrading kernel series 3.1.0 to series 3.2.0, suddenly my system
> won't boot. It hangs with the message
>
> Waiting for /dev to be fully populated

..me 2, upgrading from 3.2.0-1-rt-amd64 to 3.2.0-2-rt-amd64:
arnt@celsius:~/FG-git$ dpkg -l |grep linux-image
ii linux-image-3.2.0-1-amd64 3.2.4-1 Linux 3.2 for 64-bit PCs
ii linux-image-3.2.0-1-rt-amd64 3.2.4-1 Linux 3.2 for 64-bit PCs,
PREEMPT_RT
ii linux-image-3.2.0-2-amd64 3.2.12-1 Linux 3.2 for 64-bit PCs
ii linux-image-3.2.0-2-rt-amd64 3.2.12-1 Linux 3.2 for 64-bit PCs,
PREEMPT_RT
arnt@celsius:~/FG-git$ uname -a
Linux celsius 3.2.0-1-rt-amd64 #1 SMP PREEMPT RT Sun Feb 5 15:51:02 UTC
2012 x86_64 GNU/Linux
arnt@celsius:~/FG-git$

> I read the manual and found out how to drop into the initramfs debug
> shell, but I am unsure how to proceed from here. I guess the problem
> could be a faulty driver that I need to disable, but I know of no
> (efficient) way to find out which one. Ideally, I would load one
> driver at a time, and see when the problem occurs.
>
> Can anyone help? Or am I completely off track with this problem?
>
> I am running the latest packages in both kernel series, and the 3.1
> series does not have the problem.
>
> Regards
>
> Lars
>
>


--
..med vennlig hilsen = with Kind Regards from Arnt Karlsen
...with a number of polar bear hunters in his ancestry...
Scenarios always come in sets of three:
best case, worst case, and just in case.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120330000830.02451e8a@nb6.lan">http://lists.debian.org/20120330000830.02451e8a@nb6.lan
 
Old 03-30-2012, 06:55 PM
Lars Skovlund
 
Default Pinpointing faulty kernel driver?

On Fri, Mar 30, 2012 at 12:08:30AM +0200, Arnt Karlsen wrote:
> On Wed, 28 Mar 2012 11:07:15 +0200, Lars wrote in message
> <20120328090715.GA4262@starfish.lan>:
>
> > Hi all
> >
> > When upgrading kernel series 3.1.0 to series 3.2.0, suddenly my system
> > won't boot. It hangs with the message
> >
> > Waiting for /dev to be fully populated
>
> ..me 2, upgrading from 3.2.0-1-rt-amd64 to 3.2.0-2-rt-amd64:

Don't know if this is the same bug. I had 3.2.0-1 as well at first, and it
did not work. Hmm, I notice you're running amd64 as well.

/Lars


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120330185529.GA4231@starfish.lan">http://lists.debian.org/20120330185529.GA4231@starfish.lan
 
Old 04-04-2012, 07:19 PM
Lars Skovlund
 
Default Pinpointing faulty kernel driver?

Sorry for the delay, I am rather busy at the moment.

On Wed, Mar 28, 2012 at 04:14:05PM +0000, Camaleón wrote:
> On Wed, 28 Mar 2012 11:07:15 +0200, Lars Skovlund wrote:
>
> > When upgrading kernel series 3.1.0 to series 3.2.0, suddenly my system
> > won't boot. It hangs with the message
> >
> > Waiting for /dev to be fully populated
>
> But the old working kernel is still available for you boot with, right?

Yes. In fact, I'm using it right now.

>
> > I read the manual and found out how to drop into the initramfs debug
> > shell, but I am unsure how to proceed from here. I guess the problem
> > could be a faulty driver that I need to disable, but I know of no
> > (efficient) way to find out which one. Ideally, I would load one driver
> > at a time, and see when the problem occurs.
> >
> > Can anyone help? Or am I completely off track with this problem?
>
> Can you take a snapshot of the screen and upload somewhere? Maybe there
> is additional info alongside the "Waiting for /dev to be fully populated"
> message that can provide any insightful information about the source of
> the problem.

I realize this complicates matters a lot, but the exact crash points
are different each time. So I need to somehow get the log information
out in text format and put it somewhere. I had the idea of getting
udev to spit out copious amounts of debug information. The interesting
thing is when I do this, 3.1.x seems to get stuck for a while waiting
for timeouts. It does, however, still boot - unlike 3.2.x.

Any ideas as to how I can get the information out without resorting to
screendumps?

> > I am running the latest packages in both kernel series, and the 3.1
> > series does not have the problem.
>
> Are you using any special kernel module that may require to be re-
> compiled (as for example, the VGA driver) or any special module for the
> hard disk controller?

My list of loaded modules follows (taken from 3.1.x). I have had some
complaints about missing wifi firmware for a while (this is a desktop
box, so I don't care about it) even though I have both
linux-firmware-(non)free packages installed.

Your help is very much appreciated.

/Lars

Module Size Used by
snd_hrtimer 12604 1
acpi_cpufreq 12935 1
mperf 12453 1 acpi_cpufreq
cpufreq_conservative 13147 0
cpufreq_powersave 12454 0
cpufreq_userspace 12576 0
cpufreq_stats 12866 0
parport_pc 22364 0
ppdev 12763 0
lp 17149 0
parport 31858 3 parport_pc,ppdev,lp
bnep 17567 2
rfcomm 33622 0
bluetooth 119290 10 bnep,rfcomm
binfmt_misc 12957 1
fuse 61981 1
nfsd 259717 2
nfs 312135 0
lockd 67328 2 nfsd,nfs
fscache 36739 1 nfs
auth_rpcgss 37143 2 nfsd,nfs
nfs_acl 12511 2 nfsd,nfs
sunrpc 173516 6 nfsd,nfs,lockd,auth_rpcgss,nfs_acl
ext2 59191 1
loop 22597 0
snd_hda_codec_hdmi 26548 4
arc4 12458 2
snd_hda_codec_idt 53748 1
rt2800pci 13908 0
rt2800lib 43568 1 rt2800pci
snd_hda_intel 26182 1
snd_hda_codec 72920 3 snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel
snd_hwdep 13186 1 snd_hda_codec
snd_pcm_oss 41081 0
snd_mixer_oss 17916 1 snd_pcm_oss
snd_pcm 63744 4 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd _pcm_oss
evdev 17562 4
crc_ccitt 12347 1 rt2800lib
rt2x00pci 12847 1 rt2800pci
rt2x00lib 38202 3 rt2800pci,rt2800lib,rt2x00pci
nouveau 521365 2
joydev 17266 0
snd_seq_midi 12848 0
mac80211 183047 3 rt2800lib,rt2x00pci,rt2x00lib
snd_rawmidi 23060 1 snd_seq_midi
snd_seq_midi_event 13316 1 snd_seq_midi
snd_seq 45093 3 snd_seq_midi,snd_seq_midi_event
cfg80211 132703 2 rt2x00lib,mac80211
psmouse 55543 0
pcspkr 12579 0
i2c_i801 16870 0
snd_timer 22917 3 snd_hrtimer,snd_pcm,snd_seq
serio_raw 12850 0
snd_seq_device 13176 3 snd_seq_midi,snd_rawmidi,snd_seq
iTCO_wdt 17081 0
iTCO_vendor_support 12704 1 iTCO_wdt
rfkill 19012 3 bluetooth,cfg80211
snd 52798 15 snd_hda_codec_hdmi,snd_hda_codec_idt,snd_hda_intel ,snd_hda_codec,snd_hwdep,snd_pcm_oss,snd_mixer_oss ,snd_pcm,snd_rawmidi,snd_seq,snd_timer,snd_seq_dev ice
eeprom_93cx6 12455 1 rt2800pci
ttm 48725 1 nouveau
drm_kms_helper 27227 1 nouveau
drm 167371 4 nouveau,ttm,drm_kms_helper
i2c_algo_bit 12841 1 nouveau
i2c_core 23876 5 nouveau,i2c_i801,drm_kms_helper,drm,i2c_algo_bit
soundcore 13065 1 snd
snd_page_alloc 13003 2 snd_hda_intel,snd_pcm
mxm_wmi 12473 1 nouveau
wmi 13243 1 mxm_wmi
video 17628 1 nouveau
button 12937 1 nouveau
processor 27949 1 acpi_cpufreq
thermal_sys 18040 2 video,processor
ext4 312988 4
mbcache 13065 2 ext2,ext4
jbd2 62015 1 ext4
crc16 12343 2 bluetooth,ext4
usb_storage 43919 0
uas 13296 0
usbhid 36379 0
hid 77192 1 usbhid
sr_mod 21899 0
cdrom 35401 1 sr_mod
sd_mod 36136 6
crc_t10dif 12348 1 sd_mod
ahci 24997 5
libahci 22860 1 ahci
libata 140545 2 ahci,libahci
scsi_mod 162376 5 usb_storage,uas,sr_mod,sd_mod,libata
xhci_hcd 64215 0
r8169 42391 0
mii 12675 1 r8169
ehci_hcd 40215 0
usbcore 124095 6 usb_storage,uas,usbhid,xhci_hcd,ehci_hcd


>
> Some hints for debugging this can be found here:
>
> http://wiki.debian.org/InitramfsDebug
>
> Greetings,
>
> --
> Camaleón
>
>
> --
> To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
> Archive: http://lists.debian.org/jkvdcd$hqr$13@dough.gmane.org
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20120404191921.GA3360@starfish.lan">http://lists.debian.org/20120404191921.GA3360@starfish.lan
 
Old 04-05-2012, 02:12 PM
Camaleón
 
Default Pinpointing faulty kernel driver?

On Wed, 04 Apr 2012 21:19:21 +0200, Lars Skovlund wrote:

> Sorry for the delay, I am rather busy at the moment.

No problem.

> On Wed, Mar 28, 2012 at 04:14:05PM +0000, Camaleón wrote:

(...)

>> > I read the manual and found out how to drop into the initramfs debug
>> > shell, but I am unsure how to proceed from here. I guess the problem
>> > could be a faulty driver that I need to disable, but I know of no
>> > (efficient) way to find out which one. Ideally, I would load one
>> > driver at a time, and see when the problem occurs.
>> >
>> > Can anyone help? Or am I completely off track with this problem?
>>
>> Can you take a snapshot of the screen and upload somewhere? Maybe there
>> is additional info alongside the "Waiting for /dev to be fully
>> populated" message that can provide any insightful information about
>> the source of the problem.
>
> I realize this complicates matters a lot, but the exact crash points are
> different each time. So I need to somehow get the log information out in
> text format and put it somewhere. I had the idea of getting udev to spit
> out copious amounts of debug information. The interesting thing is when
> I do this, 3.1.x seems to get stuck for a while waiting for timeouts. It
> does, however, still boot - unlike 3.2.x.
>
> Any ideas as to how I can get the information out without resorting to
> screendumps?

You can send the output to another machine using a serial cable and
instructing the kernel to dump the message there. I had to do this years
ago to debug a kernel crash from a VM but to be sincere, I doubt I can
nowadays repeat that milestone, I don't remember the steps :-)

Look, Ubuntu has some good doc about this:

Kernel Debugging Tricks
https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks

The easiest way (which will allow you to read the screen) would be
slowing the kernel output messages, I would start from there.

>> > I am running the latest packages in both kernel series, and the 3.1
>> > series does not have the problem.
>>
>> Are you using any special kernel module that may require to be re-
>> compiled (as for example, the VGA driver) or any special module for the
>> hard disk controller?
>
> My list of loaded modules follows (taken from 3.1.x). I have had some
> complaints about missing wifi firmware for a while (this is a desktop
> box, so I don't care about it) even though I have both
> linux-firmware-(non)free packages installed.

A message warning for a missing firmware could render that device
inoperable but nothing more.

> Your help is very much appreciated.

(...)

Thanks for the module listing :-)

It seems that you're using a usual SATA configuration at the BIOS (ahci),
I mean, nothing "fancy" that may require for specific hard disk
controller module to be present and thus preventing the system from
booting normally.

Anyway, you can check if by appending "rootdelay=9" to the kernel line at
GRUB's boot menu you can get rid of the "waiting... to be fully
populated" message.

Greetings,

--
Camaleón


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: jlk988$mrm$11@dough.gmane.org">http://lists.debian.org/jlk988$mrm$11@dough.gmane.org
 
Old 04-05-2012, 08:40 PM
 
Default Pinpointing faulty kernel driver?

On Apr 5, 2012 16:12 "Camaleón" <noelamac@gmail.com> wrote:



> You can send the output to another machine using a serial cable and
instructing the kernel to dump the message there. I had to do this years
> ago to debug a kernel crash from a VM but to be sincere, I doubt I can
nowadays repeat that milestone, I don't remember the steps :-)
>
> Look, Ubuntu has some good doc about this:
>
> Kernel Debugging Tricks
> <https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks>
>
>

Thank you for this reference! It is indeed an awesome reference. I tried
getting netconsole to work, because I don't have a serial-to-USB cable. I
will be looking into this problem over the next few days. Perhaps I will
have to buy such a cable.

There is also, according to the InitramfsDebug Debian wiki page, a way to
get log data in /dev/.initramfs/initramfs.debug. I can not make this work
either, for some reason.

I tried the boot_delay option, but the delay seems to revert to full speed
at a certain point in the boot sequence, so it is no use.
There is odd behavior when the crash happens: The visible screen area
seems to "scroll back" so that when the hang occurs, the timestamps show
approximately 64.xxxxxx seconds (but actually, the crash occurs at
68.xxxxxx seconds).

On a hunch, I tried setting acpi=off - and now the new kernel boots! Of
course, this is a sub-optimal solution, so I got dmesg listings from both
the old and the new kernel.

The diff is here: http://pastebin.com/L4YXTmJh

I would be very grateful if you'd take a look.

Thanks,

LarsOn Apr 5, 2012 16:12 "Camaleón" <noelamac@gmail.com> wrote:


You can send the output to another machine using a serial cable and
instructing the kernel to dump the message there. I had to do this years
ago to debug a kernel crash from a VM but to be sincere, I doubt I can
nowadays repeat that milestone, I don't remember the steps :-)

Look, Ubuntu has some good doc about this:

Kernel Debugging Tricks
https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks

Thank you for this reference! It is indeed an awesome reference. I tried getting netconsole to work, because I don't have a serial-to-USB cable. I will be looking into this problem over the next few days.

There is also, according to the InitramfsDebug Debian wiki page, a way to get log data in /dev/.initramfs/initramfs.debug. I can not make this work either, for some reason.

The easiest way (which will allow you to read the screen) would be
slowing the kernel output messages, I would start from there.
I am running the latest packages in both kernel series, and the 3.1
series does not have the problem.
Are you using any special kernel module that may require to be re-
compiled (as for example, the VGA driver) or any special module for the
hard disk controller?
My list of loaded modules follows (taken from 3.1.x). I have had some
complaints about missing wifi firmware for a while (this is a desktop
box, so I don't care about it) even though I have both
linux-firmware-(non)free packages installed.
A message warning for a missing firmware could render that device
inoperable but nothing more.
Your help is very much appreciated.
(...)

Thanks for the module listing :-)

It seems that you're using a usual SATA configuration at the BIOS (ahci),
I mean, nothing "fancy" that may require for specific hard disk
controller module to be present and thus preventing the system from
booting normally.

Anyway, you can check if by appending "rootdelay=9" to the kernel line at
GRUB's boot menu you can get rid of the "waiting... to be fully
populated" message.

Greetings,
 
Old 04-06-2012, 03:08 PM
Camaleón
 
Default Pinpointing faulty kernel driver?

On Thu, 05 Apr 2012 22:40:15 +0200, lskovlun wrote:

> On Apr 5, 2012 16:12 "Camaleón" <noelamac@gmail.com> wrote:
>
>
>
>> You can send the output to another machine using a serial cable and
> instructing the kernel to dump the message there. I had to do this years
>> ago to debug a kernel crash from a VM but to be sincere, I doubt I can
> nowadays repeat that milestone, I don't remember the steps :-)
>>
>> Look, Ubuntu has some good doc about this:
>>
>> Kernel Debugging Tricks
>> <https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks>
>>
>>
>>
> Thank you for this reference! It is indeed an awesome reference. I tried
> getting netconsole to work, because I don't have a serial-to-USB cable.
> I will be looking into this problem over the next few days. Perhaps I
> will have to buy such a cable.
>
> There is also, according to the InitramfsDebug Debian wiki page, a way
> to get log data in /dev/.initramfs/initramfs.debug. I can not make this
> work either, for some reason.
>
> I tried the boot_delay option, but the delay seems to revert to full
> speed at a certain point in the boot sequence, so it is no use.

To get verbose logs, remember that you have to remove "quiet" from the kernel
line, so based in your logs below, it should be something like:

vmlinuz-3.1.0-1-amd64 root=UUID=b34744e5-2c76-44f3-a7b1-a2fed3ec430e boot_delay=1000

> There is odd behavior when the crash happens: The visible screen area seems
> to "scroll back" so that when the hang occurs, the timestamps show
> approximately 64.xxxxxx seconds (but actually, the crash occurs at
> 68.xxxxxx seconds).
>
> On a hunch, I tried setting acpi=off - and now the new kernel boots!

That can be indeed relevant but I can't interpret the whole meaning nor
getting and idea of what -which acpi enables- is not of the liking to the
updated kernel.

I would open a bug report in Debian explaing the issue and adding this
information with the by-pass you've found. Kernel hackers will help you
to diagnose the problem.

> Of course, this is a sub-optimal solution, so I got dmesg listings from
> both the old and the new kernel.

Have you tried with "rootdelay=9"? This was something recommended for Squeeze
when running specific hard disk layout or hardware but it can also help here.

> The diff is here: http://pastebin.com/L4YXTmJh
>
> I would be very grateful if you'd take a look.

I'll do, but that goes beyond my knowledge :-)

There are many differences between the two boot logs, that's normal but it
seems the nouveu driver is completely missing when you boot with no ACPI, is
that right? What VGA driver is loaded?

Additional tips you can read (and experiment with) here:

Debug: How to Isolate Linux ACPI Issues
http://www.lesswatts.org/projects/acpi/debug.php

Greetings,

--
Camaleón


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: jln0si$j6g$10@dough.gmane.org">http://lists.debian.org/jln0si$j6g$10@dough.gmane.org
 

Thread Tools




All times are GMT. The time now is 12:44 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org