On 03/03/2011 07:32 PM, Brian Murray wrote:
> On Thu, Mar 03, 2011 at 05:29:17PM -0500, Peter M. Petrakis wrote:
>>
>>
>> On 03/03/2011 04:46 PM, Brian Murray wrote:
>>> In reviewing the kernel debugging documentation I came across the
>>> instructions for saving a crash dump for a kernel panic[1]. I tried
>>> following them and I'm unable to generate a crash dump. Additionally,
>>> that page hadn't been touched in quite some time. I've also found a bug
>>> report[2] about another person having an issue getting crash dumps.
>>>
>>> So my question is - is there something wrong with the documentation?
>>> the crash dump process? or me?
>>
>> It's not you, the system has to be rebooted first. The older
>> versions of the instructions say to reboot, but the 10.04
>> ones don't and neither does the "just install this package"
>> line at the top of the page.
>
> I rebooted multiple and tested it multiple times.
>

Well then we have a different problem. I recall recently
someone using Natty was also trying to get crashdump
working and found that they were getting a real panic
in the kexec kernel. Perhaps you're seeing the same thing?

>> This may have also been fixed in the meantime, I noticed
>> that /var/crash isn't being created, make sure that exists
>> after the reboot.
>
> /var/crash would not exist if you were running a stable release of
> Ubuntu as apport crash reporting is turned off then.

Huh? That's where the raw dump is supposed to be written, regardless
of whether it's a stable release or if apport is even installed.

See /usr/share/initramfs-tools/scripts/init-bottom/0_kdump

Examining the apport.conf script, there might be a bug in it.

start on runlevel [2345]
stop on runlevel [!2345]

env enabled=1

pre-start script
. /etc/default/apport
[ "$enabled" = "1" ] || [ "$force_start" = "1" ]

mkdir -p -m 1777 /var/crash
...

Is the local filesystem even mounted yet?

So we need more details from you. Are you actually able to kexec
into the crash kernel upon panic? How far does it get? Does it
hang forever? Please confirm you kernel cli args, it should look
something like this.

BOOT_IMAGE=/boot/vmlinuz-2.6.32-manz10-server root=UUID=8c0f013f-aa69-40fb-afc1-b6d6edc648d7 ro crashkernel=384M-2G:64M,2G-:128M console=tty0 console=ttyS0,115200n8

Finally, even if it is correctly configured, kernels executing
in a panic context aren't one size fits all. You may have to
start ejecting device drivers and be prepared for indepth
kernel debugging.

Peter

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

On 03/03/2011 07:34 PM, Brian Murray wrote:

On Thu, Mar 03, 2011 at 06:26:18PM -0500, Joseph Salisbury wrote:

On 03/03/2011 04:46 PM, Brian Murray wrote:

In reviewing the kernel debugging documentation I came across the
instructions for saving a crash dump for a kernel panic[1]. I tried
following them and I'm unable to generate a crash dump. Additionally,
that page hadn't been touched in quite some time. I've also found a bug
report[2] about another person having an issue getting crash dumps.

So my question is - is there something wrong with the documentation?
the crash dump process? or me?


[1] https://wiki.ubuntu.com/Kernel/CrashdumpRecipe
[2] https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/710733

Thanks,



I was able to generate a crash dump on a desktop running Natty.


How? Your last comment in the bug report mentions intermittent failures
and maxsize is no longer used by apport so that shouldn't have any
effect. I'd really like to get the documentation and or the process
squared away.



Hi Brian,

I performed some more testing today on a Natty desktop. I'm able to
generate a crash dump. However, I've been getting intermittent
failures. Like you mention, changing maxsize doesn't seem to help. It
was just a coincidence that crash dump worked for the first time, after
I increased maxsize. In all the failures, the system hangs performing
the following during the dump file creation:


"Copying data : [N%] <- The percentage when the hang happens varies.

I had to perform some steps in addition to what's listed on the
CrashdumpRecipe wiki. To get crash dump working(Although
intermittently), I performed the following:



1. Installed linux-crashdump and kdump-tools.
- Should it be necessary to install kdump-tools? Without kdump-tools,
I see the following in /var/crash/vmcore.log:


"/root/usr/bin/makedumpfile: error while loading shared libraries:
libdw.so.1: cannot open shared object file: No such file or directory"


- I noticed makedumpfile lives in /usr/bin/ and not /root/usr/bin.
- I tried creating a sym link in /root/usr/bin to point to the real
makedumpfile in /usr/bin, but I still got the same error.
- I performed an ldd on makedumpfile in /usr/bin, and all the libraries
where found.
- Again, I tried these things before I installed kdump-tools. Once
kdump-tools is installed, the lib load error goes away.


2. I manually created the /var/crash directory.

3. Edited /etc/default/apport; Changed enabled from 0 to 1.

4. Edited /etc/default/kdump-tools:
- Changed USE_KDUMP from 0 to 1.
- Uncommented: #KDUMP_SYSCTL="kernel.panic_on_oops=1"
- Without kdump-tools installed, this file doesn't exist.

5. Edited /etc/default/kexec. Changed LOAD_KEXEC from false to true,
but this didn't seem to make a difference.


6. Removed 'quiet splash' from the boot parameters(So I could see where
it was hanging).



To trigger a panic, I perform:
echo c | sudo tee /proc/sysrq-trigger

I'll keep experimenting, and see if I can provide some more information.
I'll also update the bug with this latest info.


Thanks,

Joe


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team