Linux Archive
Page 1 of 2 1 2 >
Show 40 post(s) from this thread on one page

Linux Archive (http://www.linux-archive.org/)
-   Crash Utility (http://www.linux-archive.org/crash-utility/)
-   -   crash tool not working (http://www.linux-archive.org/crash-utility/629492-crash-tool-not-working.html)

Adil Mujeeb 02-06-2012 10:03 AM
crash tool not working
 
Hi List,

I am new to crash / kdump tool and failing some problem as mentioned below.

I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/)
and URL http://www.dedoimedo.com/computers/crash.html.
I am building modified kernel source (2.6.32 based) and added my
modules (for study purpose). Building, installing and booting kernel
is successful. I have enabled the options for kdump as mentioned in
the book:

Enable Kexec system call:
CONFIG_KEXEC=y

Enable kernel crash dumps:
CONFIG_CRASH_DUMP=y

Optional: Disable Symmetric Multi-Processing (SMP) support
CONFIG_SMP=y

Enable sysfs file system support:
CONFIG_SYSFS=y

Enable /proc/vmcore support:
CONFIG_PROC_VMCORE=y

Configure the kernel with debug info:
CONFIG_DEBUG_INFO=y

Configure the start section for reserved RAM for the crash kernel:
CONFIG_PHYSICAL_START=0x200000 (2MB)

Configure kdump kernel so it can be identified:
CONFIG_LOCALVERSION="-crash"


Kdump configuration /etc/sysconfig/kdump:
KDUMP_KERNELVER=""

KDUMP_COMMANDLINE=""

KDUMP_COMMANDLINE_APPEND="maxcpus=1 "

KEXEC_OPTIONS=""

KDUMP_IMMEDIATE_REBOOT="yes"

KDUMP_TRANSFER=""

KDUMP_SAVEDIR="file:///var/crash"

KDUMP_KEEP_OLD_DUMPS="5"

KDUMP_FREE_DISK_SIZE="64"

KDUMP_VERBOSE="3"

KDUMP_DUMPLEVEL="0"

KDUMP_DUMPFORMAT="compressed"

There is no option KDUMP_DUMPDEV option
There is no option KDUMP_RUNLEVEL

I booted successfully with this kernel and tried to crash it by
module. After rebooting, I found that vmcore is generated under
/var/crash/ But I am not able to analyze it with crash command.

linux:/home/adil # cat /proc/cmdline
root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1
resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2
splash=silent crashkernel=256M-:128M vga=0x31a
linux:/home/adil #

linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
/boot/vmlinuz-2.6.32.12-crash-crash

crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format

Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]

Enter "crash -h" for details.
linux:/home/adil #

The URL http://www.dedoimedo.com/computers/crash.html mentioned that
"The newer versions of Kdump can work with compressed kernel images.
Furthermore, they copy the System map file and the kernel image into
the crash directory, making the use of crash utility somewhat
simpler."

linux:/home/adil # ls -al /var/crash/2012-01-30-18:08/
total 1335536
drwxr-xr-x 2 root root 4096 2012-01-30 18:13 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt
-rw-r--r-- 1 root root 1716605 2012-01-30 18:13
System.map-2.6.32.12-0.7-default
-rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore
-rw-r--r-- 1 root root 3774506 2012-01-30 18:13
vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil # ls -al /var/crash/2012-01-31-12:17/
total 1343860
drwxr-xr-x 2 root root 4096 2012-01-31 12:24 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt
-rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore
linux:/home/adil #

linux:/home/adil # ls /boot/
backup_mbr
boot
boot.readme
config-2.6.32.12-0.7-default
config-2.6.32.12-0.7-xen
grub
initrd
initrd-2.6.32.12-0.7-default
initrd-2.6.32.12-0.7-default-kdump
initrd-2.6.32.12-0.7-xen
initrd-2.6.32.12-crash-crash
initrd-2.6.32.12-crash-crash-kdump
initrd-xen
message
symsets-2.6.32.12-0.7-default.tar.gz
symtypes-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-xen.gz
System.map-2.6.32.12-0.7-default
System.map-2.6.32.12-0.7-xen
System.map-2.6.32.12-crash-crash
vmlinux-2.6.32.12-0.7-xen.gz
vmlinuz
vmlinuz-2.6.32.12-0.7-default
vmlinuz-2.6.32.12-0.7-xen
vmlinuz-2.6.32.12-crash-crash
vmlinuz-xen
vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil #


Another observation is boot.kdump seems to on but manually start
giving me error:

linux:/home/adil # chkconfig boot.kdump
boot.kdump on
linux:/home/adil #

linux:/home/adil # /etc/init.d/boot.kdump start
Loading kdump
Regenerating kdump initrd ...
Can't find kernel text map area from kcore
Cannot load /boot/vmlinuz-2.6.32.12-crash-crash

failed
linux:/home/adil #

Other Query: Following is not clear mentioned in the book under
"section 11.2 Crash (capture) kernel":
--------------
This means that while your production kernels will most likely be
named vmlinuz, the Kdump crash kernels need to be uncompressed,
hence named vmlinux, or rather vmlinux-kdump.
---------------

Please help how to correctly setup and use crash on my machine.

Thank you,
Adil

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

"Suzuki K. Poulose" 02-06-2012 01:58 PM
crash tool not working
 
linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
/boot/vmlinuz-2.6.32.12-crash-crash

crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation

[...]

This program has absolutely no warranty. Enter "help warranty" for details.

crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format


First and formeost you should pass the vmcore file.

Thanks
Suzuki

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-06-2012 02:00 PM
crash tool not working
 
----- Original Message -----
> Hi List,
>
> I am new to crash / kdump tool and failing some problem as mentioned
> below.
>
> I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/)
> and URL http://www.dedoimedo.com/computers/crash.html.

Well, that's your first mistake...

The section in that book that has caused your confusion is this part:

| You can use the old way. Here's an example on CentOS 5.4:
|
| crash /boot/System.map-2.6.18-164.10.1.el5 /boot/vmlinuz-2.6.18-164.10.1.el5 vmcore
|
| Notice the use of vmlinuz kernel image, as opposed to vmlinux previously required.

Given CentOS 5.4 is a copy of RHEL5.4, I cannot even imagine where the
author got the impression that the /boot/vmlinuz-<release> file
could *ever* have been used by the crash utility. The vmlinuz file is
a stripped and compressed output from the vmlinux file, and is completely
unsuitable for use by the crash utility. His statement above is just
complete fanatasy. The crash utility needs an unstripped vmlinux file
containing the debuginfo sections for the base kernel. He does subsequently
state that "alternatively" you can use vmlinux file:

| Alternatively, you can use only the debug information under /usr/lib/debug. The
| information is extracted during the installation of kernel-debuginfo packages
| matching the kernel that was running at the time of the kernel crash.

But in reality -- that is the only file that can be used with the
crash utility, at least from a RHEL5/CentOS perspective. I can't
speak for any other distribution.

> I am building modified kernel source (2.6.32 based) and added my
> modules (for study purpose). Building, installing and booting kernel
> is successful. I have enabled the options for kdump as mentioned in
> the book:
>
> Enable Kexec system call:
> CONFIG_KEXEC=y
>
> Enable kernel crash dumps:
> CONFIG_CRASH_DUMP=y
>
> Optional: Disable Symmetric Multi-Processing (SMP) support
> CONFIG_SMP=y
>
> Enable sysfs file system support:
> CONFIG_SYSFS=y
>
> Enable /proc/vmcore support:
> CONFIG_PROC_VMCORE=y
>
> Configure the kernel with debug info:
> CONFIG_DEBUG_INFO=y
>
> Configure the start section for reserved RAM for the crash kernel:
> CONFIG_PHYSICAL_START=0x200000 (2MB)
>
> Configure kdump kernel so it can be identified:
> CONFIG_LOCALVERSION="-crash"
>
>
> Kdump configuration /etc/sysconfig/kdump:
> KDUMP_KERNELVER=""
>
> KDUMP_COMMANDLINE=""
>
> KDUMP_COMMANDLINE_APPEND="maxcpus=1 "
>
> KEXEC_OPTIONS=""
>
> KDUMP_IMMEDIATE_REBOOT="yes"
>
> KDUMP_TRANSFER=""
>
> KDUMP_SAVEDIR="file:///var/crash"
>
> KDUMP_KEEP_OLD_DUMPS="5"
>
> KDUMP_FREE_DISK_SIZE="64"
>
> KDUMP_VERBOSE="3"
>
> KDUMP_DUMPLEVEL="0"
>
> KDUMP_DUMPFORMAT="compressed"
>
> There is no option KDUMP_DUMPDEV option
> There is no option KDUMP_RUNLEVEL
>
> I booted successfully with this kernel and tried to crash it by
> module. After rebooting, I found that vmcore is generated under
> /var/crash/ But I am not able to analyze it with crash command.
>
> linux:/home/adil # cat /proc/cmdline
> root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1
> resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2
> splash=silent crashkernel=256M-:128M vga=0x31a
> linux:/home/adil #
>
> linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
> /boot/vmlinuz-2.6.32.12-crash-crash
>
> crash 5.0.1
> Copyright (C) 2002-2010 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
>
> Usage:
> crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
> [dumpfile]
>
> Enter "crash -h" for details.
> linux:/home/adil #

Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
file is not a supported file format. Never has been, never will be...

>
> The URL http://www.dedoimedo.com/computers/crash.html mentioned that
> "The newer versions of Kdump can work with compressed kernel images.
> Furthermore, they copy the System map file and the kernel image into
> the crash directory, making the use of crash utility somewhat
> simpler."

What has been supported since crash-5.1.3 is the usage of
vmlinux files that have been subsequently compressed with either
gzip or bzip2. But again, the /boot/vmlinuz-<release> file is
a completely different animal.

> linux:/home/adil # ls -al /var/crash/2012-01-30-18:08/
> total 1335536
> drwxr-xr-x 2 root root 4096 2012-01-30 18:13 .
> drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
> -rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt
> -rw-r--r-- 1 root root 1716605 2012-01-30 18:13 System.map-2.6.32.12-0.7-default
> -rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore
> -rw-r--r-- 1 root root 3774506 2012-01-30 18:13 vmlinux-2.6.32.12-0.7-default.gz
> linux:/home/adil # ls -al /var/crash/2012-01-31-12:17/
> total 1343860
> drwxr-xr-x 2 root root 4096 2012-01-31 12:24 .
> drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
> -rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt
> -rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore
> linux:/home/adil #
>
> linux:/home/adil # ls /boot/
> backup_mbr
> boot
> boot.readme
> config-2.6.32.12-0.7-default
> config-2.6.32.12-0.7-xen
> grub
> initrd
> initrd-2.6.32.12-0.7-default
> initrd-2.6.32.12-0.7-default-kdump
> initrd-2.6.32.12-0.7-xen
> initrd-2.6.32.12-crash-crash
> initrd-2.6.32.12-crash-crash-kdump
> initrd-xen
> message
> symsets-2.6.32.12-0.7-default.tar.gz
> symtypes-2.6.32.12-0.7-default.gz
> symvers-2.6.32.12-0.7-default.gz
> symvers-2.6.32.12-0.7-xen.gz
> System.map-2.6.32.12-0.7-default
> System.map-2.6.32.12-0.7-xen
> System.map-2.6.32.12-crash-crash
> vmlinux-2.6.32.12-0.7-xen.gz
> vmlinuz
> vmlinuz-2.6.32.12-0.7-default
> vmlinuz-2.6.32.12-0.7-xen
> vmlinuz-2.6.32.12-crash-crash
> vmlinuz-xen
> vmlinux-2.6.32.12-0.7-default.gz
> linux:/home/adil #
>
>
> Another observation is boot.kdump seems to on but manually start
> giving me error:
>
> linux:/home/adil # chkconfig boot.kdump
> boot.kdump on
> linux:/home/adil #
>
> linux:/home/adil # /etc/init.d/boot.kdump start
> Loading kdump
> Regenerating kdump initrd ...
> Can't find kernel text map area from kcore
> Cannot load /boot/vmlinuz-2.6.32.12-crash-crash
>
> failed
> linux:/home/adil #
>
> Other Query: Following is not clear mentioned in the book under
> "section 11.2 Crash (capture) kernel":
> --------------
> This means that while your production kernels will most likely be
> named vmlinuz, the Kdump crash kernels need to be uncompressed,
> hence named vmlinux, or rather vmlinux-kdump.
> ---------------
>
> Please help how to correctly setup and use crash on my machine.

With respect to the crash utility, if your kernel build procedure
has created a /usr/lib/debug/lib/modules/<kernel-version>/vmlinux,
then you don't need a System.map file, but rather:

$ crash /usr/lib/debug/lib/modules/<kernel-version>/vmlinux vmcore

If you don't have the file above, and if whatever non-RHEL system you're
running actually puts the vmlinux-<version>.gz file in the /var/crash
subdirectory with the vmcore, then use that file:

$ crash vmlinux-<kernel-version>.gz vmcore

You may also need the System.map file found there as well. I believe
Suse post-builds their debuginfo kernels instead of saving the vmlinux
file from when it was originally built. Could be that their version
of kexec-tools does the copy of the extra files into /var/crash?
I really don't know -- in a RHEL system, only the vmcore files are
copied there.

Anyway, if you have neither of the above, then check your modified kernel
build tree. In the top-level directory, there should be a "vmlinux" file,
which is the file that the crash utility requires. This also presumes that
you built the kernel with -g, which would get turned on when the kernel
is configured with CONFIG_DEBUG_INFO. However, it should be noted
that if your kernel build procedure built more than one kernel "flavor",
then the vmlinux file found there will be that of the last kernel that
was built -- which may not be the one you need.

For example, RHEL5 x86_64 builds three x86_64 kernel rpms:

kernel-2.6.18-232.el5.x86_64.rpm
kernel-debug-2.6.18-232.el5.x86_64.rpm
kernel-xen-2.6.18-232.el5.x86_64.rpm

The rpm files above contain the /boot/vmlinuz-<release> files.

Then, each kernel above has a debuginfo package associated
with it, which contains its original unstripped vmlinux file,
which gets put in /usr/lib/debug/lib/modules/<kernel-version>/vmlinux:

kernel-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-debug-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-xen-debuginfo-2.6.18-232.el5.x86_64.rpm

And there is a "common" debuginfo file that is typically installed
with the kernel-specific file above:

kernel-debuginfo-common-2.6.18-232.el5.x86_64.rpm

Again, I can't speak for the procedures used by any other distribution.

Also, do yourself a favor, and upgrade your crash utility version
from crash-5.0.1, which is two years old.

And lastly, although you may get some responses, this is not the list for
kdump-related questions. That is found here:

http://lists.infradead.org/mailman/listinfo/kexec

Dave Anderson

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Adil Mujeeb 02-09-2012 05:29 AM
crash tool not working
 
Hi,

>
> Well, that's your first mistake...
>

Thanks for pointing it out :)
..
[snip]
..
> Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
> file is not a supported file format. Never has been, never will be...
>

I tried with vmlinux and it seems ok. But got another error
hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
/var/crash/2012-02-08-14:13/vmcore

--------------------
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: seek error: kernel virtual address: ffffffff81827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
crash: seek error: kernel virtual address: ffffffff8181a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil #
----------------------------------


> Also, do yourself a favor, and upgrade your crash utility version
> from crash-5.0.1, which is two years old.

I am using for learning purpose. So i guess this version should be Ok
to use for learning purpose.

>
> And lastly, although you may get some responses, this is not the list
> for
> kdump-related questions. That is found here:
>
> http://lists.infradead.org/mailman/listinfo/kexec

Thank. I will ask the list for my kexec/kdump queries.

Thanks,
Adil
>
> Dave Anderson

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-09-2012 01:46 PM
crash tool not working
 
----- Original Message -----
> Hi,
>
> >
> > Well, that's your first mistake...
> >
>
> Thanks for pointing it out :)
> ..
> [snip]
> ..
> > Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
> > file is not a supported file format. Never has been, never will
> > be...
> >
>
> I tried with vmlinux and it seems ok. But got another error
> hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
> /var/crash/2012-02-08-14:13/vmcore
>
> --------------------
> crash 5.0.1
> Copyright (C) 2002-2010 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: seek error: kernel virtual address: ffffffff81827aa0 type: "kernel_config_data"
> WARNING: cannot read kernel_config_data
> crash: seek error: kernel virtual address: ffffffff8181a660 type: "cpu_possible_mask"
> hltncra110731:/home/adil #
> ----------------------------------
>
>
> > Also, do yourself a favor, and upgrade your crash utility version
> > from crash-5.0.1, which is two years old.
>
> I am using for learning purpose. So i guess this version should be Ok
> to use for learning purpose.

No, that's really not true. crash-5.0.1 is two years old, which is ancient
by crash standards. And it's very easy to upgrade:

$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
...
$ cd crash-6.0.3
$ make
...
$

As far as the "seek errors" you're seeing, I don't know what's happening,
other than crash can't find the physical address page associated with those
unity-mapped kernel virtual addresses in the vmcore. But I'm sorry -- I don't
spend time debugging issues encountered with obsolete crash versions. If you
can upgrade, and then post the output of:

$ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14:13/vmcore

there will be a plethora of debug output that can help determine the
problem.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Adil Mujeeb 02-09-2012 03:06 PM
crash tool not working
 
Hi Dave,

> No, that's really not true. *crash-5.0.1 is two years old, which is ancient
> by crash standards. *And it's very easy to upgrade:
>
> *$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
> *...
> *$ cd crash-6.0.3
> *$ make
> *...
> *$

I tried but at the end build failed:
-----------------------
cc -c -g -DX86_64 -DGDB_7_3_1 -g -O2 main.c
In file included from main.c:18:
defs.h:43:18: error: zlib.h: No such file or directory
-----------------------

Any idea which package it needed to get the zlib.h

Thanks for your help.

Regards,
Adil

On Thu, Feb 9, 2012 at 8:16 PM, Dave Anderson <anderson@redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi,
>>
>> >
>> > Well, that's your first mistake...
>> >
>>
>> Thanks for pointing it out :)
>> ..
>> [snip]
>> ..
>> > Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
>> > file is not a supported file format. *Never has been, never will
>> > be...
>> >
>>
>> I tried with vmlinux and it seems ok. But got another error
>> hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
>> /var/crash/2012-02-08-14:13/vmcore
>>
>> --------------------
>> crash 5.0.1
>> Copyright (C) 2002-2010 *Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006 *IBM Corporation
>> Copyright (C) 1999-2006 *Hewlett-Packard Co
>> Copyright (C) 2005, 2006 *Fujitsu Limited
>> Copyright (C) 2006, 2007 *VA Linux Systems Japan K.K.
>> Copyright (C) 2005 *NEC Corporation
>> Copyright (C) 1999, 2002, 2007 *Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002 *Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions. *Enter "help copying" to see the conditions.
>> This program has absolutely no warranty. *Enter "help warranty" for
>> details.
>>
>> GNU gdb (GDB) 7.0
>> Copyright (C) 2009 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. *Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>>
>> crash: seek error: kernel virtual address: ffffffff81827aa0 *type: "kernel_config_data"
>> WARNING: cannot read kernel_config_data
>> crash: seek error: kernel virtual address: ffffffff8181a660 *type: "cpu_possible_mask"
>> hltncra110731:/home/adil #
>> ----------------------------------
>>
>>
>> > Also, do yourself a favor, and upgrade your crash utility version
>> > from crash-5.0.1, which is two years old.
>>
>> I am using for learning purpose. So i guess this version should be Ok
>> to use for learning purpose.
>
> No, that's really not true. *crash-5.0.1 is two years old, which is ancient
> by crash standards. *And it's very easy to upgrade:
>
> *$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
> *...
> *$ cd crash-6.0.3
> *$ make
> *...
> *$
>
> As far as the "seek errors" you're seeing, I don't know what's happening,
> other than crash can't find the physical address page associated with those
> unity-mapped kernel virtual addresses in the vmcore. *But I'm sorry -- I don't
> spend time debugging issues encountered with obsolete crash versions. *If you
> can upgrade, and then post the output of:
>
> *$ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14:13/vmcore
>
> there will be a plethora of debug output that can help determine the
> problem.
>
> Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-09-2012 03:12 PM
crash tool not working
 
----- Original Message -----
> Hi Dave,
>
> > No, that's really not true. *crash-5.0.1 is two years old, which is
> > ancient
> > by crash standards. *And it's very easy to upgrade:
> >
> > *$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
> > *...
> > *$ cd crash-6.0.3
> > *$ make
> > *...
> > *$
>
> I tried but at the end build failed:
> -----------------------
> cc -c -g -DX86_64 -DGDB_7_3_1 -g -O2 main.c
> In file included from main.c:18:
> defs.h:43:18: error: zlib.h: No such file or directory
> -----------------------
>
> Any idea which package it needed to get the zlib.h
>
> Thanks for your help.
>
> Regards,
> Adil

Right -- sorry about that....

If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
file, it would have prevented the build because the crash.spec file
requires these two packages:

BuildRequires: ncurses-devel zlib-devel

Anyway, those two packages are required.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Adil Mujeeb 02-10-2012 05:28 AM
crash tool not working
 
Thanks Dave.

> If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
> file, it would have prevented the build because the crash.spec file
> requires these two packages:
>
> *BuildRequires: ncurses-devel zlib-devel
>
> Anyway, those two packages are required.

I installed zlib-devel and build was successful.

> If you can upgrade, and then post the output of:
>
> $ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14:13/vmcore
> there will be a plethora of debug output that can help determine the problem.

Please find the output below:

---------------------------
hltncra110731:/home/adil/crash-6.0.3 # ./crash -d8
../linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14:13/vmcore

crash 6.0.3
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

compressed kdump: header->utsname.machine:
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: /var/crash/2012-02-08-14:13/vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)

header: dedfe0
signature: "KDUMP "
header_version: 1
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0

sub_header: 0 (n/a)

sub_header_kdump: deeff0
phys_base: 0
dump_level: 0 (0x0)

data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f99c0b4e010
bitmap_len: 311296
dumpable_bitmap: 7f99c0b01010
byte: 0
bit: 0
compressed_page: e009a0
curbufptr: 0

page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df0990
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df1990
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df2990
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df3990
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df4990
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df5990
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df6990
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df7990
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df8990
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df9990
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfa990
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfb990
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfc990
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfd990
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfe990
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dff990
pg_hit_count: 0

page_cache_buf: df0990
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: df0000
readmem: read_diskdump()
crash: pv_init_ops exists: ARCH_PVOPS
compressed kdump: phys_base: 0
gdb ../linux-2.6.32.12-0.7/vmlinux
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(248 -> 0)
GETBUF(1500 -> 1)

FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff82827aa0, KVADDR, "kernel_config_data", 32768, (ROE), 1a1bab0>
<read_diskdump: addr: ffffffff82827aa0 paddr: 2827aa0 cnt: 1376>
read_diskdump: SEEK_ERROR: paddr/pfn: 2827aa0/2827 !page_is_ram
crash: seek error: kernel virtual address: ffffffff82827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
GETBUF(248 -> 0)
FREEBUF(0)
GETBUF(512 -> 0)
<readmem: ffffffff8281a660, KVADDR, "cpu_possible_mask", 8, (FOE), 7fff1b538678>
<read_diskdump: addr: ffffffff8281a660 paddr: 281a660 cnt: 8>
read_diskdump: SEEK_ERROR: paddr/pfn: 281a660/281a !page_is_ram
crash: seek error: kernel virtual address: ffffffff8281a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil/crash-6.0.3 #

---------------------------

BTW i tried to setup a virtual machine (using VirtualBox) on another
machine and on that Virtual machine following the same setup, crash
tool was working. But not on this :(

Thanks,
Adil

On Thu, Feb 9, 2012 at 9:42 PM, Dave Anderson <anderson@redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi Dave,
>>
>> > No, that's really not true. *crash-5.0.1 is two years old, which is
>> > ancient
>> > by crash standards. *And it's very easy to upgrade:
>> >
>> > *$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
>> > *...
>> > *$ cd crash-6.0.3
>> > *$ make
>> > *...
>> > *$
>>
>> I tried but at the end build failed:
>> -----------------------
>> cc -c -g -DX86_64 *-DGDB_7_3_1 -g -O2 main.c
>> In file included from main.c:18:
>> defs.h:43:18: error: zlib.h: No such file or directory
>> -----------------------
>>
>> Any idea which package it needed to get the zlib.h
>>
>> Thanks for your help.
>>
>> Regards,
>> Adil
>
> Right -- sorry about that....
>
> If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
> file, it would have prevented the build because the crash.spec file
> requires these two packages:
>
> *BuildRequires: ncurses-devel zlib-devel
>
> Anyway, those two packages are required.
>
> Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-10-2012 02:55 PM
crash tool not working
 
----- Original Message -----
> Thanks Dave.
>
> > If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
> > file, it would have prevented the build because the crash.spec file
> > requires these two packages:
> >
> > *BuildRequires: ncurses-devel zlib-devel
> >
> > Anyway, those two packages are required.
>
> I installed zlib-devel and build was successful.
>
> > If you can upgrade, and then post the output of:
> >
> > $ crash -d8 linux-2.6.32.12-0.7/vmlinux
> > /var/crash/2012-02-08-14:13/vmcore
> > there will be a plethora of debug output that can help determine
> > the problem.
>
> Please find the output below:
>
> ---------------------------
> hltncra110731:/home/adil/crash-6.0.3 # ./crash -d8
> ../linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14:13/vmcore
>
> crash 6.0.3
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine:
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: /var/crash/2012-02-08-14:13/vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: dedfe0
> signature: "KDUMP "
> header_version: 1
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> timestamp:
> tv_sec: 0
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 76
> max_mapnr: 1245184
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: deeff0
> phys_base: 0
> dump_level: 0 (0x0)
>
> data_offset: 4e000
> block_size: 4096
> block_shift: 12
> bitmap: 7f99c0b4e010
> bitmap_len: 311296
> dumpable_bitmap: 7f99c0b01010
> byte: 0
> bit: 0
> compressed_page: e009a0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df0990
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df1990
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df2990
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df3990
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df4990
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df5990
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df6990
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df7990
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df8990
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df9990
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfa990
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfb990
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfc990
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfd990
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfe990
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dff990
> pg_hit_count: 0
>
> page_cache_buf: df0990
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: df0000
> readmem: read_diskdump()
> crash: pv_init_ops exists: ARCH_PVOPS
> compressed kdump: phys_base: 0
> gdb ../linux-2.6.32.12-0.7/vmlinux
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(248 -> 0)
> GETBUF(1500 -> 1)
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffff82827aa0, KVADDR, "kernel_config_data", 32768,
> (ROE), 1a1bab0>
> <read_diskdump: addr: ffffffff82827aa0 paddr: 2827aa0 cnt: 1376>
> read_diskdump: SEEK_ERROR: paddr/pfn: 2827aa0/2827 !page_is_ram
> crash: seek error: kernel virtual address: ffffffff82827aa0 type:
> "kernel_config_data"
> WARNING: cannot read kernel_config_data
> GETBUF(248 -> 0)
> FREEBUF(0)
> GETBUF(512 -> 0)
> <readmem: ffffffff8281a660, KVADDR, "cpu_possible_mask", 8, (FOE),
> 7fff1b538678>
> <read_diskdump: addr: ffffffff8281a660 paddr: 281a660 cnt: 8>
> read_diskdump: SEEK_ERROR: paddr/pfn: 281a660/281a !page_is_ram
> crash: seek error: kernel virtual address: ffffffff8281a660 type:
> "cpu_possible_mask"
> hltncra110731:/home/adil/crash-6.0.3 #


The debug output looks reasonable, and it doesn't seem to
be a relocation issue because the dumpfile indicates this:

sub_header_kdump: deeff0
phys_base: 0
dump_level: 0 (0x0)

and

compressed kdump: phys_base: 0

So for the first two readmem() attempts, at ffffffff82827aa0
(kernel_config_data) and ffffffff8281a660 (cpu_possible_mask),
the kernel start map identifier of ffffffff80000000 is stripped,
leaving the physical addresses, which are then passed to the
compressed-kdump function read_diskdump(). But those physical
addresses cannot be found in the dumpfile.

I am presuming that the machine that generated the vmcore is still
running your ../linux-2.6.32.12-0.7/vmlinux kernel, and that you
can log onto it as root. If that's not the case, I can't help
much more.

If you are running that kernel on the crashed machine, for sanity's sake,
can you verify that your kernel has not relocated itself by doing this:

# nm -Bn ../linux-2.6.32.12-0.7/vmlinux | grep _stext

and comparing it to:

# cat /proc/kallsyms | grep _stext

The symbol values should be the same. For example, on this 2.6.32-based
system I see this:

# nm -Bn /usr/lib/debug/lib/modules/2.6.32-70.el6.x86_64/vmlinux | grep _stext
ffffffff81009000 T _stext
# cat /proc/kallsyms | grep _stext
ffffffff81009000 T _stext
#

Also do this:

# cat /proc/kallsyms | grep -e kernel_config_data -e cpu_possible_mask

and confirm that they are ffffffff82827aa0 and ffffffff82827aa0.

And lastly, in all cases where a dumpfile cannot be read correctly,
first verify that crash works with the live system. Presuming
t, and it was configured CONFIG_STRICT_DEVMEM, try this:

# crash ../linux-2.6.32.12-0.7/vmlinux

If that comes up OK, then we can rule out more fundamental
failure causes.

Dave




--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-10-2012 03:03 PM
crash tool not working
 
----- Original Message -----
>
> And lastly, in all cases where a dumpfile cannot be read correctly,
> first verify that crash works with the live system. Presuming
> t, and it was configured CONFIG_STRICT_DEVMEM, try this:

Sorry -- cut-and-paste error -- I meant to state "Presuming that
the kernel was *NOT* configured with CONFIG_STRICT_DEVMEM".

>
> # crash ../linux-2.6.32.12-0.7/vmlinux
>
> If that comes up OK, then we can rule out more fundamental
> failure causes.

If you did configure the kernel with CONFIG_STRICT_DEVMEM,
then you can try the supplied crash utility memory driver
as a replacement for /dev/mem:

# cd <path-to>/crash-6.0.3/memory_driver
# make
...
# insmod crash.ko

and then try running on the live system.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility


All times are GMT. The time now is 10:22 PM.
Page 1 of 2 1 2 >
Show 40 post(s) from this thread on one page

VBulletin, Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.