FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 06-25-2010, 08:22 PM
Dave Anderson
 
Default crash fails to load compressed RHEL5 vmcore

----- "marc pascual" <marc.m.pascual@gmail.com> wrote:

> Hello,
>
> I have this issue with compressed RHEL5 vmcore files, I can't get it
> to loaded to the crash utility. I don't have this problem with RHEL4 vmcores (diskdump).
> The debuginfo kernel exactly matches the kernel version that generated
> the vmcore
>
> [root@test-fc12 ~]# crash usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux /nas01/cores/2.6.18-92.1.10.el5.vmcore
>
> crash 4.0.9-2.fc12
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: page excluded: kernel virtual address: ffffffff804f1260 type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux and
> /nas01/cores/2.6.18-92.1.10.el5.vmcore do not match!
>
> Usage:
> crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
> [dumpfile]
>
> Enter "crash -h" for details.
>
> [root@test-fc12 ~]# strings usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux | grep 2.6 | head -2
> Linux version 2.6.18-92.1.10.el5debug (brewbuilder@ls20-bc2-13.build.redhat.com ) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jul 23 04:27:38 EDT 2008
>
> running strings on the rhel5 vmcore file:
>
> [root@test-fc12 cores]# strings 2.6.18-92.1.10.el5.vmcore | grep 2.6 |
> head -2
> 2X6
> 2T6(

If the diskdump was compressed (and with "makedumpfile -c" it is), then unfortunately
looking for the "Linux version" string won't help. The utsname data may be in the
compressed kdump header in more recent versions, so if you entered this:

# crash -d1 vmlinux vmcore
...

then the dumpfile header will be immediately dumped, and the utsname has most of
the relevant data from the "Linux version" string:

diskdump_data:
filename: (null)
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)

header: 19c42fe0
signature: "KDUMP "
header_version: 1
utsname:
sysname: Linux
nodename: hp-dl585g2-01.rhts.bos.redhat.com
release: 2.6.18-164.el5
version: #1 SMP Tue Aug 18 15:51:48 EDT 2009
machine: x86_64
domainname: (none)

But this was fixed fairly recently in makedumpfile, your dump may show
a bunch of (null) strings for the utsname data.

One thing I notice is that the vmlinux file is "2.6.18-92.1.10.el5debug",
and although you haven't shown exactly what kernel the crashed kernel
was running, you've named it "2.6.18-92.1.10.el5.vmcore". If the crashed
kernel was running 2.6.18-92.1.10.el5, then you're using the wrong vmlinux
file, and you should be using:

/usr/lib/debug/lib/modules/2.6.18-92.1.10.el5/vmlinux

They 2.6.18-92.1.10.el5 and 2.6.18-92.1.10.el5debug kernels are completely
different kernels.

If we presume that the crashed kernel and the vmlinux are *both* the "debug"
variety, then if you extend the debug data output by entering -d4, you'll see
every read attempt made from the dumpfile:

# crash -d4 vmlinux vmcore
... [ snip ] ...
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

<readmem: ffffffff80447280, KVADDR, "possible", 32, (ROE), c8fce0>
cpu_possible_map: 0 1 2 3 4 5 6 7
<readmem: ffffffff803ed5a0, KVADDR, "present", 32, (ROE), c8fce0>
cpu_present_map: 0 1 2 3 4 5 6 7
<readmem: ffffffff803e8260, KVADDR, "online", 32, (ROE), c8fce0>
cpu_online_map: 0 1 2 3 4 5 6 7
<readmem: ffffffff803ef200, KVADDR, "xtime", 16, (FOE), b50130>
<readmem: ffffffff80301320, KVADDR, "system_utsname", 390, (ROE), b5071c>
...

So, you can see that the very first readmem() is the cpu_possible_map bitmap.
And in your case:

> crash: page excluded: kernel virtual address: ffffffff804f1260 type: "possible"
> WARNING: cannot read cpu_possible_map

That first readmem() attempt failed because the page was explicitly
excluded by makedumpfile. But if makedumpfile's page exclusion mechanism
excluded the page containing that kernel data, I'd be very surprised.

Dave


> does makedumpconfig's compression have something to do with this? from
> kdump.conf on the machine where i got that vmcore:
> ...
> core_collector makedumpfile -c -d31
> ...
>
> Thank you in advace!
>
> Regards,
> Marc
>

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 06-28-2010, 02:55 PM
marc pascual
 
Default crash fails to load compressed RHEL5 vmcore

On Sat, Jun 26, 2010 at 4:22 AM, Dave Anderson <anderson@redhat.com> wrote:




----- "marc pascual" <marc.m.pascual@gmail.com> wrote:



> Hello,

>

> I have this issue with compressed RHEL5 vmcore files, I can't get it

> to loaded to the crash utility. I don't have this problem with RHEL4 vmcores (diskdump).

> The debuginfo kernel exactly matches the kernel version that generated

> the vmcore

>

> [root@test-fc12 ~]# crash usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux /nas01/cores/2.6.18-92.1.10.el5.vmcore

>

> crash 4.0.9-2.fc12

> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.

> Copyright (C) 2004, 2005, 2006 IBM Corporation

> Copyright (C) 1999-2006 Hewlett-Packard Co

> Copyright (C) 2005, 2006 Fujitsu Limited

> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.

> Copyright (C) 2005 NEC Corporation

> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.

> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.

> This program is free software, covered by the GNU General Public License,

> and you are welcome to change it and/or distribute copies of it under

> certain conditions. Enter "help copying" to see the conditions.

> This program has absolutely no warranty. Enter "help warranty" for

> details.

>

> GNU gdb 6.1

> Copyright 2004 Free Software Foundation, Inc.

> GDB is free software, covered by the GNU General Public License, and you are

> welcome to change it and/or distribute copies of it under certain conditions.

> Type "show copying" to see the conditions.

> There is absolutely no warranty for GDB. Type "show warranty" for details.

> This GDB was configured as "x86_64-unknown-linux-gnu"...

>

> crash: page excluded: kernel virtual address: ffffffff804f1260 type: "possible"

> WARNING: cannot read cpu_possible_map

> crash: usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux and

> /nas01/cores/2.6.18-92.1.10.el5.vmcore do not match!

>

> Usage:

> crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]

> [dumpfile]

>

> Enter "crash -h" for details.

>

> [root@test-fc12 ~]# strings usr/lib/debug/lib/modules/2.6.18-92.1.10.el5debug/vmlinux | grep 2.6 | head -2

> Linux version 2.6.18-92.1.10.el5debug (brewbuilder@ls20-bc2-13.build.redhat.com ) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jul 23 04:27:38 EDT 2008



>

> running strings on the rhel5 vmcore file:

>

> [root@test-fc12 cores]# strings 2.6.18-92.1.10.el5.vmcore | grep 2.6 |

> head -2

> 2X6

> 2T6(



If the diskdump was compressed (and with "makedumpfile -c" it is), then unfortunately

looking for the "Linux version" string won't help. *The utsname data may be in the

compressed kdump header in more recent versions, so if you entered this:



*# crash -d1 vmlinux vmcore

*...



then the dumpfile header will be immediately dumped, and the utsname has most of

the relevant data from the "Linux version" string:



*diskdump_data:

* * * * *filename: (null)

* * * * * * flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)

* * * * * * * dfd: 3

* * * * * * * ofp: 0

* * *machine_type: 62 (EM_X86_64)



* * * * * *header: 19c42fe0

* * * * * signature: "KDUMP * "

* * *header_version: 1

* * * * * * utsname:

* * * * * * * sysname: Linux

* * * * * * *nodename: hp-dl585g2-01.rhts.bos.redhat.com

* * * * * * * release: 2.6.18-164.el5

* * * * * * * version: #1 SMP Tue Aug 18 15:51:48 EDT 2009

* * * * * * * machine: x86_64

* * * * * *domainname: (none)



But this was fixed fairly recently in makedumpfile, your dump may show

a bunch of (null) strings for the utsname data.



One thing I notice is that the vmlinux file is "2.6.18-92.1.10.el5debug",

and although you haven't shown exactly what kernel the crashed kernel

was running, you've named it "2.6.18-92.1.10.el5.vmcore". *If the crashed

kernel was running 2.6.18-92.1.10.el5, then you're using the wrong vmlinux

file, and you should be using:



*/usr/lib/debug/lib/modules/2.6.18-92.1.10.el5/vmlinux



They 2.6.18-92.1.10.el5 and 2.6.18-92.1.10.el5debug kernels are completely

different kernels.



If we presume that the crashed kernel and the vmlinux are *both* the "debug"

variety, then if you extend the debug data output by entering -d4, you'll see

every read attempt made from the dumpfile:



*# crash -d4 vmlinux vmcore

*... [ snip ] ...

*GNU gdb (GDB) 7.0

*Copyright (C) 2009 Free Software Foundation, Inc.

*License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

*This is free software: you are free to change and redistribute it.

*There is NO WARRANTY, to the extent permitted by law. *Type "show copying"

*and "show warranty" for details.

*This GDB was configured as "x86_64-unknown-linux-gnu"...



*<readmem: ffffffff80447280, KVADDR, "possible", 32, (ROE), c8fce0>

*cpu_possible_map: 0 1 2 3 4 5 6 7

*<readmem: ffffffff803ed5a0, KVADDR, "present", 32, (ROE), c8fce0>

*cpu_present_map: 0 1 2 3 4 5 6 7

*<readmem: ffffffff803e8260, KVADDR, "online", 32, (ROE), c8fce0>

*cpu_online_map: 0 1 2 3 4 5 6 7

*<readmem: ffffffff803ef200, KVADDR, "xtime", 16, (FOE), b50130>

*<readmem: ffffffff80301320, KVADDR, "system_utsname", 390, (ROE), b5071c>

*...



So, you can see that the very first readmem() is the cpu_possible_map bitmap.

And in your case:



> crash: page excluded: kernel virtual address: ffffffff804f1260 type: "possible"

> WARNING: cannot read cpu_possible_map



That first readmem() attempt failed because the page was explicitly

excluded by makedumpfile. *But if makedumpfile's page exclusion mechanism

excluded the page containing that kernel data, I'd be very surprised.



Dave





> does makedumpconfig's compression have something to do with this? from

> kdump.conf on the machine where i got that vmcore:

> ...

> core_collector makedumpfile -c -d31

> ...

>

> Thank you in advace!

>

> Regards,

> Marc

>



--

Crash-utility mailing list

Crash-utility@redhat.com

https://www.redhat.com/mailman/listinfo/crash-utility


Dave,


Thank you for your reply. However I'm slightly confused. the kernel version where I got the vmcore file is 2.6.18-92.1.10.el5 (x86_64). From my previous understanding, I should download and install kernel-debug-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm. This package gives me 2.6.18-92.1.10.el5debug. However you mentioned that:


"They 2.6.18-92.1.10.el5 and 2.6.18-92.1.10.el5debug kernels are completely different kernels."

If this is the case, which package can I get 2.6.18-92.1.10.el5/vmlinux from? Assuming this is not the same kernel under /boot


Thanks,
Marc




--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 06-29-2010, 12:36 PM
Dave Anderson
 
Default crash fails to load compressed RHEL5 vmcore

----- "marc pascual" <marc.m.pascual@gmail.com> wrote:

> Dave,
>
> Thank you for your reply. However I'm slightly confused. the kernel
> version where I got the vmcore file is 2.6.18-92.1.10.el5 (x86_64).
> From my previous understanding, I should download and install
> kernel-debug-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm. This package
> gives me 2.6.18-92.1.10.el5debug. However you mentioned that:
>
> "They 2.6.18-92.1.10.el5 and 2.6.18-92.1.10.el5debug kernels are
> completely different kernels."
>
> If this is the case, which package can I get
> 2.6.18-92.1.10.el5/vmlinux from? Assuming this is not the same kernel
> under /boot?

The short answer is: kernel-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm

That RHEL5 kernel version offers 3 different x86_64 kernel packages:

kernel-2.6.18-92.1.10.el5.x86_64.rpm
kernel-xen-2.6.18-92.1.10.el5.x86_64.rpm
kernel-debug-2.6.18-92.1.10.el5.x86_64.rpm

Each of the above has its own associated debuginfo package,
which contains the relevant vmlinux file for that kernel
type:

kernel-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm
kernel-xen-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm
kernel-debug-debuginfo-2.6.18-92.1.10.el5.x86_64.rpm

And when installing any of the -debuginfo packages above,
rpm requires that the kernel-debuginfo-common package be
installed along with it:

kernel-debuginfo-common-2.6.18-92.1.10.el5.x86_64.rpm

Like the name says, the "common" package has debuginfo
data that is the same for all 3 kernel types.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 10:45 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org