FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 08-23-2010, 10:59 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

On 08/23/2010 23:25, Dave Anderson wrote:

cat /proc/kallsyms


I have attached allsymbols from the live system.

As for the debuginfo, that's exactly what I was doing(using the vmlinux
with the -g debuginfo from the root of the kernel tree)
--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-24-2010, 01:23 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> On 08/23/2010 23:25, Dave Anderson wrote:
> > cat /proc/kallsyms
>
> I have attached allsymbols from the live system.
>
> As for the debuginfo, that's exactly what I was doing(using the vmlinux
> with the -g debuginfo from the root of the kernel tree)

OK, so the /proc/kallsyms shows a set of symbols with a more typical
virtual address range, where the kernel's PAGE_OFFSET unity-map
value is c0000000:

c0101000 T _stext
c0101010 T do_one_initcall
c0101180 t init_post
c01012c0 T name_to_dev_t
c0101500 T thread_saved_pc
c0101510 T prepare_to_copy
c0101590 T get_wchan
c0101640 T __switch_to
c01017f0 T start_thread
c0101840 T copy_thread
c0101990 T release_thread
...

And that seemingly matches the symbol values captured
in the VMCOREINFO section of the vmcore's ELF header:

# grep SYMBOL crashd8.txt
SYMBOL(init_uts_ns)=c06f9120
SYMBOL(node_online_map)=c0730644
SYMBOL(swapper_pg_dir)=c06e4000
SYMBOL(_stext)=c0101000
SYMBOL(vmlist)=c07d3540
SYMBOL(mem_map)=c07d3500
SYMBOL(contig_page_data)=c072ce80
SYMBOL(log_buf)=c06fc83c
SYMBOL(log_end)=c07bb7ec
SYMBOL(log_buf_len)=c06fc838
SYMBOL(logged_chars)=c07c38a0
#

Unfortunately the /proc/kallsyms output only contains text
symbols, and "_stext" is the only text symbol in the VMCOREINFO
list value that can be cross-checked between the two. (I'm not
sure why your /proc/kallsyms doesn't contain data symbols?)
In any case, at least "_stext" does match at c0101000.

What I should have asked is for is the /boot/System.map-<version>
on that system, which will contain data symbols that *could* be
matched against the "SYMBOL(xxx)" values above.

In any case, since the symbol list from the vmlinux file that
you are using looks like this (from the "sym -l" output):

c1000000 (T) _text
c1000000 (T) startup_32
c1000054 (t) default_entry
c1001000 (T) _stext
c1001010 (T) do_one_initcall
c1001180 (t) init_post
c10012c0 (T) name_to_dev_t
c1001500 (T) thread_saved_pc
c1001510 (T) prepare_to_copy
c1001590 (T) get_wchan
c1001640 (T) __switch_to
c10017f0 (T) start_thread
...

which, BTW, you can also see by doing this:

# nm -Bn vmlinux

it's clearly not the same vmlinux that was running when
the system crashed.

By any chance did you build that vmlinux after-the-fact
in an attempt to "match" the kernel that was running?

In any case, what you can try would be this. Get the original
/boot/System-map-<version> file from the target system, and,
presuming the symbol values in that file match the VMCOREINFO's
SYMBOL(xxx) values, as well as matching the symbols in the
/proc/kallsyms output, and do this:

# crash System.map-<version> vmlinux vmcore.201008231930

Using a System.map file on the command line is pretty much an
act of desperation, but it may work. The crash utility will
take the symbols in the System.map file, go into a back door
into the embedded gdb module, and over-write the symbols from
the vmlinux file. But there's no guarantee that it will work,
because there may be differences in the definition of crucial
data structures that may cause crash to go off into the weeds.

Other than that, there's nothing much else that I can suggest.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-25-2010, 02:41 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

I tried with the system map, but it did not help, still got the same error.

I am pretty sure I didn't recompile after installing the kernel, see:
$ ll /boot/vmlinuz-2.6.35.3-saber /usr/src/linux-2.6.35.3/arch/x86
/boot/bzImage vmcore.201008232109

-rw-r--r-- 1 root root 3982336 Aug 23 21:03 /boot/vmlinuz-2.6.35.3-saber
-rw-r--r-- 1 root root 3982336 Aug 23 21:02
/usr/src/linux-2.6.35.3/arch/x86/boot/bzImage

-r-------- 1 root root 995686196 Aug 23 21:10 vmcore.201008232109

I also tried many times with many crashdumps and it never seemed to
work. Could the stripping or bzImage'ing change something?
I would try directly the vmlinux kernel, but it's over 120MB and my boot
partition is smaller than that.
Maybe there's some kernel options that change something in the bzImaged
vmlinux?


- Paul-Kenji

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-25-2010, 02:58 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> I tried with the system map, but it did not help, still got the same error.
>
> I am pretty sure I didn't recompile after installing the kernel, see:
> $ ll /boot/vmlinuz-2.6.35.3-saber /usr/src/linux-2.6.35.3/arch/x86 /boot/bzImage vmcore.201008232109
> -rw-r--r-- 1 root root 3982336 Aug 23 21:03 /boot/vmlinuz-2.6.35.3-saber
> -rw-r--r-- 1 root root 3982336 Aug 23 21:02 /usr/src/linux-2.6.35.3/arch/x86/boot/bzImage
> -r-------- 1 root root 995686196 Aug 23 21:10 vmcore.201008232109
>
> I also tried many times with many crashdumps and it never seemed to
> work. Could the stripping or bzImage'ing change something?
> I would try directly the vmlinux kernel, but it's over 120MB and my boot
> partition is smaller than that.
> Maybe there's some kernel options that change something in the bzImaged
> vmlinux?

The vmlinuz file is a compressed, stripped-down, and otherwise-manipulated
file that is created from the /usr/src/linux-2.6.35.3/vmlinux file -- which
is the file that crash requires.

And the vmlinux-to-bzImage process would not change the compiled-in
kernel virtual addresses that you see if you do this:

# nm -Bn /usr/src/linux-2.6.35.3/vmlinux

If it does, then that's news to me...

Dave





--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-25-2010, 03:08 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> I tried with the system map, but it did not help, still got the same
> error.

I don't know why you would have gotten "the same error" when you used
the System.map file.

Originally you saw:

<readmem: c148a9a0, KVADDR, "kernel_config_data", 32768, (ROE), 8bad3d8>
crash: read error: kernel virtual address: c148a9a0 type: "kernel_config_data"
WARNING: cannot read kernel_config_data
<readmem: c1487e28, KVADDR, "cpu_possible_mask", 4, (FOE), bfed4bbc>
crash: read error: kernel virtual address: c1487e28 type: "cpu_possible_mask"

And that's because the vmlinux file that you were using had compiled-in
kernel virtual addresses that started with "c1xxxxxx". If the System.map
file was put on the command line, it would have overwritten those kernel
addresses with the ones found in the System.map, i.e., those that begin
with "c0xxxxxx".

Did the System.map file that you used contain symbols that started
with "c1xxxxxx" or "c0xxxxxx"?

And if they did start with "c0xxxxxx", can you attach the "crash -d8 ..."
output?

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 07:19 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org