Hi,

Could someone please explain why we cannot get source line numbers when
disassembling modules? Is the problem missing functionality in
store_load_module_symbols() or is there an inherent limitation with the
gdb linked in with crash in that it cannot accept additional debug_line
information for modules? I ask because gdb has this functionality for
user-space shared libraries and it seems to work OK. Or am I missing
something which is much more complex?


Thanks,

Alan Tyson,
Hewlett-Packard.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Alan Tyson wrote:

Hi,

Could someone please explain why we cannot get source line numbers when
disassembling modules? Is the problem missing functionality in
store_load_module_symbols() or is there an inherent limitation with the
gdb linked in with crash in that it cannot accept additional debug_line
information for modules? I ask because gdb has this functionality for
user-space shared libraries and it seems to work OK. Or am I missing
something which is much more complex?


Thanks,

Alan Tyson,
Hewlett-Packard.


What architecture?

I'm presuming that you're running the "mod" command to load
the debuginfo data from the target module:

# crash
...
crash> dis -l ext3_count_dirs
0xe08aa000 <ext3_count_dirs>: push %edi
0xe08aa001 <ext3_count_dirs+1>: mov %eax,%edi
0xe08aa003 <ext3_count_dirs+3>: push %esi
0xe08aa004 <ext3_count_dirs+4>: xor %esi,%esi
0xe08aa006 <ext3_count_dirs+6>: push %ebx
0xe08aa007 <ext3_count_dirs+7>: xor %ebx,%ebx
0xe08aa009 <ext3_count_dirs+9>: jmp 0xe08aa021
0xe08aa00b <ext3_count_dirs+11>: xor %ecx,%ecx
0xe08aa00d <ext3_count_dirs+13>: mov %ebx,%edx
0xe08aa00f <ext3_count_dirs+15>: mov %edi,%eax
0xe08aa011 <ext3_count_dirs+17>: call 0xe08a809b <ext3_get_group_desc>
0xe08aa016 <ext3_count_dirs+22>: test %eax,%eax
0xe08aa018 <ext3_count_dirs+24>: je 0xe08aa020
0xe08aa01a <ext3_count_dirs+26>: movzwl 0x10(%eax),%eax
0xe08aa01e <ext3_count_dirs+30>: add %eax,%esi
0xe08aa020 <ext3_count_dirs+32>: inc %ebx
0xe08aa021 <ext3_count_dirs+33>: mov 0x160(%edi),%eax
0xe08aa027 <ext3_count_dirs+39>: cmp 0x24(%eax),%ebx
0xe08aa02a <ext3_count_dirs+42>: jb 0xe08aa00b
0xe08aa02c <ext3_count_dirs+44>: mov %esi,%eax
0xe08aa02e <ext3_count_dirs+46>: pop %ebx
0xe08aa02f <ext3_count_dirs+47>: pop %esi
0xe08aa030 <ext3_count_dirs+48>: pop %edi
0xe08aa031 <ext3_count_dirs+49>: ret
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
e08c4f80 ext3 123337
/lib/modules/2.6.18-53.el5/kernel/fs/ext3/ext3.ko

crash> dis -l ext3_count_dirs
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 746
0xe08aa000 <ext3_count_dirs>: push %edi
0xe08aa001 <ext3_count_dirs+1>: mov %eax,%edi
0xe08aa003 <ext3_count_dirs+3>: push %esi
0xe08aa004 <ext3_count_dirs+4>: xor %esi,%esi
0xe08aa006 <ext3_count_dirs+6>: push %ebx
0xe08aa007 <ext3_count_dirs+7>: xor %ebx,%ebx
0xe08aa009 <ext3_count_dirs+9>: jmp 0xe08aa021 <ext3_count_dirs+33>
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 751
0xe08aa00b <ext3_count_dirs+11>: xor %ecx,%ecx
0xe08aa00d <ext3_count_dirs+13>: mov %ebx,%edx
0xe08aa00f <ext3_count_dirs+15>: mov %edi,%eax
0xe08aa011 <ext3_count_dirs+17>: call 0xe08a809b <ext3_get_group_desc>
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 752
0xe08aa016 <ext3_count_dirs+22>: test %eax,%eax
0xe08aa018 <ext3_count_dirs+24>: je 0xe08aa020 <ext3_count_dirs+32>
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 754
0xe08aa01a <ext3_count_dirs+26>: movzwl 0x10(%eax),%eax
0xe08aa01e <ext3_count_dirs+30>: add %eax,%esi
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 750
0xe08aa020 <ext3_count_dirs+32>: inc %ebx
0xe08aa021 <ext3_count_dirs+33>: mov 0x160(%edi),%eax
0xe08aa027 <ext3_count_dirs+39>: cmp 0x24(%eax),%ebx
0xe08aa02a <ext3_count_dirs+42>: jb 0xe08aa00b <ext3_count_dirs+11>
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/ext3/ialloc.c: 757
0xe08aa02c <ext3_count_dirs+44>: mov %esi,%eax
0xe08aa02e <ext3_count_dirs+46>: pop %ebx
0xe08aa02f <ext3_count_dirs+47>: pop %esi
0xe08aa030 <ext3_count_dirs+48>: pop %edi
0xe08aa031 <ext3_count_dirs+49>: ret
crash>

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alan Tyson wrote:
> Hi,
>
> Could someone please explain why we cannot get source line numbers when
> disassembling modules? Is the problem missing functionality in
> store_load_module_symbols() or is there an inherent limitation with the
> gdb linked in with crash in that it cannot accept additional debug_line
> information for modules? I ask because gdb has this functionality for
> user-space shared libraries and it seems to work OK. Or am I missing
> something which is much more complex?

Hi Alan,

What versions of crash/kernel are you using? I've encountered this
before, but it seems to be dependent on the version of crash and/or the
kernel that you're working on.

E.g. on some RHEL4 crash builds, and with some kernels, "dis -l" fails
to give useful line number references, while with others it's fine.

I didn't note down which combinations I found worked and which
didn't - I ended up using it as an opportunity to improve my assembler
reading skills and did without the line number info

I'll go back over this and try to figure out what does & doesn't work
for me.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFHjhPP6YSQoMYUY94RAiwQAJ9IO8spfCnB/yFAcGk9E+5BpB7U/wCg2NMd
isaV6VQ2GakMoadduPW5e9I=
=EYWz
-----END PGP SIGNATURE-----

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson wrote:


Alan Tyson wrote:



Hi,

Could someone please explain why we cannot get source line numbers when
disassembling modules? Is the problem missing functionality in
store_load_module_symbols() or is there an inherent limitation with the
gdb linked in with crash in that it cannot accept additional debug_line
information for modules? I ask because gdb has this functionality for
user-space shared libraries and it seems to work OK. Or am I missing
something which is much more complex?

Thanks,

Alan Tyson,
Hewlett-Packard.




What architecture?



x86_64. Sorry I omitted to mention this.


I'm presuming that you're running the "mod" command to load
the debuginfo data from the target module:

<< working example on i686 snipped >>


Yes I am. Indeed it seems that i686 and ia64 work fine. But x86_64 has
the problem.


Thanks,
Alan.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Alan Tyson wrote:

Dave Anderson wrote:


Alan Tyson wrote:



Hi,

Could someone please explain why we cannot get source line numbers when
disassembling modules? Is the problem missing functionality in
store_load_module_symbols() or is there an inherent limitation with the
gdb linked in with crash in that it cannot accept additional debug_line
information for modules? I ask because gdb has this functionality for
user-space shared libraries and it seems to work OK. Or am I missing
something which is much more complex?

Thanks,

Alan Tyson,
Hewlett-Packard.



What architecture?



x86_64. Sorry I omitted to mention this.


I'm presuming that you're running the "mod" command to load
the debuginfo data from the target module:

<< working example on i686 snipped >>


Yes I am. Indeed it seems that i686 and ia64 work fine. But x86_64 has
the problem.


Thanks,
Alan.


Hmmm -- seems to work with RHEL5:

# crash

crash 4.0-4.13
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/lib/modules/2.6.18-61.el5/vmlinux
DUMPFILE: /dev/crash
CPUS: 2
DATE: Wed Jan 16 08:43:47 2008
UPTIME: 5 days, 01:36:07
LOAD AVERAGE: 0.21, 0.06, 0.02
TASKS: 101
NODENAME: dhcp83-53.boston.redhat.com
RELEASE: 2.6.18-61.el5
VERSION: #1 SMP Tue Jan 8 13:56:19 EST 2008
MACHINE: x86_64 (2793 Mhz)
MEMORY: 1 GB
PID: 32197
COMMAND: "crash"
TASK: ffff810030dd3860 [THREAD_INFO: ffff81002c3d4000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)

crash> dis -l ext3_count_dirs
0xffffffff8804c34c <ext3_count_dirs>: push %r12
0xffffffff8804c34e <ext3_count_dirs+2>: mov %rdi,%r12
0xffffffff8804c351 <ext3_count_dirs+5>: push %rbp
0xffffffff8804c352 <ext3_count_dirs+6>: xor %ebp,%ebp
0xffffffff8804c354 <ext3_count_dirs+8>: push %rbx
0xffffffff8804c355 <ext3_count_dirs+9>: xor %ebx,%ebx
0xffffffff8804c357 <ext3_count_dirs+11>: jmp 0xffffffff8804c374
0xffffffff8804c359 <ext3_count_dirs+13>: xor %edx,%edx
0xffffffff8804c35b <ext3_count_dirs+15>: mov %ebx,%esi
0xffffffff8804c35d <ext3_count_dirs+17>: mov %r12,%rdi
0xffffffff8804c360 <ext3_count_dirs+20>: callq 0xffffffff8804a0a0
<ext3_get_group_desc>

0xffffffff8804c365 <ext3_count_dirs+25>: test %rax,%rax
0xffffffff8804c368 <ext3_count_dirs+28>: je 0xffffffff8804c371
0xffffffff8804c36a <ext3_count_dirs+30>: movzwl 0x10(%rax),%eax
0xffffffff8804c36e <ext3_count_dirs+34>: add %rax,%rbp
0xffffffff8804c371 <ext3_count_dirs+37>: inc %rbx
0xffffffff8804c374 <ext3_count_dirs+40>: mov 0x240(%r12),%rax
0xffffffff8804c37c <ext3_count_dirs+48>: cmp 0x48(%rax),%rbx
0xffffffff8804c380 <ext3_count_dirs+52>: jb 0xffffffff8804c359
0xffffffff8804c382 <ext3_count_dirs+54>: pop %rbx
0xffffffff8804c383 <ext3_count_dirs+55>: mov %rbp,%rax
0xffffffff8804c386 <ext3_count_dirs+58>: pop %rbp
0xffffffff8804c387 <ext3_count_dirs+59>: pop %r12
0xffffffff8804c389 <ext3_count_dirs+61>: retq
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
ffffffff8806aa00 ext3 166993
/lib/modules/2.6.18-61.el5.bz232489.2/kernel/fs/ext3/ext3.ko

crash> dis -l ext3_count_dirs
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 746
0xffffffff8804c34c <ext3_count_dirs>: push %r12
0xffffffff8804c34e <ext3_count_dirs+2>: mov %rdi,%r12
0xffffffff8804c351 <ext3_count_dirs+5>: push %rbp
0xffffffff8804c352 <ext3_count_dirs+6>: xor %ebp,%ebp
0xffffffff8804c354 <ext3_count_dirs+8>: push %rbx
0xffffffff8804c355 <ext3_count_dirs+9>: xor %ebx,%ebx
0xffffffff8804c357 <ext3_count_dirs+11>: jmp 0xffffffff8804c374
<ext3_count_dirs+40>

/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 751
0xffffffff8804c359 <ext3_count_dirs+13>: xor %edx,%edx
0xffffffff8804c35b <ext3_count_dirs+15>: mov %ebx,%esi
0xffffffff8804c35d <ext3_count_dirs+17>: mov %r12,%rdi
0xffffffff8804c360 <ext3_count_dirs+20>: callq 0xffffffff8804a0a0
<ext3_get_group_desc>

/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 752
0xffffffff8804c365 <ext3_count_dirs+25>: test %rax,%rax
0xffffffff8804c368 <ext3_count_dirs+28>: je 0xffffffff8804c371
<ext3_count_dirs+37>

/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 754
0xffffffff8804c36a <ext3_count_dirs+30>: movzwl 0x10(%rax),%eax
0xffffffff8804c36e <ext3_count_dirs+34>: add %rax,%rbp
0xffffffff8804c371 <ext3_count_dirs+37>: inc %rbx
/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 750
0xffffffff8804c374 <ext3_count_dirs+40>: mov 0x240(%r12),%rax
0xffffffff8804c37c <ext3_count_dirs+48>: cmp 0x48(%rax),%rbx
0xffffffff8804c380 <ext3_count_dirs+52>: jb 0xffffffff8804c359
<ext3_count_dirs+13>

/usr/src/debug/kernel-2.6.18/linux-2.6.18.x86_64/fs/ext3/ialloc.c: 757
0xffffffff8804c382 <ext3_count_dirs+54>: pop %rbx
0xffffffff8804c383 <ext3_count_dirs+55>: mov %rbp,%rax
0xffffffff8804c386 <ext3_count_dirs+58>: pop %rbp
0xffffffff8804c387 <ext3_count_dirs+59>: pop %r12
0xffffffff8804c389 <ext3_count_dirs+61>: retq
crash>

RHEL4 x86_64 doesn't appear to work though...

Dave


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Anderson wrote:
> RHEL4 x86_64 doesn't appear to work though...
>

That's what I'm seeing - broken on RHEL4 x86_64 (on various kernels:
- -55.EL, -67.EL)

KERNEL: /usr/lib/debug/lib/modules/2.6.9-55.ELsmp/vmlinux
DUMPFILE: /dev/crash
CPUS: 8
DATE: Wed Jan 16 18:53:23 2008
UPTIME: 8 days, 20:14:38
LOAD AVERAGE: 0.98, 0.36, 0.12
TASKS: 127
NODENAME: hs21-8853-1.gsslab.rdu.redhat.com
RELEASE: 2.6.9-55.ELsmp
VERSION: #1 SMP Fri Apr 20 16:36:54 EDT 2007
MACHINE: x86_64 (1866 Mhz)
MEMORY: 1 GB
PID: 23313
COMMAND: "crash"
TASK: 1003c2057f0 [THREAD_INFO: 1001aa52000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)

crash> mod -S
<snip>

crash> dis -l ext3_count_dirs
include/linux/seqlock.h: 77
0xffffffffa00a81e2 <ext3_count_dirs>: push %r12
0xffffffffa00a81e4 <ext3_count_dirs+2>: xor %r12d,%r12d
0xffffffffa00a81e7 <ext3_count_dirs+5>: push %rbp
0xffffffffa00a81e8 <ext3_count_dirs+6>: xor %ebp,%ebp
0xffffffffa00a81ea <ext3_count_dirs+8>: push %rbx
0xffffffffa00a81eb <ext3_count_dirs+9>: mov 0x260(%rdi),%rax
0xffffffffa00a81f2 <ext3_count_dirs+16>: mov %rdi,%rbx
0xffffffffa00a81f5 <ext3_count_dirs+19>: cmp 0x48(%rax),%r12
0xffffffffa00a81f9 <ext3_count_dirs+23>: jae
0xffffffffa00a8226 <ext3_count_dirs+68>
0xffffffffa00a81fb <ext3_count_dirs+25>: xor %edx,%edx
0xffffffffa00a81fd <ext3_count_dirs+27>: mov %ebp,%esi
0xffffffffa00a81ff <ext3_count_dirs+29>: mov %rbx,%rdi
0xffffffffa00a8202 <ext3_count_dirs+32>: callq
0xffffffffa00a5000 <ext3_get_group_desc>
0xffffffffa00a8207 <ext3_count_dirs+37>: test %rax,%rax
0xffffffffa00a820a <ext3_count_dirs+40>: je
0xffffffffa00a8214 <ext3_count_dirs+50>
0xffffffffa00a820c <ext3_count_dirs+42>: movzwq 0x10(%rax),%rax
0xffffffffa00a8211 <ext3_count_dirs+47>: add %rax,%r12
0xffffffffa00a8214 <ext3_count_dirs+50>: mov 0x260(%rbx),%rax
0xffffffffa00a821b <ext3_count_dirs+57>: inc %ebp
0xffffffffa00a821d <ext3_count_dirs+59>: movslq %ebp,%rdx
0xffffffffa00a8220 <ext3_count_dirs+62>: cmp 0x48(%rax),%rdx
0xffffffffa00a8224 <ext3_count_dirs+66>: jmp
0xffffffffa00a81f9 <ext3_count_dirs+23>
0xffffffffa00a8226 <ext3_count_dirs+68>: pop %rbx
0xffffffffa00a8227 <ext3_count_dirs+69>: pop %rbp
0xffffffffa00a8228 <ext3_count_dirs+70>: mov %r12,%rax
0xffffffffa00a822b <ext3_count_dirs+73>: pop %r12
0xffffffffa00a822d <ext3_count_dirs+75>: retq
0xffffffffa00a822e <ext3_count_dirs+76>: nop
0xffffffffa00a822f <ext3_count_dirs+77>: nop

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFHjlTj6YSQoMYUY94RAiQRAJ95/FQ+iLKP/z8QQjdJrp889iaJNACdHRIG
ltOlwhvZCP6vZ6ZsRX9mFmg=
=2WhK
-----END PGP SIGNATURE-----

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility