crash-6.0.2 won't do module source lines on our kernel

01-25-2012, 11:16 PM

I've reverted back to crash-5.1.9 and applied my kmem patch to that for
our use here. We're using a 3.1-based kernel, and need the kmem patch
so crash can deal with the change in CONFIG_SLAB, but we're building
with gcc-4.4.5 and don't really need the new gdb in crash-6.0.2, and
crash-6.0.2 is not giving module source line numbers for us with "dis
-l".

This is just a heads up. I don't know why 6.0.2 is failing this, and
since I found the last module source line number problem, it's not my
turn ;-)

Bob Montgomery

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

01-26-2012, 01:12 PM

----- Original Message -----
> I've reverted back to crash-5.1.9 and applied my kmem patch to that
> for
> our use here. We're using a 3.1-based kernel, and need the kmem
> patch
> so crash can deal with the change in CONFIG_SLAB, but we're building
> with gcc-4.4.5 and don't really need the new gdb in crash-6.0.2, and
> crash-6.0.2 is not giving module source line numbers for us with "dis -l".
>
> This is just a heads up. I don't know why 6.0.2 is failing this, and
> since I found the last module source line number problem, it's not my
> turn ;-)
>
> Bob Montgomery

What kernel version? I'll try to reproduce it.

BTW, you might try this patch to symbols.c:

https://www.redhat.com/archives/crash-utility/2011-December/msg00043.html

> I could not found out corresponding condition from gdb-7.3.1/
> but maybe -readnow is simple and proper way for this problem.
> I've tested and got the good result.
>
> // sprintf(buf, "add-symbol-file %s 0x%lx",
> sprintf(buf, "add-symbol-file %s 0x%lx
> -readnow",
> lm->mod_namelist, section_vaddr);

Although I still don't know why the module line-number fixes that went
into crash-6.0.1 wouldn't accomplish the same thing without the overhead
of doing it for all symbols of all modules.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

01-26-2012, 07:24 PM

----- Original Message -----
>
>
> ----- Original Message -----
> > I've reverted back to crash-5.1.9 and applied my kmem patch to that for
> > our use here. We're using a 3.1-based kernel, and need the kmem patch
> > so crash can deal with the change in CONFIG_SLAB, but we're building
> > with gcc-4.4.5 and don't really need the new gdb in crash-6.0.2, and
> > crash-6.0.2 is not giving module source line numbers for us with "dis -l".
> >
> > This is just a heads up. I don't know why 6.0.2 is failing this, and
> > since I found the last module source line number problem, it's not my
> > turn ;-)
> >
> > Bob Montgomery
>
> What kernel version? I'll try to reproduce it.
>
> BTW, you might try this patch to symbols.c:
>
> https://www.redhat.com/archives/crash-utility/2011-December/msg00043.html
>
> > I could not found out corresponding condition from gdb-7.3.1/
> > but maybe -readnow is simple and proper way for this problem.
> > I've tested and got the good result.
> >
> > // sprintf(buf, "add-symbol-file %s 0x%lx",
> > sprintf(buf, "add-symbol-file %s 0x%lx
> > -readnow",
> > lm->mod_namelist, section_vaddr);
>
> Although I still don't know why the module line-number fixes that went
> into crash-6.0.1 wouldn't accomplish the same thing without the overhead
> of doing it for all symbols of all modules.
>
> Dave

So I tried to reproduce it -- at least with this 3.1-based kernel built
with gcc 4.6.2:

crash> help -k | grep proc_version
proc_version: Linux version 3.1.7-1.fc16.x86_64 (mockbuild@x86-18.phx2.fedoraproject.org) (gcc version 4.6.2 20111027 (Red Hat 4.6.2-1) (GCC) ) #1 SMP Tue Jan 3 19:45:05 UTC 2012
crash>

but module line numbers come out OK:

crash> mod -s bnx2 bnx2.ko.debug
MODULE NAME SIZE OBJECT FILE
ffffffffa00c69c0 bnx2 75492 bnx2.ko.debug
crash> dis -l bnx2_open
/usr/src/debug/kernel-3.1.fc16/linux-3.1.x86_64/drivers/net/bnx2.c: 6265
0xffffffffa00c0e80 <bnx2_open>: push %rbp
0xffffffffa00c0e81 <bnx2_open+1>: mov %rsp,%rbp
0xffffffffa00c0e84 <bnx2_open+4>: push %r15
0xffffffffa00c0e86 <bnx2_open+6>: push %r14
0xffffffffa00c0e88 <bnx2_open+8>: push %r13
0xffffffffa00c0e8a <bnx2_open+10>: push %r12
0xffffffffa00c0e8c <bnx2_open+12>: push %rbx
0xffffffffa00c0e8d <bnx2_open+13>: sub $0x8,%rsp
0xffffffffa00c0e91 <bnx2_open+17>: data32 data32 data32 xchg %ax,%ax
/usr/src/debug/kernel-3.1.fc16/linux-3.1.x86_64/drivers/net/bnx2.c: 6266
0xffffffffa00c0e96 <bnx2_open+22>: lea 0x740(%rdi),%r12
/usr/src/debug/kernel-3.1.fc16/linux-3.1.x86_64/drivers/net/bnx2.c: 6265
0xffffffffa00c0e9d <bnx2_open+29>: mov %rdi,%rbx
...

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

01-30-2012, 09:05 PM

On Thu, 2012-01-26 at 15:24 -0500, Dave Anderson wrote:
>
> ----- Original Message -----
> >
> >
> > ----- Original Message -----
> > > I've reverted back to crash-5.1.9 and applied my kmem patch to that for
> > > our use here. We're using a 3.1-based kernel, and need the kmem patch
> > > so crash can deal with the change in CONFIG_SLAB, but we're building
> > > with gcc-4.4.5 and don't really need the new gdb in crash-6.0.2, and
> > > crash-6.0.2 is not giving module source line numbers for us with "dis -l".
> > >
> > > This is just a heads up. I don't know why 6.0.2 is failing this, and
> > > since I found the last module source line number problem, it's not my
> > > turn ;-)
> > >
> > > Bob Montgomery
> >
> > What kernel version? I'll try to reproduce it.

Here's your test on my system:

crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2 gdb version: 7.3.1

crash-6.0.2> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash-6.0.2> mod -s bnx2 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
MODULE NAME SIZE OBJECT FILE
ffffffffa01f85e0 bnx2 65655 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko

/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6265
0xffffffffa01f32e1 <bnx2_open>: push %rbp
0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>: push %r15
0xffffffffa01f32e7 <bnx2_open+6>: push %r14
0xffffffffa01f32e9 <bnx2_open+8>: push %r13
0xffffffffa01f32eb <bnx2_open+10>: push %r12
0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6266
0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6265
0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6269

Ummmm. Ooops? Is it time to prepare my embarrassing explanation of my
prior claim???

=====================

First start over and do what I actually was doing when I saw the
problem:

$ crash-6.0.2 --no_kmem_cache dump.201201062015 kernel_link
...

crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2 gdb version: 7.3.1

crash> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash-6.0.2> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...

crash-6.0.2> mod | grep sunrpc
ffffffffa01cab80 auth_rpcgss
40572 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
ffffffffa03a7070 sunrpc
202679 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/sunrpc.ko

crash-6.0.2> dis -l rpc_sleep_on_priority
0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov 0x70(%rsi),%rax
0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
0xffffffffa038f738
0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2

Well, that's a relief...

Oh, by the way, now that I've used mod -S to load all modules instead of
individually loading bnx2.ko:

crash-6.0.2> mod | grep bnx2
ffffffffa01f85e0 bnx2 65655 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko

crash-6.0.2> dis -l bnx2_open
0xffffffffa01f32e1 <bnx2_open>: push %rbp
0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>: push %r15
0xffffffffa01f32e7 <bnx2_open+6>: push %r14
0xffffffffa01f32e9 <bnx2_open+8>: push %r13
0xffffffffa01f32eb <bnx2_open+10>: push %r12
0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
0xffffffffa01f32fc <bnx2_open+27>: callq 0xffffffff8129477b <netif_carrier_off>

No source lines.

===================

If I do the same steps on my 5.1.9 version of crash, I get:

crash> help | grep version
crash version: 5.1.9 gdb version: 7.0

crash> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...

crash> dis -l rpc_sleep_on_priority
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c: 350
0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/arch/x86/include/asm/bitops.h: 312
0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov 0x70(%rsi),%rax
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c: 352
0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
0xffffffffa038f738 <rpc_sleep_on_priority+29>
0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2a

So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
That's interesting.

Bob Montgomery

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

01-31-2012, 01:16 PM

Hi Bob,

OK, I'm currently provisioning a fresh Fedora 16 system and
I'll retry it.

I wonder if it has something to do with this crash-6.0.1 update:

- If the "--mod <directory-tree>" command line option, or the
setting of the CRASH_MODULE_PATH environment variable, or the
"mod -S <directory-tree>" point to a tree that contains only the
separate debuginfo "<module>.ko.debug" files, then those
debuginfo files will be used as the internal "add-symbol-file"
arguments to the embedded gdb module. Without the patch, it was
only acceptable to point to a directory tree that contained the
base "<module>.ko" files, and the separate debuginfo files
were found automatically based upon the directory path to the
base module file. This will allow an alternate module-debuginfo
directory tree to be set up like so:

# cd <directory>
# rpm2cpio kernel-debuginfo-<release>.rpm | cpio -idv

Having done that, the <directory> may be used with the "--mod",
command line argument, or as the CRASH_MODULE_PATH environment
variable, or as the "mod -S <directory> argument.
(anderson@redhat.com)

I'll check it myself, but I wonder what happens if you pass "mod -S"
the directory tree containing the <module>.ko.debug files instead of
the base <module>.ko files? It seems to be acting as if line between
the base <module>.ko and its <module>.ko.debug is not being recognized.
But then again, why would it work when individually loaded?

BTW, how's the kmem patch looking? I'm guessing it's more involved
than you first thought?

Thanks,
Dave

----- Original Message -----
> On Thu, 2012-01-26 at 15:24 -0500, Dave Anderson wrote:
> >
> > ----- Original Message -----
> > >
> > >
> > > ----- Original Message -----
> > > > I've reverted back to crash-5.1.9 and applied my kmem patch to
> > > > that for
> > > > our use here. We're using a 3.1-based kernel, and need the
> > > > kmem patch
> > > > so crash can deal with the change in CONFIG_SLAB, but we're
> > > > building
> > > > with gcc-4.4.5 and don't really need the new gdb in
> > > > crash-6.0.2, and
> > > > crash-6.0.2 is not giving module source line numbers for us
> > > > with "dis -l".
> > > >
> > > > This is just a heads up. I don't know why 6.0.2 is failing
> > > > this, and
> > > > since I found the last module source line number problem, it's
> > > > not my
> > > > turn ;-)
> > > >
> > > > Bob Montgomery
> > >
> > > What kernel version? I'll try to reproduce it.
>
> Here's your test on my system:
>
> crash-6.0.2> help | grep version
> crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
>
> crash-6.0.2> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash-6.0.2> mod -s bnx2
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
> MODULE NAME SIZE OBJECT FILE
> ffffffffa01f85e0 bnx2 65655
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
>
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6265
> 0xffffffffa01f32e1 <bnx2_open>: push %rbp
> 0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
> 0xffffffffa01f32e5 <bnx2_open+4>: push %r15
> 0xffffffffa01f32e7 <bnx2_open+6>: push %r14
> 0xffffffffa01f32e9 <bnx2_open+8>: push %r13
> 0xffffffffa01f32eb <bnx2_open+10>: push %r12
> 0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
> 0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6266
> 0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6265
> 0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6269
>
> Ummmm. Ooops? Is it time to prepare my embarrassing explanation of
> my
> prior claim???
>
> =====================
>
> First start over and do what I actually was doing when I saw the
> problem:
>
>
> $ crash-6.0.2 --no_kmem_cache dump.201201062015 kernel_link
> ...
>
> crash-6.0.2> help | grep version
> crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
>
> crash> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash-6.0.2> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
> ...
>
> crash-6.0.2> mod | grep sunrpc
> ffffffffa01cab80 auth_rpcgss
> 40572
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
> ffffffffa03a7070 sunrpc
> 202679
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/sunrpc.ko
>
> crash-6.0.2> dis -l rpc_sleep_on_priority
> 0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
> 0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
> 0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
> 0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
> 0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
> 0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
> 0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
> 0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
> 0x70(%rsi),%rax
> 0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
> 0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
> 0xffffffffa038f738
> 0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2
>
> Well, that's a relief...
>
> Oh, by the way, now that I've used mod -S to load all modules instead
> of
> individually loading bnx2.ko:
>
> crash-6.0.2> mod | grep bnx2
> ffffffffa01f85e0 bnx2 65655
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
>
>
> crash-6.0.2> dis -l bnx2_open
> 0xffffffffa01f32e1 <bnx2_open>: push %rbp
> 0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
> 0xffffffffa01f32e5 <bnx2_open+4>: push %r15
> 0xffffffffa01f32e7 <bnx2_open+6>: push %r14
> 0xffffffffa01f32e9 <bnx2_open+8>: push %r13
> 0xffffffffa01f32eb <bnx2_open+10>: push %r12
> 0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
> 0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
> 0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
> 0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
> 0xffffffffa01f32fc <bnx2_open+27>: callq 0xffffffff8129477b
> <netif_carrier_off>
>
> No source lines.
>
> ===================
>
> If I do the same steps on my 5.1.9 version of crash, I get:
>
> crash> help | grep version
> crash version: 5.1.9 gdb version: 7.0
>
> crash> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
> ...
>
> crash> dis -l rpc_sleep_on_priority
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
> 350
> 0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
> 0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
> 0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
> 0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
> 0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
> 0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
> 0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/arch/x86/include/asm/bitops.h:
> 312
> 0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
> 0x70(%rsi),%rax
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
> 352
> 0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
> 0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
> 0xffffffffa038f738 <rpc_sleep_on_priority+29>
> 0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2a
>
> So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
> That's interesting.
>
> Bob Montgomery
>
>
>
>
>

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

01-31-2012, 02:16 PM

----- Original Message -----
>
> So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
> That's interesting.
>
> Bob Montgomery

Strange -- here "mod -S" alone works OK on a freshly-installed F16 kernel:

# /usr/bin/crash

crash 6.0.2-1.fc16
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/vmlinux
DUMPFILE: /dev/crash
CPUS: 4
DATE: Tue Jan 31 09:51:19 2012
UPTIME: 00:43:03
LOAD AVERAGE: 0.13, 0.14, 0.19
TASKS: 110
NODENAME: dell-pec5125-02.lab.bos.redhat.com
RELEASE: 3.2.2-1.fc16.x86_64
VERSION: #1 SMP Thu Jan 26 03:21:58 UTC 2012
MACHINE: x86_64 (2599 Mhz)
MEMORY: 2 GB
PID: 32235
COMMAND: "crash"
TASK: ffff88007ba8ae40 [THREAD_INFO: ffff88003e310000]
CPU: 3
STATE: TASK_RUNNING (ACTIVE)

crash> mod -S
MODULE NAME SIZE OBJECT FILE
ffffffffa0002420 pata_atiixp 13197 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_atiixp.ko
ffffffffa000a5c0 ata_generic 12899 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/ata_generic.ko
ffffffffa0012420 pata_acpi 13027 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_acpi.ko
ffffffffa0041580 sunrpc 235041 /lib/modules/3.2.2-1.fc16.x86_64/kernel/net/sunrpc/sunrpc.ko
ffffffffa0055360 dcdbas 14828 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/firmware/dcdbas.ko
ffffffffa005a1e0 k10temp 13119 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/hwmon/k10temp.ko
ffffffffa00600a0 joydev 17412 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/input/joydev.ko
ffffffffa0069140 edac_mce_amd 22882 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_mce_amd.ko
ffffffffa0073f20 edac_core 51619 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_core.ko
ffffffffa007c060 dca 14601 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/dca/dca.ko
ffffffffa00834c0 amd64_edac_mod 23795 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/amd64_edac_mod.ko
ffffffffa008c600 i2c_core 37955 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/i2c-core.ko
ffffffffa0093cc0 i2c_piix4 13694 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/busses/i2c-piix4.ko
ffffffffa0098120 sp5100_tco 13697 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/watchdog/sp5100_tco.ko
ffffffffa009d140 crash 12576 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/char/crash.ko
ffffffffa00a7380 microcode 23240 /lib/modules/3.2.2-1.fc16.x86_64/kernel/arch/x86/kernel/microcode.ko
ffffffffa00ba9a0 lockd 84578 /lib/modules/3.2.2-1.fc16.x86_64/kernel/fs/lockd/lockd.ko
ffffffffa00e3000 igb 147890 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/net/ethernet/intel/igb/igb.ko
crash> dis -l rpc_sleep_on_priority 10
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022650 <rpc_sleep_on_priority>: push %rbp
0xffffffffa0022651 <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa0022654 <rpc_sleep_on_priority+4>: sub $0x20,%rsp
0xffffffffa0022658 <rpc_sleep_on_priority+8>: mov %rbx,-0x20(%rbp)
0xffffffffa002265c <rpc_sleep_on_priority+12>: mov %r12,-0x18(%rbp)
0xffffffffa0022660 <rpc_sleep_on_priority+16>: mov %r13,-0x10(%rbp)
0xffffffffa0022664 <rpc_sleep_on_priority+20>: mov %r14,-0x8(%rbp)
0xffffffffa0022668 <rpc_sleep_on_priority+24>: data32 data32 data32 xchg %ax,%ax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/arch/x86/include/asm/bitops.h: 312
0xffffffffa002266d <rpc_sleep_on_priority+29>: mov 0x70(%rsi),%rax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022671 <rpc_sleep_on_priority+33>: mov %rdi,%rbx
crash>

So then I tried using "mod -S <path>" using the path containing
the <module>.ko.debug files, and that works OK:

# /usr/bin/crash

crash 6.0.2-1.fc16
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/vmlinux
DUMPFILE: /dev/crash
CPUS: 4
DATE: Tue Jan 31 10:06:05 2012
UPTIME: 00:57:50
LOAD AVERAGE: 0.01, 0.04, 0.10
TASKS: 111
NODENAME: dell-pec5125-02.lab.bos.redhat.com
RELEASE: 3.2.2-1.fc16.x86_64
VERSION: #1 SMP Thu Jan 26 03:21:58 UTC 2012
MACHINE: x86_64 (2599 Mhz)
MEMORY: 2 GB
PID: 1445
COMMAND: "crash"
TASK: ffff880078410000 [THREAD_INFO: ffff88003e310000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)

crash> mod -S /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64
MODULE NAME SIZE OBJECT FILE
ffffffffa0002420 pata_atiixp 13197 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_atiixp.ko.debug
ffffffffa000a5c0 ata_generic 12899 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/ata_generic.ko.debug
ffffffffa0012420 pata_acpi 13027 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_acpi.ko.debug
ffffffffa0041580 sunrpc 235041 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/net/sunrpc/sunrpc.ko.debug
ffffffffa0055360 dcdbas 14828 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/firmware/dcdbas.ko.debug
ffffffffa005a1e0 k10temp 13119 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/hwmon/k10temp.ko.debug
ffffffffa00600a0 joydev 17412 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/input/joydev.ko.debug
ffffffffa0069140 edac_mce_amd 22882 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_mce_amd.ko.debug
ffffffffa0073f20 edac_core 51619 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_core.ko.debug
ffffffffa007c060 dca 14601 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/dca/dca.ko.debug
ffffffffa00834c0 amd64_edac_mod 23795 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/amd64_edac_mod.ko.debug
ffffffffa008c600 i2c_core 37955 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/i2c-core.ko.debug
ffffffffa0093cc0 i2c_piix4 13694 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/busses/i2c-piix4.ko.debug
ffffffffa0098120 sp5100_tco 13697 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/watchdog/sp5100_tco.ko.debug
ffffffffa00a7380 microcode 23240 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/arch/x86/kernel/microcode.ko.debug
ffffffffa00ba9a0 lockd 84578 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/fs/lockd/lockd.ko.debug
ffffffffa00e3000 igb 147890 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/net/ethernet/intel/igb/igb.ko.debug
ffffffffa00ed140 crash 12576 /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/char/crash.ko.debug
crash> dis -l rpc_sleep_on_priority 10
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022650 <rpc_sleep_on_priority>: push %rbp
0xffffffffa0022651 <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa0022654 <rpc_sleep_on_priority+4>: sub $0x20,%rsp
0xffffffffa0022658 <rpc_sleep_on_priority+8>: mov %rbx,-0x20(%rbp)
0xffffffffa002265c <rpc_sleep_on_priority+12>: mov %r12,-0x18(%rbp)
0xffffffffa0022660 <rpc_sleep_on_priority+16>: mov %r13,-0x10(%rbp)
0xffffffffa0022664 <rpc_sleep_on_priority+20>: mov %r14,-0x8(%rbp)
0xffffffffa0022668 <rpc_sleep_on_priority+24>: data32 data32 data32 xchg %ax,%ax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/arch/x86/include/asm/bitops.h: 312
0xffffffffa002266d <rpc_sleep_on_priority+29>: mov 0x70(%rsi),%rax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022671 <rpc_sleep_on_priority+33>: mov %rdi,%rbx
crash>

And then finally, with "mod -S" pointing to the base "/lib/modules" path:

# /usr/bin/crash

crash 6.0.2-1.fc16
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/lib/modules/3.2.2-1.fc16.x86_64/vmlinux
DUMPFILE: /dev/crash
CPUS: 4
DATE: Tue Jan 31 10:11:35 2012
UPTIME: 01:03:20
LOAD AVERAGE: 0.24, 0.10, 0.10
TASKS: 111
NODENAME: dell-pec5125-02.lab.bos.redhat.com
RELEASE: 3.2.2-1.fc16.x86_64
VERSION: #1 SMP Thu Jan 26 03:21:58 UTC 2012
MACHINE: x86_64 (2599 Mhz)
MEMORY: 2 GB
PID: 1799
COMMAND: "crash"
TASK: ffff88007ba8c560 [THREAD_INFO: ffff88003e310000]
CPU: 3
STATE: TASK_RUNNING (ACTIVE)

crash> mod -S /lib/modules/3.2.2-1.fc16.x86_64
MODULE NAME SIZE OBJECT FILE
ffffffffa0002420 pata_atiixp 13197 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_atiixp.ko
ffffffffa000a5c0 ata_generic 12899 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/ata_generic.ko
ffffffffa0012420 pata_acpi 13027 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/ata/pata_acpi.ko
ffffffffa0041580 sunrpc 235041 /lib/modules/3.2.2-1.fc16.x86_64/kernel/net/sunrpc/sunrpc.ko
ffffffffa0055360 dcdbas 14828 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/firmware/dcdbas.ko
ffffffffa005a1e0 k10temp 13119 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/hwmon/k10temp.ko
ffffffffa00600a0 joydev 17412 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/input/joydev.ko
ffffffffa0069140 edac_mce_amd 22882 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_mce_amd.ko
ffffffffa0073f20 edac_core 51619 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/edac_core.ko
ffffffffa007c060 dca 14601 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/dca/dca.ko
ffffffffa00834c0 amd64_edac_mod 23795 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/edac/amd64_edac_mod.ko
ffffffffa008c600 i2c_core 37955 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/i2c-core.ko
ffffffffa0093cc0 i2c_piix4 13694 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/i2c/busses/i2c-piix4.ko
ffffffffa0098120 sp5100_tco 13697 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/watchdog/sp5100_tco.ko
ffffffffa00a7380 microcode 23240 /lib/modules/3.2.2-1.fc16.x86_64/kernel/arch/x86/kernel/microcode.ko
ffffffffa00ba9a0 lockd 84578 /lib/modules/3.2.2-1.fc16.x86_64/kernel/fs/lockd/lockd.ko
ffffffffa00e3000 igb 147890 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/net/ethernet/intel/igb/igb.ko
ffffffffa00ed140 crash 12576 /lib/modules/3.2.2-1.fc16.x86_64/kernel/drivers/char/crash.ko
crash> dis -l rpc_sleep_on_priority 10
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022650 <rpc_sleep_on_priority>: push %rbp
0xffffffffa0022651 <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa0022654 <rpc_sleep_on_priority+4>: sub $0x20,%rsp
0xffffffffa0022658 <rpc_sleep_on_priority+8>: mov %rbx,-0x20(%rbp)
0xffffffffa002265c <rpc_sleep_on_priority+12>: mov %r12,-0x18(%rbp)
0xffffffffa0022660 <rpc_sleep_on_priority+16>: mov %r13,-0x10(%rbp)
0xffffffffa0022664 <rpc_sleep_on_priority+20>: mov %r14,-0x8(%rbp)
0xffffffffa0022668 <rpc_sleep_on_priority+24>: data32 data32 data32 xchg %ax,%ax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/arch/x86/include/asm/bitops.h: 312
0xffffffffa002266d <rpc_sleep_on_priority+29>: mov 0x70(%rsi),%rax
/usr/src/debug/kernel-3.2.fc16/linux-3.2.x86_64/net/sunrpc/sched.c: 351
0xffffffffa0022671 <rpc_sleep_on_priority+33>: mov %rdi,%rbx
crash>

I'll keep trying...

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility