FAQ Search Today's Posts Mark Forums Read

» Linux Archive
Home
New Posts
Search
FAQ


Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 04-30-2008, 10:43 PM
Dave Anderson
 
Default source line numbers and modules (on x86_64)

Mike Snitzer wrote:

Hi,

I searched the archives and found that you've discussed an issue I'm
seeing with x86_64 kernels where crash doesn't have line numbers for
modules' symbols:
https://www.redhat.com/archives/crash-utility/2008-January/msg00021.html

I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a custom
2.6.22.19 kernel. Given that the RHEL5U1 x86_64 kernels clearly do
provide accurate line numbers for modules, has anyone identified how
that is? I have to believe the redhat kernel is patched to fix this
issue.

I looked over the various redhat patches that are applied to RHEL5's
2.6.18 sources but can't see a patch that stands out as specifically
addressing this x86_64 issue. But I could easily be overlooking some
patch.

please advise, thanks.
Mike

ps. please cc me as I've not yet been able to join the list


You're on the list now...

Anyway, there are no kernel patches applied to RHEL5 kernels
to make it work that I'm aware of. That being said, I don't
have an answer as to why they don't work for modules on your
custom kernel (or why they don't work in RHEL4 kernels...).

Dave


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 04-30-2008, 11:08 PM
"Mike Snitzer"
 
Default source line numbers and modules (on x86_64)

On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > Hi,
> >
> > I searched the archives and found that you've discussed an issue I'm
> > seeing with x86_64 kernels where crash doesn't have line numbers for
> > modules' symbols:
> > https://www.redhat.com/archives/crash-utility/2008-January/msg00021.html
> >
> > I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a custom
> > 2.6.22.19 kernel. Given that the RHEL5U1 x86_64 kernels clearly do
> > provide accurate line numbers for modules, has anyone identified how
> > that is? I have to believe the redhat kernel is patched to fix this
> > issue.
> >
> > I looked over the various redhat patches that are applied to RHEL5's
> > 2.6.18 sources but can't see a patch that stands out as specifically
> > addressing this x86_64 issue. But I could easily be overlooking some
> > patch.
> >
> > please advise, thanks.
> > Mike
> >
> > ps. please cc me as I've not yet been able to join the list
> >
> >
> You're on the list now...
>
> Anyway, there are no kernel patches applied to RHEL5 kernels
> to make it work that I'm aware of. That being said, I don't
> have an answer as to why they don't work for modules on your
> custom kernel (or why they don't work in RHEL4 kernels...).

I configured my x86_64 kernel to be relocatable. Could this be the
reason for modules' line number debug data being off? From:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5

"Using /proc/kallsyms or a --reloc=[size] as a command line argument
is similar to using a System.map file, in that it results in the loss
of the use of line number debug data."

Still doesn't explain why RHEL4 isn't happy in this regard either...

I'm trying a non-relocatable variant of my kernel now and will report
back with the results.

Mike

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 04-30-2008, 11:41 PM
Dave Anderson
 
Default source line numbers and modules (on x86_64)

Mike Snitzer wrote:

On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com> wrote:


Mike Snitzer wrote:



Hi,

I searched the archives and found that you've discussed an issue I'm
seeing with x86_64 kernels where crash doesn't have line numbers for
modules' symbols:
https://www.redhat.com/archives/crash-utility/2008-January/msg00021.html

I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a custom
2.6.22.19 kernel. Given that the RHEL5U1 x86_64 kernels clearly do
provide accurate line numbers for modules, has anyone identified how
that is? I have to believe the redhat kernel is patched to fix this
issue.

I looked over the various redhat patches that are applied to RHEL5's
2.6.18 sources but can't see a patch that stands out as specifically
addressing this x86_64 issue. But I could easily be overlooking some
patch.

please advise, thanks.
Mike

ps. please cc me as I've not yet been able to join the list




You're on the list now...

Anyway, there are no kernel patches applied to RHEL5 kernels
to make it work that I'm aware of. That being said, I don't
have an answer as to why they don't work for modules on your
custom kernel (or why they don't work in RHEL4 kernels...).



I configured my x86_64 kernel to be relocatable. Could this be the
reason for modules' line number debug data being off? From:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5

"Using /proc/kallsyms or a --reloc=[size] as a command line argument
is similar to using a System.map file, in that it results in the loss
of the use of line number debug data."



No, that particular issue only applies to the base kernel, and is
irrelevant with respect to modules.

(Note that the RHEL5 kernel is relocatable as well.)

Still doesn't explain why RHEL4 isn't happy in this regard either...

I'm trying a non-relocatable variant of my kernel now and will report
back with the results.



What you might confirm is whether you're getting any debuginfo data when
you load

the module? Are you sure that your modules are being built with -g?

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 05-01-2008, 12:08 AM
"Mike Snitzer"
 
Default source line numbers and modules (on x86_64)

On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson@redhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com>

> > > Anyway, there are no kernel patches applied to RHEL5 kernels
> > > to make it work that I'm aware of. That being said, I don't
> > > have an answer as to why they don't work for modules on your
> > > custom kernel (or why they don't work in RHEL4 kernels...).
> > >
> > >
> >
> > I configured my x86_64 kernel to be relocatable. Could this be the
> > reason for modules' line number debug data being off? From:
> > http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
> >
> > "Using /proc/kallsyms or a --reloc=[size] as a command line argument
> > is similar to using a System.map file, in that it results in the loss
> > of the use of line number debug data."
> >
> >
>
> No, that particular issue only applies to the base kernel, and is
> irrelevant with respect to modules.
> (Note that the RHEL5 kernel is relocatable as well.)

OK, yeah making the base kernel non-relocatable didn't help.

> What you might confirm is whether you're getting any debuginfo data when
> you load the module? Are you sure that your modules are being built with -g?

How would I confirm that I'm getting debuginfo data when I 'mod -s
ext3'? I'm using an spec file that is based heavily (at least
relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to
verify that I'm building with -g...

Mike

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 05-01-2008, 12:21 AM
Dave Anderson
 
Default source line numbers and modules (on x86_64)

Mike Snitzer wrote:

On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson@redhat.com> wrote:


Mike Snitzer wrote:



On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com>





Anyway, there are no kernel patches applied to RHEL5 kernels
to make it work that I'm aware of. That being said, I don't
have an answer as to why they don't work for modules on your
custom kernel (or why they don't work in RHEL4 kernels...).




I configured my x86_64 kernel to be relocatable. Could this be the
reason for modules' line number debug data being off? From:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5

"Using /proc/kallsyms or a --reloc=[size] as a command line argument
is similar to using a System.map file, in that it results in the loss
of the use of line number debug data."




No, that particular issue only applies to the base kernel, and is
irrelevant with respect to modules.
(Note that the RHEL5 kernel is relocatable as well.)



OK, yeah making the base kernel non-relocatable didn't help.



What you might confirm is whether you're getting any debuginfo data when
you load the module? Are you sure that your modules are being built with -g?



How would I confirm that I'm getting debuginfo data when I 'mod -s
ext3'? I'm using an spec file that is based heavily (at least
relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to
verify that I'm building with -g...

Mike



So your build procedure ends up with a kernel-debuginfo package that
contains
a set of "split" <module-name>.ko.debug files? If you take the module
filename
displayed after the "mod -s" is done, and run it through "readelf -a",
you'll see
a .gnu_debuglink section. That /lib/modules/<path-to>/module.ko module
will have
a link to its partner in
/usr/lib/debug/lib/modules/<path-to>/module.ko.debug.


Dave



--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 05-01-2008, 01:17 AM
"Mike Snitzer"
 
Default source line numbers and modules (on x86_64)

On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson <anderson@redhat.com> wrote:
>
> Mike Snitzer wrote:
>
> > On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson@redhat.com>
> wrote:
> >
> >
> > > Mike Snitzer wrote:
> > >
> > >
> > >
> > > > On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com>
> > > >
> > > >
> > >
> >
> >
> >
> > >
> > > >
> > > > > Anyway, there are no kernel patches applied to RHEL5 kernels
> > > > > to make it work that I'm aware of. That being said, I don't
> > > > > have an answer as to why they don't work for modules on your
> > > > > custom kernel (or why they don't work in RHEL4 kernels...).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > I configured my x86_64 kernel to be relocatable. Could this be the
> > > > reason for modules' line number debug data being off? From:
> > > > http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
> > > >
> > > > "Using /proc/kallsyms or a --reloc=[size] as a command line argument
> > > > is similar to using a System.map file, in that it results in the loss
> > > > of the use of line number debug data."
> > > >
> > > >
> > > >
> > > >
> > > No, that particular issue only applies to the base kernel, and is
> > > irrelevant with respect to modules.
> > > (Note that the RHEL5 kernel is relocatable as well.)
> > >
> > >
> >
> > OK, yeah making the base kernel non-relocatable didn't help.
> >
> >
> >
> > > What you might confirm is whether you're getting any debuginfo data
> when
> > > you load the module? Are you sure that your modules are being built
> with -g?
> > >
> > >
> >
> > How would I confirm that I'm getting debuginfo data when I 'mod -s
> > ext3'? I'm using an spec file that is based heavily (at least
> > relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to
> > verify that I'm building with -g...
> >
> > Mike
> >
> >
>
> So your build procedure ends up with a kernel-debuginfo package that
> contains a set of "split" <module-name>.ko.debug files?

Yes.

> If you take the module filename displayed after the "mod -s" is done,
> and run it through "readelf -a", you'll see a .gnu_debuglink section.

My kernel's ext3.ko has:
[23] .gnu_debuglink PROGBITS 0000000000000000 00036558
0000000000000014 0000000000000000 0 0 4

which would appear fairly comparable to what the RHEL5 kernel's ext3.ko has:
[22] .gnu_debuglink PROGBITS 0000000000000000 00037418
0000000000000014 0000000000000000 0 0 4

> That /lib/modules/<path-to>/module.ko module will have
> a link to its partner in
> /usr/lib/debug/lib/modules/<path-to>/module.ko.debug.

readelf -x $SECTION /lib/modules/<path-to>/ext3.ko shows that
.gnu_debuglink points to ext3.ko.debug as expected.

Given the "so far so good" nature of what you're helping me verify
here it still begs the question: so why doesn't it work on my
2.6.22.19 x86_64 kernel?

Where does the line number debug data get stored in the
module.ko.debug? In my case the source comes up as
"include/linux/clocksource.h: 121" for all kernel modules' symbols.

Any further pointers would be greatly appreciated, thanks a lot for
your time already!
Mike

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 05-01-2008, 01:59 AM
Dave Anderson
 
Default source line numbers and modules (on x86_64)

Mike Snitzer wrote:

On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson <anderson@redhat.com> wrote:


Mike Snitzer wrote:



On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson@redhat.com>


wrote:



Mike Snitzer wrote:





On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson@redhat.com>







Anyway, there are no kernel patches applied to RHEL5 kernels
to make it work that I'm aware of. That being said, I don't
have an answer as to why they don't work for modules on your
custom kernel (or why they don't work in RHEL4 kernels...).






I configured my x86_64 kernel to be relocatable. Could this be the
reason for modules' line number debug data being off? From:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5

"Using /proc/kallsyms or a --reloc=[size] as a command line argument
is similar to using a System.map file, in that it results in the loss
of the use of line number debug data."






No, that particular issue only applies to the base kernel, and is
irrelevant with respect to modules.
(Note that the RHEL5 kernel is relocatable as well.)




OK, yeah making the base kernel non-relocatable didn't help.





What you might confirm is whether you're getting any debuginfo data


when


you load the module? Are you sure that your modules are being built


with -g?



How would I confirm that I'm getting debuginfo data when I 'mod -s
ext3'? I'm using an spec file that is based heavily (at least
relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to
verify that I'm building with -g...

Mike




So your build procedure ends up with a kernel-debuginfo package that
contains a set of "split" <module-name>.ko.debug files?



Yes.



If you take the module filename displayed after the "mod -s" is done,
and run it through "readelf -a", you'll see a .gnu_debuglink section.



My kernel's ext3.ko has:
[23] .gnu_debuglink PROGBITS 0000000000000000 00036558
0000000000000014 0000000000000000 0 0 4

which would appear fairly comparable to what the RHEL5 kernel's ext3.ko has:
[22] .gnu_debuglink PROGBITS 0000000000000000 00037418
0000000000000014 0000000000000000 0 0 4



That /lib/modules/<path-to>/module.ko module will have
a link to its partner in
/usr/lib/debug/lib/modules/<path-to>/module.ko.debug.



readelf -x $SECTION /lib/modules/<path-to>/ext3.ko shows that
.gnu_debuglink points to ext3.ko.debug as expected.

Given the "so far so good" nature of what you're helping me verify
here it still begs the question: so why doesn't it work on my
2.6.22.19 x86_64 kernel?

Where does the line number debug data get stored in the
module.ko.debug? In my case the source comes up as
"include/linux/clocksource.h: 121" for all kernel modules' symbols.

Any further pointers would be greatly appreciated, thanks a lot for
your time already!
Mike



In the crash sources, gdb-6.1/gdb/symtab.c contains several crash
utility "hooks" into
the native gdb code. The function called gdb_get_line_number() calls
the pre-existing
gdb find_pc_line() function to get the line number information. As to
why it works

with some kernels and not with others is the question at hand, and to which
I have no answer. The behavior you see is similar to what happens with
x86_64 in RHEL4 kernels, although on my RHEL4 x86_64 machine, it always
seems to return a line in include/linux/seqlock.h (?). In any case, the
module
debuginfo data contains a linkage between text offset values to the
source code line
numbers. In other words, do an "nm -Bn <module>.ko" and you'll see that
module
text locations are offsets, as opposed to compiled-in addresses that you
see if you
do an "nm -Bn vmlinux". So my guess is that the problem is related to
the fact that
the module objects are relocatable, where the module object has offset
values applied
to each text offset, whereas the the PC values that get passed to the
gdb code

are the kernel vmalloc()'d addresses where the module got loaded. When the
"mod" command is done, it does a gdb "add-symbol-file" command, which
essentially transforms of the PC offset values in the module object into the
loaded vmalloc address. But why the resultant vmalloc() addresses work for
some kernels and not others is a mystery to me.

Dave





--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 01:49 PM.

VBulletin, Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org