FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 08-13-2012, 06:07 PM
Mark Tinguely
 
Default Crash on Linux 3.6 rc1

I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux 3.5.

Thanks,

--Mark Tinguely.

------

~/xfs # crash System.map vmlinux

crash 6.0.8
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...


crash: invalid structure member offset: kmem_cache_s_next
FILE: memory.c LINE: 7945 FUNCTION: kmem_cache_init()

[/usr/bin/crash] error trace: 468317 => 49dbb2 => 487f28 => 5083da

5083da: OFFSET_verify+202
487f28: kmem_cache_init+312
49dbb2: vm_init+5794
468317: main_loop+215

~/xfs # cat /proc/version
Linux version 3.6.0-rc1 (root@cxfsxe12) (gcc version 4.3.4
[gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Fri Aug 10
17:03:36 CDT 2012


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-13-2012, 06:56 PM
Dave Anderson
 
Default Crash on Linux 3.6 rc1

----- Original Message -----
> I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux 3.5.
>
> Thanks,
>
> --Mark Tinguely.
>
> ------
>
> ~/xfs # crash System.map vmlinux
>
> crash 6.0.8
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
>
> crash: invalid structure member offset: kmem_cache_s_next
> FILE: memory.c LINE: 7945 FUNCTION: kmem_cache_init()
>
> [/usr/bin/crash] error trace: 468317 => 49dbb2 => 487f28 => 5083da
>
> 5083da: OFFSET_verify+202
> 487f28: kmem_cache_init+312
> 49dbb2: vm_init+5794
> 468317: main_loop+215
>
> ~/xfs # cat /proc/version
> Linux version 3.6.0-rc1 (root@cxfsxe12) (gcc version 4.3.4
> [gcc-4_3-branch revision 152973] (SUSE Linux) ) #1 SMP Fri Aug 10
> 17:03:36 CDT 2012
>

Mark,

It's this part of commit 3b0efdfa1e719303536c04d9abca43abeb40f80a
that changed the kmem_cache.next member from "next" to "list":

diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h
index fbd1117..1d93f27 100644
--- a/include/linux/slab_def.h
+++ b/include/linux/slab_def.h
@@ -27,7 +27,7 @@ struct kmem_cache {
unsigned int limit;
unsigned int shared;

- unsigned int buffer_size;
+ unsigned int size;
u32 reciprocal_buffer_size;
/* 2) touched by every alloc & free from the backend */

@@ -52,7 +52,10 @@ struct kmem_cache {

/* 4) cache creation/removal */
const char *name;
- struct list_head next;
+ struct list_head list;
+ int refcount;
+ int object_size;
+ int align;

/* 5) statistics */
#ifdef CONFIG_DEBUG_SLAB

I'm curious whether if you change line 505 in memory.c:

502 } else {
503 STRUCT_SIZE_INIT(kmem_cache_s, "kmem_cache");
504 MEMBER_OFFSET_INIT(kmem_cache_s_num, "kmem_cache", "num");
505 =====> MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache", "next");
506 MEMBER_OFFSET_INIT(kmem_cache_s_name, "kmem_cache", "name");
507 MEMBER_OFFSET_INIT(kmem_cache_s_colour_off, "kmem_cache",
508 "colour_off");

to this:

MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache", "list");

that it might work?

Worse case, you can work around it by using the "--no_kmem_cache" option
on the crash command line.

Thanks,
Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-13-2012, 08:52 PM
Mark Tinguely
 
Default Crash on Linux 3.6 rc1

On 08/13/12 13:56, Dave Anderson wrote:



----- Original Message -----

I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux 3.5.


...



crash: invalid structure member offset: kmem_cache_s_next
FILE: memory.c LINE: 7945 FUNCTION: kmem_cache_init()

[/usr/bin/crash] error trace: 468317 => 49dbb2 => 487f28 => 5083da

5083da: OFFSET_verify+202
487f28: kmem_cache_init+312
49dbb2: vm_init+5794
468317: main_loop+215



...



It's this part of commit 3b0efdfa1e719303536c04d9abca43abeb40f80a
that changed the kmem_cache.next member from "next" to "list":

diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h
index fbd1117..1d93f27 100644
--- a/include/linux/slab_def.h
+++ b/include/linux/slab_def.h
@@ -27,7 +27,7 @@ struct kmem_cache {
unsigned int limit;
unsigned int shared;

- unsigned int buffer_size;
+ unsigned int size;
u32 reciprocal_buffer_size;
/* 2) touched by every alloc& free from the backend */

@@ -52,7 +52,10 @@ struct kmem_cache {

/* 4) cache creation/removal */
const char *name;
- struct list_head next;
+ struct list_head list;
+ int refcount;
+ int object_size;
+ int align;

/* 5) statistics */
#ifdef CONFIG_DEBUG_SLAB

I'm curious whether if you change line 505 in memory.c:

502 } else {
503 STRUCT_SIZE_INIT(kmem_cache_s, "kmem_cache");
504 MEMBER_OFFSET_INIT(kmem_cache_s_num, "kmem_cache", "num");
505 =====> MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache", "next");
506 MEMBER_OFFSET_INIT(kmem_cache_s_name, "kmem_cache", "name");
507 MEMBER_OFFSET_INIT(kmem_cache_s_colour_off, "kmem_cache",
508 "colour_off");

to this:

MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache", "list");

that it might work?

Worse case, you can work around it by using the "--no_kmem_cache" option
on the crash command line.

Thanks,
Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility


The error changed to:

~/xfs # crash System.map vmlinux

crash 6.0.8
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...


crash: cannot resolve: "cache_chain"

--Mark.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-13-2012, 09:15 PM
Dave Anderson
 
Default Crash on Linux 3.6 rc1

----- Original Message -----
> On 08/13/12 13:56, Dave Anderson wrote:
> >
> >
> > ----- Original Message -----
> >> I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux
> >> 3.5.
>
> ...
>
> >>
> >> crash: invalid structure member offset: kmem_cache_s_next
> >> FILE: memory.c LINE: 7945 FUNCTION: kmem_cache_init()
> >>
> >> [/usr/bin/crash] error trace: 468317 => 49dbb2 => 487f28 =>
> >> 5083da
> >>
> >> 5083da: OFFSET_verify+202
> >> 487f28: kmem_cache_init+312
> >> 49dbb2: vm_init+5794
> >> 468317: main_loop+215
> >>
>
> ...
>
> >
> > It's this part of commit 3b0efdfa1e719303536c04d9abca43abeb40f80a
> > that changed the kmem_cache.next member from "next" to "list":
> >
> > diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h
> > index fbd1117..1d93f27 100644
> > --- a/include/linux/slab_def.h
> > +++ b/include/linux/slab_def.h
> > @@ -27,7 +27,7 @@ struct kmem_cache {
> > unsigned int limit;
> > unsigned int shared;
> >
> > - unsigned int buffer_size;
> > + unsigned int size;
> > u32 reciprocal_buffer_size;
> > /* 2) touched by every alloc& free from the backend */
> >
> > @@ -52,7 +52,10 @@ struct kmem_cache {
> >
> > /* 4) cache creation/removal */
> > const char *name;
> > - struct list_head next;
> > + struct list_head list;
> > + int refcount;
> > + int object_size;
> > + int align;
> >
> > /* 5) statistics */
> > #ifdef CONFIG_DEBUG_SLAB
> >
> > I'm curious whether if you change line 505 in memory.c:
> >
> > 502 } else {
> > 503 STRUCT_SIZE_INIT(kmem_cache_s,
> > "kmem_cache");
> > 504
> > MEMBER_OFFSET_INIT(kmem_cache_s_num,
> > "kmem_cache", "num");
> > 505 =====>
> > MEMBER_OFFSET_INIT(kmem_cache_s_next,
> > "kmem_cache", "next");
> > 506
> > MEMBER_OFFSET_INIT(kmem_cache_s_name,
> > "kmem_cache", "name");
> > 507
> > MEMBER_OFFSET_INIT(kmem_cache_s_colour_off,
> > "kmem_cache",
> > 508 "colour_off");
> >
> > to this:
> >
> > MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache",
> > "list");
> >
> > that it might work?
> >
> > Worse case, you can work around it by using the "--no_kmem_cache" option
> > on the crash command line.
> >
> > Thanks,
> > Dave
> >
> > --
> > Crash-utility mailing list
> > Crash-utility@redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> The error changed to:
>
> ~/xfs # crash System.map vmlinux
>
> crash 6.0.8
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
>
> crash: cannot resolve: "cache_chain"
>
> --Mark.

Yeah, that would have been too convenient...

So anyway, there's more to it than just the structure member name change.
For this error, it's because of this global data name change, from:

static struct list_head cache_chain;

to:

LIST_HEAD(slab_caches);

Cscope shows this as the result of a string search for "cache_chain":

0 memory.c 7744 int has_cache_chain;
1 memory.c 7745 ulong cache_chain;
2 memory.c 7758 if (symbol_exists("cache_chain")) {
3 memory.c 7759 has_cache_chain = TRUE;
4 memory.c 7760 cache_chain = symbol_value("cache_chain");
5 memory.c 7762 has_cache_chain = FALSE;
6 memory.c 7763 cache_chain = 0;
7 memory.c 7802 if (has_cache_chain && (cache == cache_chain))
8 memory.c 7804 "cache_chain", FAULT_ON_ERROR);
9 memory.c 7973 * slab chain list. 2.6 uses the "cache_chain" list_head.
a memory.c 7976 get_symbol_data("cache_chain", sizeof(ulong), &cache);
b memory.c 7978 cache_end = symbol_value("cache_chain");
c memory.c 8908 get_symbol_data("cache_chain", sizeof(ulong), &si->cache);
d memory.c 8910 cache_end = symbol_value("cache_chain");

So it would be a matter of recognizing tha "cache_chain" does not exist
as a symbol and "slab_caches" does, and baiting-and-switching as appropriate.

I'm not sure when I'm going to get the chance to look at this.
If anybody else on the list has some cycles to spare, fixing this
probably won't be a major undertaking.

Dave




--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-16-2012, 06:04 PM
Dave Anderson
 
Default Crash on Linux 3.6 rc1

----- Original Message -----
>
>
> ----- Original Message -----
> > On 08/13/12 13:56, Dave Anderson wrote:
> > >
> > >
> > > ----- Original Message -----
> > >> I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux
> > >> 3.5.
> >
> > ...
> >
> > >>
> > >> crash: invalid structure member offset: kmem_cache_s_next
> > >> FILE: memory.c LINE: 7945 FUNCTION: kmem_cache_init()
> > >>
> > >> [/usr/bin/crash] error trace: 468317 => 49dbb2 => 487f28 =>
> > >> 5083da
> > >>
> > >> 5083da: OFFSET_verify+202
> > >> 487f28: kmem_cache_init+312
> > >> 49dbb2: vm_init+5794
> > >> 468317: main_loop+215
> > >>
> >
> > ...
> >
> > >
> > > It's this part of commit 3b0efdfa1e719303536c04d9abca43abeb40f80a
> > > that changed the kmem_cache.next member from "next" to "list":
> > >
> > > diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h
> > > index fbd1117..1d93f27 100644
> > > --- a/include/linux/slab_def.h
> > > +++ b/include/linux/slab_def.h
> > > @@ -27,7 +27,7 @@ struct kmem_cache {
> > > unsigned int limit;
> > > unsigned int shared;
> > >
> > > - unsigned int buffer_size;
> > > + unsigned int size;
> > > u32 reciprocal_buffer_size;
> > > /* 2) touched by every alloc& free from the backend */
> > >
> > > @@ -52,7 +52,10 @@ struct kmem_cache {
> > >
> > > /* 4) cache creation/removal */
> > > const char *name;
> > > - struct list_head next;
> > > + struct list_head list;
> > > + int refcount;
> > > + int object_size;
> > > + int align;
> > >
> > > /* 5) statistics */
> > > #ifdef CONFIG_DEBUG_SLAB
> > >
> > > I'm curious whether if you change line 505 in memory.c:
> > >
> > > 502 } else {
> > > 503 STRUCT_SIZE_INIT(kmem_cache_s,
> > > "kmem_cache");
> > > 504
> > > MEMBER_OFFSET_INIT(kmem_cache_s_num,
> > > "kmem_cache", "num");
> > > 505 =====>
> > > MEMBER_OFFSET_INIT(kmem_cache_s_next,
> > > "kmem_cache", "next");
> > > 506
> > > MEMBER_OFFSET_INIT(kmem_cache_s_name,
> > > "kmem_cache", "name");
> > > 507
> > > MEMBER_OFFSET_INIT(kmem_cache_s_colour_off,
> > > "kmem_cache",
> > > 508 "colour_off");
> > >
> > > to this:
> > >
> > > MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache",
> > > "list");
> > >
> > > that it might work?
> > >
> > > Worse case, you can work around it by using the "--no_kmem_cache"
> > > option
> > > on the crash command line.
> > >
> > > Thanks,
> > > Dave
> > >
> > > --
> > > Crash-utility mailing list
> > > Crash-utility@redhat.com
> > > https://www.redhat.com/mailman/listinfo/crash-utility
> >
> > The error changed to:
> >
> > ~/xfs # crash System.map vmlinux
> >
> > crash 6.0.8
> > Copyright (C) 2002-2012 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.3.1
> > Copyright (C) 2011 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >
> >
> > crash: cannot resolve: "cache_chain"
> >
> > --Mark.
>
> Yeah, that would have been too convenient...
>
> So anyway, there's more to it than just the structure member name change.
> For this error, it's because of this global data name change, from:
>
> static struct list_head cache_chain;
>
> to:
>
> LIST_HEAD(slab_caches);
>
> Cscope shows this as the result of a string search for "cache_chain":
>
> 0 memory.c 7744 int has_cache_chain;
> 1 memory.c 7745 ulong cache_chain;
> 2 memory.c 7758 if (symbol_exists("cache_chain")) {
> 3 memory.c 7759 has_cache_chain = TRUE;
> 4 memory.c 7760 cache_chain = symbol_value("cache_chain");
> 5 memory.c 7762 has_cache_chain = FALSE;
> 6 memory.c 7763 cache_chain = 0;
> 7 memory.c 7802 if (has_cache_chain && (cache == cache_chain))
> 8 memory.c 7804 "cache_chain", FAULT_ON_ERROR);
> 9 memory.c 7973 * slab chain list. 2.6 uses the "cache_chain" list_head.
> a memory.c 7976 get_symbol_data("cache_chain", sizeof(ulong), &cache);
> b memory.c 7978 cache_end = symbol_value("cache_chain");
> c memory.c 8908 get_symbol_data("cache_chain", sizeof(ulong), &si->cache);
> d memory.c 8910 cache_end = symbol_value("cache_chain");
>
> So it would be a matter of recognizing tha "cache_chain" does not exist
> as a symbol and "slab_caches" does, and baiting-and-switching as appropriate.
>
> I'm not sure when I'm going to get the chance to look at this.
> If anybody else on the list has some cycles to spare, fixing this
> probably won't be a major undertaking.
>
> Dave

I've got a fix queued for this for crash-6.0.9. It was a little
bit more involved that the bait-and-switch above, and there was
also a related "kmem -s" bug in 3.6 kernels that are configured
with CONFIG_SLUB (Fedora 18).

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 06:44 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org