FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 02-08-2012, 10:49 PM
Bob Montgomery
 
Default More fixes for kmem on slabs

More testing revealed a machine in our stable that either failed to
initialize kmem:

please wait... (gathering kmem slab cache data)
crash-6.0.3: page excluded: kernel virtual address: ffff8801263d6000 type: "kmem_cache buffer"

crash-6.0.3: unable to initialize kmem slab cache subsystem

Or succeeded on initialize and then failed on a kmem -s command:

crash-6.0.3> kmem -s
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
Segmentation fault


The problem is that the array struct at the end of kmem_cache remains declared as
32 elements, but for all dynamically allocated copies, is actually trimmed down
to nr_cpu_ids in length.

crash-6.0.3.best> struct kmem_cache
struct kmem_cache {
unsigned int batchcount;
...

struct list_head next;
struct kmem_list3 **nodelists;
struct array_cache *array[32];
}
SIZE: 368


On my normal play machine, nr_cpu_ids = 32 and actual cpus = 16.

On the failing machine, nr_cpus_ids and actual cpus are both 2.

Two problems occur:

1) max_cpudata_limit traverses the array until it finds a 0x0 or
reaches the real size. On the 2-cpu system, the "third" element in the
array belonged elsewhere, was non-zero, and pointed to data that caused
the apparent limit to be 0xffffffffffff8801, which didn't work well as
a length in a memcopy.

2) kmem_cache structs can be allocated near enough to the edge of a page
that the old incorrect length crosses the page boundary, even though the
real smaller structure fits in the page. That caused a readmem of the
structure to cross into a coincidentally missing page in the dump.

This patch fixes both of those (after wrestling ARRAY_LENGTH to the
ground), but *does not* fix the similar page crossing problem when I try
to use a "struct kmem_cache" command on the particular structure at the
end of the page.

Reference this unfortunate comment in include/linux/slab_def.h:

/* 6) per-cpu/per-node data, touched during every alloc/free */
/*
* We put array[] at the end of kmem_cache, because we want to size
* this array to nr_cpu_ids slots instead of NR_CPUS
* (see kmem_cache_init())
* We still use [NR_CPUS] and not [1] or [0] because cache_cache
* is statically defined, so we reserve the max number of cpus.
*/
struct kmem_list3 **nodelists;
struct array_cache *array[NR_CPUS];
/*
* Do not add fields after array[]
*/
};

Bob Montgomery

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 02-09-2012, 03:06 PM
Dave Anderson
 
Default More fixes for kmem on slabs

----- Original Message -----
> More testing revealed a machine in our stable that either failed to
> initialize kmem:
>
> please wait... (gathering kmem slab cache data)
> crash-6.0.3: page excluded: kernel virtual address: ffff8801263d6000
> type: "kmem_cache buffer"
>
> crash-6.0.3: unable to initialize kmem slab cache subsystem
>
> Or succeeded on initialize and then failed on a kmem -s command:
>
> crash-6.0.3> kmem -s
> CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
> Segmentation fault
>
>
> The problem is that the array struct at the end of kmem_cache remains declared as
> 32 elements, but for all dynamically allocated copies, is actually trimmed down
> to nr_cpu_ids in length.
>
> crash-6.0.3.best> struct kmem_cache
> struct kmem_cache {
> unsigned int batchcount;
> ...
>
> struct list_head next;
> struct kmem_list3 **nodelists;
> struct array_cache *array[32];
> }
> SIZE: 368
>
>
> On my normal play machine, nr_cpu_ids = 32 and actual cpus = 16.
>
> On the failing machine, nr_cpus_ids and actual cpus are both 2.
>
> Two problems occur:
>
> 1) max_cpudata_limit traverses the array until it finds a 0x0 or
> reaches the real size. On the 2-cpu system, the "third" element in the
> array belonged elsewhere, was non-zero, and pointed to data that caused
> the apparent limit to be 0xffffffffffff8801, which didn't work well as
> a length in a memcopy.

But your patch does this:

@@ -8117,8 +8135,9 @@ kmem_cache_s_array_nodes:
"array cache array", RETURN_ON_ERROR))
goto bail_out;

- for (i = max_limit = 0; (i < ARRAY_LENGTH(kmem_cache_s_array)) &&
- cpudata[i]; i++) {
+ for (i = max_limit = 0; (i < kmem_cache_nr_cpu)
+ && (i < ARRAY_LENGTH(kmem_cache_s_array))
+ && cpudata[i]; i++) {
if (!readmem(cpudata[i]+OFFSET(array_cache_limit),
KVADDR, &limit, sizeof(int),
"array cache limit", RETURN_ON_ERROR))

On "old" slab systems, your new "kmem_cache_nr_cpu" variable remains at
its initialized value of zero, and the loop never gets entered. So I don't
think you wanted to keep the (i < kmem_cache_nr_cpu) there, right?

> 2) kmem_cache structs can be allocated near enough to the edge of a page
> that the old incorrect length crosses the page boundary, even though the
> real smaller structure fits in the page. That caused a readmem of the
> structure to cross into a coincidentally missing page in the dump.

Right -- that was the genesis of the kmem_cache_downsize() function.

> This patch fixes both of those (after wrestling ARRAY_LENGTH to the
> ground), but *does not* fix the similar page crossing problem when I try
> to use a "struct kmem_cache" command on the particular structure at the
> end of the page.

Yeah, damn, I don't know what can be done for that, aside from some
horrific kludge to gdb_readmem_callback() to return successfully even
if the readmem() failed.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 02-09-2012, 06:08 PM
Dave Anderson
 
Default More fixes for kmem on slabs

----- Original Message -----

> But your patch does this:
>
> @@ -8117,8 +8135,9 @@ kmem_cache_s_array_nodes:
> "array cache array", RETURN_ON_ERROR))
> goto bail_out;
>
> - for (i = max_limit = 0; (i < ARRAY_LENGTH(kmem_cache_s_array)) &&
> - cpudata[i]; i++) {
> + for (i = max_limit = 0; (i < kmem_cache_nr_cpu)
> + && (i < ARRAY_LENGTH(kmem_cache_s_array))
> + && cpudata[i]; i++) {
> if (!readmem(cpudata[i]+OFFSET(array_cache_limit),
> KVADDR, &limit, sizeof(int),
> "array cache limit", RETURN_ON_ERROR))
>
> On "old" slab systems, your new "kmem_cache_nr_cpu" variable remains at
> its initialized value of zero, and the loop never gets entered. So I don't
> think you wanted to keep the (i < kmem_cache_nr_cpu) there, right?

How's this work for you? (patch also attached)

--- crash-6.0.3/memory.c.orig
+++ crash-6.0.3/memory.c
@@ -7977,14 +7977,32 @@ kmem_cache_downsize(void)
char *cache_buf;
uint buffer_size;
int nr_node_ids;
+ int nr_cpu_ids;

if ((THIS_KERNEL_VERSION < LINUX(2,6,22)) ||
- (vt->flags & NODELISTS_IS_PTR) ||
!(vt->flags & PERCPU_KMALLOC_V2_NODES) ||
!kernel_symbol_exists("cache_cache") ||
!MEMBER_EXISTS("kmem_cache", "buffer_size"))
return;

+ if (vt->flags & NODELISTS_IS_PTR) {
+ /*
+ * kmem_cache.array[] is actually sized by
+ * the number of cpus; real value is nr_cpu_ids,
+ * but fallback is kt->cpus.
+ */
+ if (kernel_symbol_exists("nr_cpu_ids"))
+ get_symbol_data("nr_cpu_ids", sizeof(int),
+ &nr_cpu_ids);
+ else
+ nr_cpu_ids = kt->cpus;
+
+ ARRAY_LENGTH(kmem_cache_s_array) = nr_cpu_ids;
+ ASSIGN_SIZE(kmem_cache_s) = OFFSET(kmem_cache_s_array) +
+ sizeof(ulong) * nr_cpu_ids;
+ return;
+ }
+
cache_buf = GETBUF(SIZE(kmem_cache_s));

if (!readmem(symbol_value("cache_cache"), KVADDR, cache_buf,

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 02-13-2012, 05:08 PM
Bob Montgomery
 
Default More fixes for kmem on slabs

On Thu, 2012-02-09 at 11:06 -0500, Dave Anderson wrote:

> But your patch does this:
>
> @@ -8117,8 +8135,9 @@ kmem_cache_s_array_nodes:
> "array cache array", RETURN_ON_ERROR))
> goto bail_out;
>
> - for (i = max_limit = 0; (i < ARRAY_LENGTH(kmem_cache_s_array)) &&
> - cpudata[i]; i++) {
> + for (i = max_limit = 0; (i < kmem_cache_nr_cpu)
> + && (i < ARRAY_LENGTH(kmem_cache_s_array))
> + && cpudata[i]; i++) {
> if (!readmem(cpudata[i]+OFFSET(array_cache_limit),
> KVADDR, &limit, sizeof(int),
> "array cache limit", RETURN_ON_ERROR))
>
> On "old" slab systems, your new "kmem_cache_nr_cpu" variable remains at
> its initialized value of zero, and the loop never gets entered. So I don't
> think you wanted to keep the (i < kmem_cache_nr_cpu) there, right?

Of course you're right, sorry about that. I originally tried to fix the
problem by putting (i < kt->cpus) there before I fixed the array length,
then substituted instead of removing that clause.

>
> > 2) kmem_cache structs can be allocated near enough to the edge of a page
> > that the old incorrect length crosses the page boundary, even though the
> > real smaller structure fits in the page. That caused a readmem of the
> > structure to cross into a coincidentally missing page in the dump.
>
> Right -- that was the genesis of the kmem_cache_downsize() function.
>
> > This patch fixes both of those (after wrestling ARRAY_LENGTH to the
> > ground), but *does not* fix the similar page crossing problem when I try
> > to use a "struct kmem_cache" command on the particular structure at the
> > end of the page.
>
> Yeah, damn, I don't know what can be done for that, aside from some
> horrific kludge to gdb_readmem_callback() to return successfully even
> if the readmem() failed.

At least you won't see the problem on startup, only if you accidentally
ask for that particular struct.

Bob M.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 02-13-2012, 05:35 PM
Bob Montgomery
 
Default More fixes for kmem on slabs

On Thu, 2012-02-09 at 14:08 -0500, Dave Anderson wrote:

> How's this work for you? (patch also attached)
>
> --- crash-6.0.3/memory.c.orig
> +++ crash-6.0.3/memory.c
> @@ -7977,14 +7977,32 @@ kmem_cache_downsize(void)
> char *cache_buf;
> uint buffer_size;
> int nr_node_ids;
> + int nr_cpu_ids;
>
> if ((THIS_KERNEL_VERSION < LINUX(2,6,22)) ||
> - (vt->flags & NODELISTS_IS_PTR) ||
> !(vt->flags & PERCPU_KMALLOC_V2_NODES) ||
> !kernel_symbol_exists("cache_cache") ||
> !MEMBER_EXISTS("kmem_cache", "buffer_size"))
> return;
>
> + if (vt->flags & NODELISTS_IS_PTR) {
> + /*
> + * kmem_cache.array[] is actually sized by
> + * the number of cpus; real value is nr_cpu_ids,
> + * but fallback is kt->cpus.
> + */
> + if (kernel_symbol_exists("nr_cpu_ids"))
> + get_symbol_data("nr_cpu_ids", sizeof(int),
> + &nr_cpu_ids);
> + else
> + nr_cpu_ids = kt->cpus;
> +
> + ARRAY_LENGTH(kmem_cache_s_array) = nr_cpu_ids;
> + ASSIGN_SIZE(kmem_cache_s) = OFFSET(kmem_cache_s_array) +
> + sizeof(ulong) * nr_cpu_ids;
> + return;
> + }
> +
> cache_buf = GETBUF(SIZE(kmem_cache_s));
>
> if (!readmem(symbol_value("cache_cache"), KVADDR, cache_buf,
>
> Dave
>

Looks good. Identical answers to my original patch on my two 3.1.4 test
cases, and a cleaner solution.

Thanks,
Bob Montgomery

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 06:10 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org