Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Crash Utility (http://www.linux-archive.org/crash-utility/)
-   -   Crash fails with the error: "crash: read error: kernel virtual address:..." (http://www.linux-archive.org/crash-utility/252058-crash-fails-error-crash-read-error-kernel-virtual-address.html)

Adhiraj Joshi 02-25-2009 06:28 AM

Crash fails with the error: "crash: read error: kernel virtual address:..."
 
Hi All,

On my x86 machine with fedora 10, I get an error which says: "crash: read error: kernel virtual address:...". The running kernel is not the Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12) on my machine.


On googling, I found that someone has reported a similar problem but which was on Fedora kernel and x86_64 arch. Here is the link to that bug:
https://bugzilla.redhat.com/show_bug.cgi?id=237383


In my case, this is the output from the crash command:

[root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
crash 4.0-7
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008* Red Hat, Inc.
Copyright (C) 2004, 2005, 2006* IBM Corporation

Copyright (C) 1999-2006* Hewlett-Packard Co
Copyright (C) 2005, 2006* Fujitsu Limited
Copyright (C) 2006, 2007* VA Linux Systems Japan K.K.
Copyright (C) 2005* NEC Corporation
Copyright (C) 1999, 2002, 2007* Silicon Graphics, Inc.

Copyright (C) 1999, 2000, 2001, 2002* Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.* Enter "help copying" to see the conditions.

This program has absolutely no warranty.* Enter "help warranty" for details.
*
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.* Type "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu"...

crash: read error: kernel virtual address: c08b0ae8* type: "cpu_possible_map"
WARNING: cannot read cpu_possible_map
crash: read error: kernel virtual address: c0801a94* type: "cpu_present_map"

WARNING: cannot read cpu_present_map
crash: read error: kernel virtual address: c080155c* type: "cpu_online_map"
WARNING: cannot read cpu_online_map
crash: read error: kernel virtual address: c08dd700* type: "xtime"

[root@maveric ~]#

This is the "uname -a" output:
"Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686 i686 i386 GNU/Linux"

Does anyone has any idea on this?

Thanks and regards,

Adhiraj Joshi.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-25-2009 01:15 PM

Crash fails with the error: "crash: read error: kernel virtual address:..."
 
----- "Adhiraj Joshi" <adhiraj@linsyssoft.com> wrote:

> Hi All,
>
> On my x86 machine with fedora 10, I get an error which says: "crash:
> read error: kernel virtual address:...". The running kernel is not the
> Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12)
> on my machine.
>
> On googling, I found that someone has reported a similar problem but
> which was on Fedora kernel and x86_64 arch. Here is the link to that bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=237383
>
> In my case, this is the output from the crash command:
>
> [root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
> crash 4.0-7
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
>
> crash: read error: kernel virtual address: c08b0ae8 type: "cpu_possible_map"
> WARNING: cannot read cpu_possible_map
> crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map"
> WARNING: cannot read cpu_present_map
> crash: read error: kernel virtual address: c080155c type: "cpu_online_map"
> WARNING: cannot read cpu_online_map
> crash: read error: kernel virtual address: c08dd700 type: "xtime"
> [root@maveric ~]#
>
> This is the "uname -a" output:
> "Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686
> i686 i386 GNU/Linux"
>
> Does anyone has any idea on this?

Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows
the reading of /dev/mem above 1MB physical. That makes it useless for
the crash utility:

https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html

Fedora and RHEL kernels contain the /dev/crash driver, which gets automatically
installed upon crash invocation on the live system. That driver, found in
the kernel file "drivers/char/crash.c", is a replacement for the (restricted)
/dev/mem driver.

For your kernel, you've got 3 options:

(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
(3) Write a kretprobe module that tinkers with the return value of the
kernel's devmem_is_allowed() function such that it always returns 1.

And w/respect to each option:

(1) Obviously this is the course of least resistance.

(2) Porting the RHEL/Fedora /dev/crash driver is possible, but since page_is_ram()
is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the driver
would have to be removed. And if you were to make it EXPORT_GPL(), then it
makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead.

(3) I'll append some information re: writing a kretprobe module, but like
porting the /dev/crash driver, it may require rebuilding your kernel anyway.
And if that's the case, just remove the CONFIG_STRICT_DEVMEM restriction.

Also, you should upgrade from 4.0-7. There have been several fixes
for 2.6.27 kernels since that version.

Dave

----------------------------------------------------------------------------

Writing a kretprobe for devmem_is_allowed():

There are are currently three types of probes: kprobes, jprobes, and
kretprobes (also called return probes), but for this purpose, a
kretprobe is required because it allows the return value of the
devmem_is_allowed() function to be modified such that it will
always return 1.

The kernel documentation contains both directions for building
a kretprobe module and example module files for each kprobe type.
Pre-2.6.25 kernel trees put sample cut-and-pastable module files
contained within the "./Documentation/kprobes.txt" file itself.
2.6.25 and later kernel trees locate them in the "./samples/kprobes"
directory.

Note that pre-2.6.25 kernels must have been configured with both
CONFIG_KPROBES, CONFIG_KALLSYMS and CONFIG_MODULES turned on.
Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES,
In any case, if the target kernel configs preclude the installation
of the module, then it probably makes more sense to rebuild the
kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe
approach entirely.

Take the "kretprobe-example.c" cut-and-pastable file from the
the "./Documentation/kprobes.txt" file, or the standalone
"./samples/kprobes/kretprobe_example.c" file, whichever is
appropriate, and modify the kretprobe_init() and ret_handler()
functions like so:

static int __init kretprobe_init(void)
{
int ret;

- my_kretprobe.kp.symbol_name = func_name;
+ my_kretprobe.kp.symbol_name = "devmem_is_allowed";
ret = register_kretprobe(&my_kretprobe);
if (ret < 0) {
printk(KERN_INFO "register_kretprobe failed, returned %d
",
ret);
return -1;
}
printk(KERN_INFO "Planted return probe at %s: %p
",
my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);
return 0;
}

and make the ret_handler() function to simply do this:

static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
regs->ax = 1;
return 0;
}

Earlier kernel versions may have different pt_regs structure member
names for the return register, i.e., "regs->eax" or "regs->rax" for
the x86 and x86_64 architectures respectively.

Build the kretprobe module as directed in ./Documentation/kprobes.txt"
and insmod it, after which crash will run just fine with /dev/mem.






--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-27-2009 01:33 PM

Crash fails with the error: "crash: read error: kernel virtual address:..."
 
----- "Adhiraj Joshi" <adhiraj@linsyssoft.com> wrote:

> Hi Dave,
>
> Thanks for the quick reply! I followed approach 1 of disabling
> CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem
> :-) On invoking crash, I get the following warning:
> WARNING: cannot access vmalloc'd module memory
>
> Because of this I am not able to run commands like "mod":
> crash> mod
> mod: cannot access vmalloc'd module memory
> crash>

What you're running into now is yet another problem with the /dev/mem
driver on the 32-bit x86 architecture when used on a live system with
more than 896MB of physical memory. Here's the beginning of the
/dev/mem driver's read routine in ./drivers/char/mem.c:

static ssize_t read_mem(struct file * file, char __user * buf,
size_t count, loff_t *ppos)
{
unsigned long p = *ppos;
ssize_t read, sz;
char *ptr;

if (!valid_phys_addr_range(p, count))
return -EFAULT;

where on x86, valid_phys_addr_range() looks like this:

static inline int valid_phys_addr_range(unsigned long addr, size_t count)
{
if (addr + count > __pa(high_memory))
return 0;

return 1;
}

And "high_memory" is the dividing line between "lowmem" and "highmem",
which is 896MB physical on your machine. So any reference to a physical
address above 896MB is rejected.

Kernel module memory is vmalloc'd with the __GFP_HIGHMEM modifier set,
so the allocation is biased to use physical memory from the highmem zone.
That being the case, and since your machine apparently has physical memory
above 896MB, any crash utility request to read module memory that
exists up there fails. Hence the "cannot access vmalloc'd module
memory" warning. You'll also see failures to access user memory,
and the PTEs to translate them as well. But given that the crucial
kernel memory is always located in lowmem, at least you can run a
crash session.

> Should the approach 2 or 3 that you mentioned below solve this
> problem? I have upgraded crash to the latest version (4.0-7.7).

... [ snip ] ...

> > For your kernel, you've got 3 options:
> >
> > (1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
> > (2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
> > (3) Write a kretprobe module that tinkers with the return value of the
> > kernel's devmem_is_allowed() function such that it always returns 1.

(3) won't help since you can't get that far -- and since you've turned
off CONFIG_STRICT_DEVMEM, it's unnecessary anyway.

You also can't kludge/kretprobe valid_phys_addr_range() to return 1, because
later on in read_mem() the physical address is translated into a unity-mapped
kernel virtual address here:

ptr = xlate_dev_mem_ptr(p);
if (!ptr)
return -EFAULT;

if (copy_to_user(buf, ptr, sz)) {
unxlate_dev_mem_ptr(p, ptr);
return -EFAULT;
}

where for x86, xlate_dev_mem_ptr() simply does a __va() on the
physical address:

void *xlate_dev_mem_ptr(unsigned long phys)
{
void *addr;
unsigned long start = phys & PAGE_MASK;

/* If page is RAM, we can use __va. Otherwise ioremap and unmap. */
if (page_is_ram(start >> PAGE_SHIFT))
return __va(phys);

and __va() simply adds PAGE_OFFSET to the physical address to make a
unity-mapped kernel virtual address -- which does not apply to "highmem"
physical addresses.

So that basically leaves (2) as the best option. That driver uses
kmap() on the physical address, and so it can handle any physical
memory request.

The driver should be portable, one caveat being is that it
uses page_is_ram() as a physical memory qualifier, and therefore
requires that the function be EXPORT_SYMBOL()'d to build. And in
upstream kernels, it currently is not.

It looks like you may be able to replace page_is_ram() with the
e820_any_mapped() function -- which is exported -- and look for
the E820_RAM "type":

int
e820_any_mapped(u64 start, u64 end, unsigned type)
{
int i;

for (i = 0; i < e820.nr_map; i++) {
struct e820entry *ei = &e820.map[i];

if (type && ei->type != type)
continue;
if (ei->addr >= end || ei->addr + ei->size <= start)
continue;
return 1;
}
return 0;
}
EXPORT_SYMBOL_GPL(e820_any_mapped);

Or you can make page_is_ram() EXPORT_SYMBOL_GPL().

Or you can just comment out the page_is_ram() call in the crash
driver's crash.h file.

Another issue with the port is that the driver was written for
kernels prior to the x86/x86_64 merge. So the driver's
"./include/asm-i386/crash.h" and "./include/asm-x86_64/crash.h"
header files will need to be merged.

Or you can always boot your kernel with "mem=896M" on the
kernel command line, and the problem goes away... ;-)

Dave





--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Dave Anderson 02-27-2009 02:17 PM

Crash fails with the error: "crash: read error: kernel virtual address:..."
 
----- "Dave Anderson" <anderson@redhat.com> wrote:
>
> Another issue with the port is that the driver was written for
> kernels prior to the x86/x86_64 merge. So the driver's
> "./include/asm-i386/crash.h" and "./include/asm-x86_64/crash.h"
> header files will need to be merged.

Actually that's already been done in the Fedora driver, so that's
not an issue -- I was thinking about the RHEL version, which is
based upon 2.6.18...

Also, the Fedora driver does EXPORT_SYMBOL_GPL(page_is_ram).
So if you can rebuild your kernel, you might just add that
instead of substituting e820_any_mapped().

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility


All times are GMT. The time now is 09:43 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.