FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.

» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

LinkBack Thread Tools
Old 02-27-2009, 08:56 AM
Adhiraj Joshi
Default Crash fails with the error: "crash: read error: kernel virtual address:..."

Hi Dave,

Thanks for the quick reply! I followed approach 1 of disabling CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem :-) On invoking crash, I get the following warning:
WARNING: cannot access vmalloc'd module memory

Because of this I am not able to run commands like "mod":
crash> mod
mod: cannot access vmalloc'd module memory

Should the approach 2 or 3 that you mentioned below solve this problem? I have upgraded crash to the latest version (4.0-7.7).

Thanks and regards,

On Wed, Feb 25, 2009 at 7:45 PM, Dave Anderson <anderson@redhat.com> wrote:

----- "Adhiraj Joshi" <adhiraj@linsyssoft.com> wrote:

> Hi All,


> On my x86 machine with fedora 10, I get an error which says: "crash:

> read error: kernel virtual address:...". The running kernel is not the

> Fedora 10 kernel, I have installed a latest vanilla kernel (

> on my machine.


> On googling, I found that someone has reported a similar problem but

> which was on Fedora kernel and x86_64 arch. Here is the link to that bug:

> https://bugzilla.redhat.com/show_bug.cgi?id=237383


> In my case, this is the output from the crash command:


> [root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux

> crash 4.0-7

> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.

> Copyright (C) 2004, 2005, 2006 IBM Corporation

> Copyright (C) 1999-2006 Hewlett-Packard Co

> Copyright (C) 2005, 2006 Fujitsu Limited

> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.

> Copyright (C) 2005 NEC Corporation

> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.

> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.

> This program is free software, covered by the GNU General Public License,

> and you are welcome to change it and/or distribute copies of it under

> certain conditions. Enter "help copying" to see the conditions.

> This program has absolutely no warranty. Enter "help warranty" for

> details.


> GNU gdb 6.1

> Copyright 2004 Free Software Foundation, Inc.

> GDB is free software, covered by the GNU General Public License, and you are

> welcome to change it and/or distribute copies of it under certain conditions.

> Type "show copying" to see the conditions.

> There is absolutely no warranty for GDB. Type "show warranty" for details.

> This GDB was configured as "i686-pc-linux-gnu"...


> crash: read error: kernel virtual address: c08b0ae8 type: "cpu_possible_map"

> WARNING: cannot read cpu_possible_map

> crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map"

> WARNING: cannot read cpu_present_map

> crash: read error: kernel virtual address: c080155c type: "cpu_online_map"

> WARNING: cannot read cpu_online_map

> crash: read error: kernel virtual address: c08dd700 type: "xtime"

> [root@maveric ~]#


> This is the "uname -a" output:

> "Linux maveric #1 SMP Fri Feb 20 22:13:54 IST 2009 i686

> i686 i386 GNU/Linux"


> Does anyone has any idea on this?

Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows

the reading of /dev/mem above 1MB physical. *That makes it useless for

the crash utility:


Fedora and RHEL kernels contain the /dev/crash driver, which gets automatically

installed upon crash invocation on the live system. *That driver, found in

the kernel file "drivers/char/crash.c", is a replacement for the (restricted)

/dev/mem driver.

For your kernel, you've got 3 options:

(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.

(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.

(3) Write a kretprobe module that tinkers with the return value of the

* *kernel's devmem_is_allowed() function such that it always returns 1.

And w/respect to each option:

(1) Obviously this is the course of least resistance.

(2) Porting the RHEL/Fedora /dev/crash driver is possible, but since page_is_ram()

* *is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the driver

* *would have to be removed. *And if you were to make it EXPORT_GPL(), then it

* *makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead.

(3) I'll append some information re: writing a kretprobe module, but like

* *porting the /dev/crash driver, it may require rebuilding your kernel anyway.

* *And if that's the case, just remove the CONFIG_STRICT_DEVMEM restriction.

Also, you should upgrade from 4.0-7. *There have been several fixes

for 2.6.27 kernels since that version.



Writing a kretprobe for devmem_is_allowed():

There are are currently three types of probes: kprobes, jprobes, and

kretprobes (also called return probes), but for this purpose, a

kretprobe is required because it allows the return value of the

devmem_is_allowed() function to be modified such that it will

always return 1.

The kernel documentation contains both directions for building

a kretprobe module and example module files for each kprobe type.

Pre-2.6.25 kernel trees put sample cut-and-pastable module files

contained within the "./Documentation/kprobes.txt" file itself.

2.6.25 and later kernel trees locate them in the "./samples/kprobes"


Note that pre-2.6.25 kernels must have been configured with both


Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES,

In any case, if the target kernel configs preclude the installation

of the module, then it probably makes more sense to rebuild the

kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe

approach entirely.

Take the "kretprobe-example.c" cut-and-pastable file from the

the "./Documentation/kprobes.txt" file, or the standalone

"./samples/kprobes/kretprobe_example.c" file, whichever is

appropriate, and modify the kretprobe_init() and ret_handler()

functions like so:

*static int __init kretprobe_init(void)


* * * * int ret;

- * * * *my_kretprobe.kp.symbol_name = func_name;

+ * * * *my_kretprobe.kp.symbol_name = "devmem_is_allowed";

* * * * ret = register_kretprobe(&my_kretprobe);

* * * * if (ret < 0) {

* * * * * * * * printk(KERN_INFO "register_kretprobe failed, returned %d

* * * * * * * * * * * * * * * * ret);

* * * * * * * * return -1;

* * * * }

* * * * printk(KERN_INFO "Planted return probe at %s: %p

* * * * * * * * * * * * my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);

* * * * return 0;


and make the ret_handler() function to simply do this:

*static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)


* * * * regs->ax = 1;

* * * * return 0;


Earlier kernel versions may have different pt_regs structure member

names for the return register, i.e., "regs->eax" or "regs->rax" for

the x86 and x86_64 architectures respectively.

Build the kretprobe module as directed in ./Documentation/kprobes.txt"

and insmod it, after which crash will run just fine with /dev/mem.


Crash-utility mailing list



Crash-utility mailing list

Thread Tools

All times are GMT. The time now is 09:41 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org