FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 08-23-2010, 06:34 PM
Paul Cahier
 
Default Crash issue when loading vmcore

Hello,

I have finished setting up kdump and kexec today, recompiling my kernel
to add everything needed in there.
I have triggered a kernel panic by echo c>/proc/sysrq-trigger, and found
that the vmcore dump was indeed there after all was done.


However I can not get any traces out of that crash dump(short version,
long version at the end of the email):


crash /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930
[...]
crash: read error: kernel virtual address: c148a9a0 type:
"kernel_config_data"

WARNING: cannot read kernel_config_data
crash: read error: kernel virtual address: c1487e28 type:
"cpu_possible_mask"



If I try crash --minimal things do load but I'm stuck with the minimal
error set that's not very helpful.

All I'm looking at is getting a full trace of the kernel panic.


- Paul-Kenji Cahier


PS, the full version:
crash -d7 /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930

crash 5.0.6
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: b77344c0
header_size: 1860
num_pt_load_segments: 9
pt_load_segment[0]:
file_offset: 744
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a0744
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[2]:
file_offset: fa0744
phys_start: 5000000
phys_end: 38000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 33fa0744
phys_start: 38000000
phys_end: 3e5ff000
zero_fill: 0
pt_load_segment[4]:
file_offset: 3a59f744
phys_start: 3e6c6000
phys_end: 3f594000
zero_fill: 0
pt_load_segment[5]:
file_offset: 3b46d744
phys_start: 3f59c000
phys_end: 3f62a000
zero_fill: 0
pt_load_segment[6]:
file_offset: 3b4fb744
phys_start: 3f62e000
phys_end: 3f6a9000
zero_fill: 0
pt_load_segment[7]:
file_offset: 3b576744
phys_start: 3f6e9000
phys_end: 3f6ed000
zero_fill: 0
pt_load_segment[8]:
file_offset: 3b57a744
phys_start: 3f6ff000
phys_end: 3f700000
zero_fill: 0
elf_header: 85368c0
elf32: 85368c0
notes32: 85368f4
load32: 8536914
elf64: 0
notes64: 0
load64: 0
nt_prstatus: 8536a34
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 2
vmcoreinfo: 0
size_vmcoreinfo: 0
nt_prstatus_percpu:
08536a34 08536ad8

Elf32_Ehdr:
e_ident: 177ELF
e_ident[EI_CLASS]: 1 (ELFCLASS32)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 3 (EM_386)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 34
e_shoff: 0
e_flags: 0
e_ehsize: 34
e_phentsize: 20
e_phnum: a
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf32_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 372 (174)
p_vaddr: 0
p_paddr: 0
p_filesz: 1488 (5d0)
p_memsz: 1488 (5d0)
p_flags: 0 ()
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 1860 (744)
p_vaddr: c0000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 657220 (a0744)
p_vaddr: c0100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 16385860 (fa0744)
p_vaddr: c5000000
p_paddr: 5000000
p_filesz: 855638016 (33000000)
p_memsz: 855638016 (33000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 872023876 (33fa0744)
p_vaddr: ffffffff
p_paddr: 38000000
p_filesz: 106950656 (65ff000)
p_memsz: 106950656 (65ff000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 978974532 (3a59f744)
p_vaddr: ffffffff
p_paddr: 3e6c6000
p_filesz: 15523840 (ece000)
p_memsz: 15523840 (ece000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 994498372 (3b46d744)
p_vaddr: ffffffff
p_paddr: 3f59c000
p_filesz: 581632 (8e000)
p_memsz: 581632 (8e000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995080004 (3b4fb744)
p_vaddr: ffffffff
p_paddr: 3f62e000
p_filesz: 503808 (7b000)
p_memsz: 503808 (7b000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995583812 (3b576744)
p_vaddr: ffffffff
p_paddr: 3f6e9000
p_filesz: 16384 (4000)
p_memsz: 16384 (4000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995600196 (3b57a744)
p_vaddr: ffffffff
p_paddr: 3f6ff000
p_filesz: 4096 (1000)
p_memsz: 4096 (1000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000401
c06fef00 00000001 0000f004 0000f008
00000000 c06e3ecc 00000282 00000282
00000024 c06e3fa4 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000dbd 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 c0712420 00007e7e
00000000 00000063 00000000 f0639f0c
00000063 0000007b 0000007b 000000d8
00000033 ffffffff c03145b2 00000060
00010086 f0639f0c 00000068 00000000
Elf32_Nhdr:
n_namesz: 11 ("VMCOREINFO")
n_descsz: 1134
n_type: 0 (unused)
OSRELEASE=2.6.35.3-saber
PAGESIZE=4096
SYMBOL(init_uts_ns)=c06f9120
SYMBOL(node_online_map)=c0730644
SYMBOL(swapper_pg_dir)=c06e4000
SYMBOL(_stext)=c0101000
SYMBOL(vmlist)=c07d3540
SYMBOL(mem_map)=c07d3500
SYMBOL(contig_page_data)=c072ce80
SIZE(page)=32
SIZE(pglist_data)=4224
SIZE(zone)=1024
SIZE(free_area)=44
SIZE(list_head)=8
SIZE(nodemask_t)=4
OFFSET(page.flags)=0
OFFSET(page._count)=4
OFFSET(page.mapping)=16
OFFSET(page.lru)=24
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=4140
OFFSET(pglist_data.node_mem_map)=4144
OFFSET(pglist_data.node_start_pfn)=4148
OFFSET(pglist_data.node_spanned_pages)=4156
OFFSET(pglist_data.node_id)=4160
OFFSET(zone.free_area)=40
OFFSET(zone.vm_stat)=728
OFFSET(zone.spanned_pages)=916
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=4
OFFSET(vm_struct.addr)=4
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=c06fc83c
SYMBOL(log_end)=c07bb7ec
SYMBOL(log_buf_len)=c06fc838
SYMBOL(logged_chars)=c07c38a0
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
CONFIG_X86_PAE=y
CRASHTIME=1282584565
cannot determine relocation value: not a live system
gdb /usr/src/linux-2.6.35.3/vmlinux
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

<readmem: c148a9a0, KVADDR, "kernel_config_data", 32768, (ROE), 8bad3d8>
crash: read error: kernel virtual address: c148a9a0 type:
"kernel_config_data"

WARNING: cannot read kernel_config_data
<readmem: c1487e28, KVADDR, "cpu_possible_mask", 4, (FOE), bfed4bbc>
crash: read error: kernel virtual address: c1487e28 type:
"cpu_possible_mask"


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 07:20 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul Cahier" <pkc@f1-photo.com> wrote:

> Hello,
>
> I have finished setting up kdump and kexec today, recompiling my kernel
> to add everything needed in there.
> I have triggered a kernel panic by echo c>/proc/sysrq-trigger, and found
> that the vmcore dump was indeed there after all was done.
>
> However I can not get any traces out of that crash dump(short version,
> long version at the end of the email):
>
> crash /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930
> [...]
> crash: read error: kernel virtual address: c148a9a0 type: "kernel_config_data"
> WARNING: cannot read kernel_config_data
> crash: read error: kernel virtual address: c1487e28 type: "cpu_possible_mask"

The virtual addresses for "kernel_config_data" and "cpu_possible_mask" are
strange (too high?) -- I'll continue the analysis at the end of your "d7"
output below...

> If I try crash --minimal things do load but I'm stuck with the minimal
> error set that's not very helpful.
> All I'm looking at is getting a full trace of the kernel panic.
>
>
> - Paul-Kenji Cahier
>
>
> PS, the full version:
> crash -d7 /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930
>
> crash 5.0.6
> Copyright (C) 2002-2010 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> vmcore_data:
> flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
> ndfd: 3
> ofp: b77344c0
> header_size: 1860
> num_pt_load_segments: 9
> pt_load_segment[0]:
> file_offset: 744
> phys_start: 0
> phys_end: a0000
> zero_fill: 0
> pt_load_segment[1]:
> file_offset: a0744
> phys_start: 100000
> phys_end: 1000000
> zero_fill: 0
> pt_load_segment[2]:
> file_offset: fa0744
> phys_start: 5000000
> phys_end: 38000000
> zero_fill: 0
> pt_load_segment[3]:
> file_offset: 33fa0744
> phys_start: 38000000
> phys_end: 3e5ff000
> zero_fill: 0
> pt_load_segment[4]:
> file_offset: 3a59f744
> phys_start: 3e6c6000
> phys_end: 3f594000
> zero_fill: 0
> pt_load_segment[5]:
> file_offset: 3b46d744
> phys_start: 3f59c000
> phys_end: 3f62a000
> zero_fill: 0
> pt_load_segment[6]:
> file_offset: 3b4fb744
> phys_start: 3f62e000
> phys_end: 3f6a9000
> zero_fill: 0
> pt_load_segment[7]:
> file_offset: 3b576744
> phys_start: 3f6e9000
> phys_end: 3f6ed000
> zero_fill: 0
> pt_load_segment[8]:
> file_offset: 3b57a744
> phys_start: 3f6ff000
> phys_end: 3f700000
> zero_fill: 0
> elf_header: 85368c0
> elf32: 85368c0
> notes32: 85368f4
> load32: 8536914
> elf64: 0
> notes64: 0
> load64: 0
> nt_prstatus: 8536a34
> nt_prpsinfo: 0
> nt_taskstruct: 0
> task_struct: 0
> page_size: 0
> switch_stack: 0
> xen_kdump_data: (unused)
> num_prstatus_notes: 2
> vmcoreinfo: 0
> size_vmcoreinfo: 0
> nt_prstatus_percpu:
> 08536a34 08536ad8
>
> Elf32_Ehdr:
> e_ident: 177ELF
> e_ident[EI_CLASS]: 1 (ELFCLASS32)
> e_ident[EI_DATA]: 1 (ELFDATA2LSB)
> e_ident[EI_VERSION]: 1 (EV_CURRENT)
> e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
> e_ident[EI_ABIVERSION]: 0
> e_type: 4 (ET_CORE)
> e_machine: 3 (EM_386)
> e_version: 1 (EV_CURRENT)
> e_entry: 0
> e_phoff: 34
> e_shoff: 0
> e_flags: 0
> e_ehsize: 34
> e_phentsize: 20
> e_phnum: a
> e_shentsize: 0
> e_shnum: 0
> e_shstrndx: 0
> Elf32_Phdr:
> p_type: 4 (PT_NOTE)
> p_offset: 372 (174)
> p_vaddr: 0
> p_paddr: 0
> p_filesz: 1488 (5d0)
> p_memsz: 1488 (5d0)
> p_flags: 0 ()
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 1860 (744)
> p_vaddr: c0000000
> p_paddr: 0
> p_filesz: 655360 (a0000)
> p_memsz: 655360 (a0000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 657220 (a0744)
> p_vaddr: c0100000
> p_paddr: 100000
> p_filesz: 15728640 (f00000)
> p_memsz: 15728640 (f00000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 16385860 (fa0744)
> p_vaddr: c5000000
> p_paddr: 5000000
> p_filesz: 855638016 (33000000)
> p_memsz: 855638016 (33000000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 872023876 (33fa0744)
> p_vaddr: ffffffff
> p_paddr: 38000000
> p_filesz: 106950656 (65ff000)
> p_memsz: 106950656 (65ff000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 978974532 (3a59f744)
> p_vaddr: ffffffff
> p_paddr: 3e6c6000
> p_filesz: 15523840 (ece000)
> p_memsz: 15523840 (ece000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 994498372 (3b46d744)
> p_vaddr: ffffffff
> p_paddr: 3f59c000
> p_filesz: 581632 (8e000)
> p_memsz: 581632 (8e000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 995080004 (3b4fb744)
> p_vaddr: ffffffff
> p_paddr: 3f62e000
> p_filesz: 503808 (7b000)
> p_memsz: 503808 (7b000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 995583812 (3b576744)
> p_vaddr: ffffffff
> p_paddr: 3f6e9000
> p_filesz: 16384 (4000)
> p_memsz: 16384 (4000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Phdr:
> p_type: 1 (PT_LOAD)
> p_offset: 995600196 (3b57a744)
> p_vaddr: ffffffff
> p_paddr: 3f6ff000
> p_filesz: 4096 (1000)
> p_memsz: 4096 (1000)
> p_flags: 7 (PF_X|PF_W|PF_R)
> p_align: 0
> Elf32_Nhdr:
> n_namesz: 5 ("CORE")
> n_descsz: 144
> n_type: 1 (NT_PRSTATUS)
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000401
> c06fef00 00000001 0000f004 0000f008
> 00000000 c06e3ecc 00000282 00000282
> 00000024 c06e3fa4 00000068 00000000
> Elf32_Nhdr:
> n_namesz: 5 ("CORE")
> n_descsz: 144
> n_type: 1 (NT_PRSTATUS)
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000dbd 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000 00000000
> 00000000 00000000 c0712420 00007e7e
> 00000000 00000063 00000000 f0639f0c
> 00000063 0000007b 0000007b 000000d8
> 00000033 ffffffff c03145b2 00000060
> 00010086 f0639f0c 00000068 00000000
> Elf32_Nhdr:
> n_namesz: 11 ("VMCOREINFO")
> n_descsz: 1134
> n_type: 0 (unused)
> OSRELEASE=2.6.35.3-saber
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=c06f9120
> SYMBOL(node_online_map)=c0730644
> SYMBOL(swapper_pg_dir)=c06e4000
> SYMBOL(_stext)=c0101000
> SYMBOL(vmlist)=c07d3540
> SYMBOL(mem_map)=c07d3500
> SYMBOL(contig_page_data)=c072ce80
> SIZE(page)=32
> SIZE(pglist_data)=4224
> SIZE(zone)=1024
> SIZE(free_area)=44
> SIZE(list_head)=8
> SIZE(nodemask_t)=4
> OFFSET(page.flags)=0
> OFFSET(page._count)=4
> OFFSET(page.mapping)=16
> OFFSET(page.lru)=24
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=4140
> OFFSET(pglist_data.node_mem_map)=4144
> OFFSET(pglist_data.node_start_pfn)=4148
> OFFSET(pglist_data.node_spanned_pages)=4156
> OFFSET(pglist_data.node_id)=4160
> OFFSET(zone.free_area)=40
> OFFSET(zone.vm_stat)=728
> OFFSET(zone.spanned_pages)=916
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=4
> OFFSET(vm_struct.addr)=4
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=c06fc83c
> SYMBOL(log_end)=c07bb7ec
> SYMBOL(log_buf_len)=c06fc838
> SYMBOL(logged_chars)=c07c38a0
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=11
> NUMBER(PG_swapcache)=16
> CONFIG_X86_PAE=y
> CRASHTIME=1282584565
> cannot determine relocation value: not a live system
> gdb /usr/src/linux-2.6.35.3/vmlinux
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
>
> <readmem: c148a9a0, KVADDR, "kernel_config_data", 32768, (ROE), 8bad3d8>
> crash: read error: kernel virtual address: c148a9a0 type: "kernel_config_data"
> WARNING: cannot read kernel_config_data
> <readmem: c1487e28, KVADDR, "cpu_possible_mask", 4, (FOE), bfed4bbc>
> crash: read error: kernel virtual address: c1487e28 type: "cpu_possible_mask"

The read error with for the "kernel_config_data" symbol at c148a9a0 and (which returns on
error -- that's what ROE means), and then the "cpu_possible_mask" symbol at c1487e28 (which
cause the session to fault or bail out -- FOE), mean that -- after translating those virtual
addresses to physical addresses by stripping off the c0000000 unity-map identifier -- those
physical addresses (at 148a9a0 and 1487e28 respectively) were not found in the dumpfile.
And that's because the ELF header of the vmcore does not show a PT_LOAD segment
that contains those physical addresses.

But as I mentioned before, the virtual addresses seem to be too high for
static kernel data symbols. If you run --minimal, does the "sym" command
show "cpu_possible_mask" at that address? I don't have anything later than
a 2.6.34 x86 dumpfile to use as a reference, but the symbol is much lower
in value in that kernel:

crash> sym cpu_possible_mask
c07ffa28 (R) cpu_possible_mask
crash>

And if I dump all of the symbols from within a --minimal session with that
dumpfile, I see this, where the "_end" of the static kernel virtual memory
is at c0c77000:

crash> sym -l
... [ cut ] ...
c0b50ffc (b) netlbl_unlhsh_lock
c0b51000 (b) klist_remove_lock
c0b51004 (B) __bss_stop
c0b52000 (b) .brk
c0b52000 (B) __brk_base
c0b62000 (b) .brk.pagetables
c0c67000 (b) .brk.dmi_alloc
c0c77000 (B) __brk_limit
c0c77000 (A) _end
crash>

And if you look at the "VMCOREINFO" data above in your dump for items that are
kernel symbol values, they make sense, i.e.,

> SYMBOL(node_online_map)=c0730644
> SYMBOL(swapper_pg_dir)=c06e4000
> SYMBOL(_stext)=c0101000
> SYMBOL(vmlist)=c07d3540
> SYMBOL(mem_map)=c07d3500
> SYMBOL(contig_page_data)=c072ce80

If you run a --minimal session, what do you see when you run the
two commands that I show above? (i.e., "sym cpu_possible_mask" & the output
of the tail end of "sym -l")

Dave




But for starters, if you run the --minimal session and then execute the

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 07:33 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

Here's for sym cpu_possible_mask in minimal mode:
crash> sym cpu_possible_mask
c14e7d88 (R) cpu_possible_mask

And here's the tail of sym -l:
c175d4e0 (b) sunrpc_table_header
c175d4e4 (B) sctp_assocs_id_lock
c175d4e8 (B) proc_net_sctp
c175d4ec (B) sctp_assocs_id
c175d500 (B) sysctl_sctp_mem
c175d50c (B) sysctl_sctp_rmem
c175d518 (B) sysctl_sctp_wmem
c175d524 (b) __key.46606
c175d524 (b) sctp_ctl_sock
c175d528 (b) sctp_pf_inet6_specific
c175d52c (b) sctp_pf_inet_specific
c175d530 (b) sctp_af_v4_specific
c175d534 (b) sctp_af_v6_specific
c175d538 (b) __key.44408
c175d538 (b) sctp_rand.42824
c175d53c (B) sctp_sockets_allocated
c175d54c (b) sctp_memory_pressure
c175d550 (b) sctp_memory_allocated
c175d554 (b) sctp_sysctl_header
c175d558 (b) zero
c175d55c (b) klist_remove_lock
c175d560 (B) __bss_stop
c175e000 (b) .brk
c175e000 (B) __brk_base
c176e000 (b) .brk.pagetables
c17ee000 (b) .brk.dmi_alloc
c17fe000 (B) __brk_limit
c17fe000 (A) _end

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 07:57 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

I forgot that I had recompiled the kernel between my first mail and your
answer, so here's an up to date dump with the current kernel and last
dump done with it:




/var/crash/201008232109$ crash -d7 /usr/src/linux-2.6.35.3/vmlinux
vmcore.201008232109


crash 5.0.6
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: b782a4c0
header_size: 1844
num_pt_load_segments: 9
pt_load_segment[0]:
file_offset: 734
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a0734
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[2]:
file_offset: fa0734
phys_start: 5000000
phys_end: 38000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 33fa0734
phys_start: 38000000
phys_end: 3e628000
zero_fill: 0
pt_load_segment[4]:
file_offset: 3a5c8734
phys_start: 3e6db000
phys_end: 3f594000
zero_fill: 0
pt_load_segment[5]:
file_offset: 3b481734
phys_start: 3f59c000
phys_end: 3f62a000
zero_fill: 0
pt_load_segment[6]:
file_offset: 3b50f734
phys_start: 3f62e000
phys_end: 3f6a9000
zero_fill: 0
pt_load_segment[7]:
file_offset: 3b58a734
phys_start: 3f6e9000
phys_end: 3f6ed000
zero_fill: 0
pt_load_segment[8]:
file_offset: 3b58e734
phys_start: 3f6ff000
phys_end: 3f700000
zero_fill: 0
elf_header: 85368c0
elf32: 85368c0
notes32: 85368f4
load32: 8536914
elf64: 0
notes64: 0
load64: 0
nt_prstatus: 8536a34
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 2
vmcoreinfo: 0
size_vmcoreinfo: 0
nt_prstatus_percpu:
08536a34 08536ad8

Elf32_Ehdr:
e_ident: 177ELF
e_ident[EI_CLASS]: 1 (ELFCLASS32)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 3 (EM_386)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 34
e_shoff: 0
e_flags: 0
e_ehsize: 34
e_phentsize: 20
e_phnum: a
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf32_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 372 (174)
p_vaddr: 0
p_paddr: 0
p_filesz: 1472 (5c0)
p_memsz: 1472 (5c0)
p_flags: 0 ()
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 1844 (734)
p_vaddr: c0000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 657204 (a0734)
p_vaddr: c0100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 16385844 (fa0734)
p_vaddr: c5000000
p_paddr: 5000000
p_filesz: 855638016 (33000000)
p_memsz: 855638016 (33000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 872023860 (33fa0734)
p_vaddr: ffffffff
p_paddr: 38000000
p_filesz: 107118592 (6628000)
p_memsz: 107118592 (6628000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 979142452 (3a5c8734)
p_vaddr: ffffffff
p_paddr: 3e6db000
p_filesz: 15437824 (eb9000)
p_memsz: 15437824 (eb9000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 994580276 (3b481734)
p_vaddr: ffffffff
p_paddr: 3f59c000
p_filesz: 581632 (8e000)
p_memsz: 581632 (8e000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995161908 (3b50f734)
p_vaddr: ffffffff
p_paddr: 3f62e000
p_filesz: 503808 (7b000)
p_memsz: 503808 (7b000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995665716 (3b58a734)
p_vaddr: ffffffff
p_paddr: 3f6e9000
p_filesz: 16384 (4000)
p_memsz: 16384 (4000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995682100 (3b58e734)
p_vaddr: ffffffff
p_paddr: 3f6ff000
p_filesz: 4096 (1000)
p_memsz: 4096 (1000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 369c6c02 00000040
f70a650c f70a6000 f8098000 00000000
00000000 0000245c 00000000 00000000
00000000 00000040 f70a650c 00000000
c074dee4 c074dfa4 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000a9e 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 c077b6c0 00000303
00000000 00000063 00000000 f5a0ff0c
00000063 0000007b 0000007b 000000d8
00000033 ffffffff c0377692 00000060
00010086 f5a0ff0c 00000068 00000000
Elf32_Nhdr:
n_namesz: 11 ("VMCOREINFO")
n_descsz: 1117
n_type: 0 (unused)
OSRELEASE=2.6.35.3-saber
PAGESIZE=4096
SYMBOL(init_uts_ns)=c0762120
SYMBOL(node_online_map)=c0799944
SYMBOL(swapper_pg_dir)=c0816000
SYMBOL(_stext)=c0101000
SYMBOL(vmlist)=c083ed3c
SYMBOL(mem_map)=c083ed00
SYMBOL(contig_page_data)=c0796180
SIZE(page)=32
SIZE(pglist_data)=4224
SIZE(zone)=1024
SIZE(free_area)=44
SIZE(list_head)=8
SIZE(nodemask_t)=4
OFFSET(page.flags)=0
OFFSET(page._count)=4
OFFSET(page.mapping)=16
OFFSET(page.lru)=24
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=4140
OFFSET(pglist_data.node_mem_map)=4144
OFFSET(pglist_data.node_start_pfn)=4148
OFFSET(pglist_data.node_spanned_pages)=4156
OFFSET(pglist_data.node_id)=4160
OFFSET(zone.free_area)=40
OFFSET(zone.vm_stat)=728
OFFSET(zone.spanned_pages)=916
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=4
OFFSET(vm_struct.addr)=4
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=c07656bc
SYMBOL(log_end)=c08247ec
SYMBOL(log_buf_len)=c07656b8
SYMBOL(logged_chars)=c082c8a0
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
CRASHTIME=1282590563
cannot determine relocation value: not a live system
gdb /usr/src/linux-2.6.35.3/vmlinux
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

<readmem: c14ea900, KVADDR, "kernel_config_data", 32768, (ROE), 8bdda28>
crash: read error: kernel virtual address: c14ea900 type:
"kernel_config_data"

WARNING: cannot read kernel_config_data
<readmem: c14e7d88, KVADDR, "cpu_possible_mask", 4, (FOE), bf8cf28c>
crash: read error: kernel virtual address: c14e7d88 type:
"cpu_possible_mask"











And here's what you asked with --minimal:
crash> log
<readmem: c16656b8, KVADDR, "log_buf_len", 4, (FOE), bfa8f594>
log: read error: kernel virtual address: c16656b8 type: "log_buf_len"
crash> sym cpu_possible_mask
c14e7d88 (R) cpu_possible_mask
crash> sym kernel_config_data
c14ea900 (r) kernel_config_data
crash> sym -l
[...]
c175d4d0 (B) rpc_debug
c175d4d4 (B) nfs_debug
c175d4d8 (B) nfsd_debug
c175d4dc (B) nlm_debug
c175d4e0 (b) sunrpc_table_header
c175d4e4 (B) sctp_assocs_id_lock
c175d4e8 (B) proc_net_sctp
c175d4ec (B) sctp_assocs_id
c175d500 (B) sysctl_sctp_mem
c175d50c (B) sysctl_sctp_rmem
c175d518 (B) sysctl_sctp_wmem
c175d524 (b) __key.46606
c175d524 (b) sctp_ctl_sock
c175d528 (b) sctp_pf_inet6_specific
c175d52c (b) sctp_pf_inet_specific
c175d530 (b) sctp_af_v4_specific
c175d534 (b) sctp_af_v6_specific
c175d538 (b) __key.44408
c175d538 (b) sctp_rand.42824
c175d53c (B) sctp_sockets_allocated
c175d54c (b) sctp_memory_pressure
c175d550 (b) sctp_memory_allocated
c175d554 (b) sctp_sysctl_header
c175d558 (b) zero
c175d55c (b) klist_remove_lock
c175d560 (B) __bss_stop
c175e000 (b) .brk
c175e000 (B) __brk_base
c176e000 (b) .brk.pagetables
c17ee000 (b) .brk.dmi_alloc
c17fe000 (B) __brk_limit
c17fe000 (A) _end

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 07:58 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> Here's for sym cpu_possible_mask in minimal mode:
> crash> sym cpu_possible_mask
> c14e7d88 (R) cpu_possible_mask
>
> And here's the tail of sym -l:
> c175d4e0 (b) sunrpc_table_header
> c175d4e4 (B) sctp_assocs_id_lock
> c175d4e8 (B) proc_net_sctp
> c175d4ec (B) sctp_assocs_id
> c175d500 (B) sysctl_sctp_mem
> c175d50c (B) sysctl_sctp_rmem
> c175d518 (B) sysctl_sctp_wmem
> c175d524 (b) __key.46606
> c175d524 (b) sctp_ctl_sock
> c175d528 (b) sctp_pf_inet6_specific
> c175d52c (b) sctp_pf_inet_specific
> c175d530 (b) sctp_af_v4_specific
> c175d534 (b) sctp_af_v6_specific
> c175d538 (b) __key.44408
> c175d538 (b) sctp_rand.42824
> c175d53c (B) sctp_sockets_allocated
> c175d54c (b) sctp_memory_pressure
> c175d550 (b) sctp_memory_allocated
> c175d554 (b) sctp_sysctl_header
> c175d558 (b) zero
> c175d55c (b) klist_remove_lock
> c175d560 (B) __bss_stop
> c175e000 (b) .brk
> c175e000 (B) __brk_base
> c176e000 (b) .brk.pagetables
> c17ee000 (b) .brk.dmi_alloc
> c17fe000 (B) __brk_limit
> c17fe000 (A) _end

That's interesting -- did you add some huge data structure or something
to the kernel?

OK -- three more requests -- can you bring up the --minimal session, and
then do this:

crash> sym -l > sym.l

and send the "sym.l" file? (It's long, so send it as an attachment)

Secondly, send the output of:

# crash -d8 /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930

The -d8 output will also show the physical address translation,
like this:

<readmem: c07ffa28, KVADDR, "cpu_possible_mask", 4, (FOE), bff3d43c>
addr: c07ffa28 paddr: 7ffa28 cnt: 4

And third, send the output of:

# readelf -a vmcore.201008231930

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 08:08 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> I forgot that I had recompiled the kernel between my first mail and your
> answer, so here's an up to date dump with the current kernel and last
> dump done with it:

No problem -- I still need the output of the 3 requests in my last
email, plus one more question: can you run crash on the live system?

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 08:09 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

crash> sym -l> sym.l

and send the "sym.l" file? (It's long, so send it as an attachment)

Secondly, send the output of:

# crash -d8 /usr/src/linux-2.6.35.3/vmlinux vmcore.201008231930

The -d8 output will also show the physical address translation,
like this:

<readmem: c07ffa28, KVADDR, "cpu_possible_mask", 4, (FOE), bff3d43c>
addr: c07ffa28 paddr: 7ffa28 cnt: 4

And third, send the output of:

# readelf -a vmcore.201008231930

I have joined all 3 files to this email.

ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: CORE (Core file)
Machine: Intel 80386
Version: 0x1
Entry point address: 0x0
Start of program headers: 52 (bytes into file)
Start of section headers: 0 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 10
Size of section headers: 0 (bytes)
Number of section headers: 0
Section header string table index: 0

There are no sections in this file.

There are no sections in this file.

Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x000174 0x00000000 0x00000000 0x005d0 0x005d0 0
LOAD 0x000744 0xc0000000 0x00000000 0xa0000 0xa0000 RWE 0
LOAD 0x0a0744 0xc0100000 0x00100000 0xf00000 0xf00000 RWE 0
LOAD 0xfa0744 0xc5000000 0x05000000 0x33000000 0x33000000 RWE 0
LOAD 0x33fa0744 0xffffffff 0x38000000 0x65ff000 0x65ff000 RWE 0
LOAD 0x3a59f744 0xffffffff 0x3e6c6000 0xece000 0xece000 RWE 0
LOAD 0x3b46d744 0xffffffff 0x3f59c000 0x8e000 0x8e000 RWE 0
LOAD 0x3b4fb744 0xffffffff 0x3f62e000 0x7b000 0x7b000 RWE 0
LOAD 0x3b576744 0xffffffff 0x3f6e9000 0x04000 0x04000 RWE 0
LOAD 0x3b57a744 0xffffffff 0x3f6ff000 0x01000 0x01000 RWE 0

There is no dynamic section in this file.

There are no relocations in this file.

There are no unwind sections in this file.

No version information found in this file.

Notes at offset 0x00000174 with length 0x000005d0:
Owner Data size Description
CORE 0x00000090 NT_PRSTATUS (prstatus structure)
CORE 0x00000090 NT_PRSTATUS (prstatus structure)
VMCOREINFO 0x0000046e Unknown note type: (0x00000000)
--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 08:14 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

On 08/23/2010 22:08, Dave Anderson wrote:


----- "Paul-Kenji Cahier Furuya"<pkc@f1-photo.com> wrote:


I forgot that I had recompiled the kernel between my first mail and your
answer, so here's an up to date dump with the current kernel and last
dump done with it:


No problem -- I still need the output of the 3 requests in my last
email, plus one more question: can you run crash on the live system?
Wouldn't running it live require a physical/serial tty? I do not have
such since this is a remote computer.

If not, please tell me what course of action to take to do that.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 08:20 PM
Dave Anderson
 
Default Crash issue when loading vmcore

----- "Paul-Kenji Cahier Furuya" <pkc@f1-photo.com> wrote:

> On 08/23/2010 22:08, Dave Anderson wrote:
> >
> > ----- "Paul-Kenji Cahier Furuya"<pkc@f1-photo.com> wrote:
> >
> >> I forgot that I had recompiled the kernel between my first mail and your
> >> answer, so here's an up to date dump with the current kernel and last
> >> dump done with it:
> >
> > No problem -- I still need the output of the 3 requests in my last
> > email, plus one more question: can you run crash on the live system?
> Wouldn't running it live require a physical/serial tty? I do not have
> such since this is a remote computer.
> If not, please tell me what course of action to take to do that.

Well, yes, you'd have to be able to log into the machine, and
then just run:

# crash

or if the /vmlinux file is not in a common location, do this:

# crash /path/to/vmlinux

And that presumes you've got crash installed on the system as well.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 08-23-2010, 08:23 PM
Paul-Kenji Cahier Furuya
 
Default Crash issue when loading vmcore

On 08/23/2010 22:20, Dave Anderson wrote:

Well, yes, you'd have to be able to log into the machine, and
then just run:

# crash

or if the /vmlinux file is not in a common location, do this:

# crash /path/to/vmlinux

And that presumes you've got crash installed on the system as well.

I might be able to get a physical access at some point, but right now I
have none.


Anything that helps from the logs?

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 08:51 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org