Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Crash Utility (http://www.linux-archive.org/crash-utility/)
-   -   (pvops 2.6.32.21) crash: cannot read/find cr3 page (http://www.linux-archive.org/crash-utility/425576-pvops-2-6-32-21-crash-cannot-read-find-cr3-page.html)

tom anderson 09-10-2010 11:32 PM
(pvops 2.6.32.21) crash: cannot read/find cr3 page
 
if I* use*a*pvops domU kernel version 2.6.32.18 crash works fine.* However if I use a pvops domU kernel version 2.6.32.21*I get the error messages:
*
* * crash: cannot find mfn 874307 (0xd5743) in page index**
*** crash: cannot read/find cr3 page

Any suggestions as to what is wrong?
*
-Thomas
*
*
(1) domU information:
*** cat /proc/version
*** Linux version 2.6.32.21-1 (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #1 SMP Tue Sep 7 15:40:55 PDT 2010

*** cat /proc/cmdline
*** root=UUID=ee471711-e94d-4ff1-973c-e45526adea25 ro* crashkernel=384M-2G:64M,2G-:128M* iommu=soft swiotlb=force* console=hvc0,115200n8 ip=:127.0.255.255::::eth0:dhcp

****cat /sys/kernel/kexec_crash_loaded
****1

**** xen.cfg is using on_crash*** = 'coredump-restart'
***
**** core is generated by executing* echo c > /proc/sysrq-trigger on the domU.
*
*
(2) core file is generated by the domU crash
*** -rw------- 1 root root 2145653832 Sep* 9 09:42 /var/xen/dump/xyz/2010-0909-0942.13-xyz.1.core

*
(3) crash information
** crash-d99 /boot/vmlinux-2.6.32.21-1 /var/xen/dump/xyz/2010-0909-0942.13-xyz.1.core*
**
***crash 5.0.7
***...
** crash: /var/xen/dump/xyz/2010-0909-0942.13-xyz.1.core: not a netdump ELF dumpfile
** crash: /var/xen/dump/xyz/2010-0909-0942.13-xyz.1.core: not a kdump ELF dumpfile

*********** flags: 109 (XENDUMP_LOCAL|XC_CORE_ELF|XC_CORE_P2M_CREATE)
************* xfd: 3
******* page_size: 4096
************* ofp: 0
************ page: 1d67c70
******** panic_pc: 0
******** panic_sp: 0
******** accesses: 0
****** cache_hits: 0
******** last_pfn: -1
******* redundant: 0
******* poc[5000]: 1d68c80 (none used)
***
********* xc_save:
********************* nr_pfns: 0 (0x0)
*************** vmconfig_size: 0 (0x0)
**************** vmconfig_buf: 0
************** p2m_frame_list: 0 (none)
******************** pfns_not: 0
************* pfns_not_offset: 0
************ vcpu_ctxt_offset: 0
***** shared_info_page_offset: 0
************* region_pfn_type: 0
***************** batch_count: 0
*************** batch_offsets: 0 (none)
**************** ia64_version: 0
*********** ia64_page_offsets: 0 (none)
***
********* xc_core:
********************** header:
******************* xch_magic: f00febed (XC_CORE_MAGIC)
**************** xch_nr_vcpus: 7
**************** xch_nr_pages: 521792 (0x7f640)
************* xch_ctxt_offset: 1896 (0x768)
************ xch_index_offset: 2137305088 (0x7f64b000)
************ xch_pages_offset: 45056 (0xb000)
******************* elf_class: ELFCLASS64
*********** elf_strtab_offset: 2145653760 (0x7fe41400)
************** format_version: 0000000000000001
********** shared_info_offset: 38072 (0x94b8)
********** elf_index_pfn[128]:
*** 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1
*** 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1
*** 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1 0:-1
****************** last_batch:
*********************** index: 0 (0 - 0)
******************** accesses: 0
****************** duplicates: 0
*********************** elf32: 0
*********************** elf64: 1d67040
****************** p2m_frames: 0
******** p2m_frame_index_list:
***
*** Elf64_Ehdr:
******************* e_ident: 177ELF
********* e_ident[EI_CLASS]: 2 (ELFCLASS64)
********** e_ident[EI_DATA]: 1 (ELFDATA2LSB)
******* e_ident[EI_VERSION]: 1 (EV_CURRENT)
********* e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
**** e_ident[EI_ABIVERSION]: 1
******************** e_type: 4 (ET_CORE)
***************** e_machine: 62 (EM_X86_64)
***************** e_version: 1 (EV_CURRENT)
******************* e_entry: 0
******************* e_phoff: 0
******************* e_shoff: 40
******************* e_flags: 0
****************** e_ehsize: 40
*************** e_phentsize: 38
******************* e_phnum: 0
*************** e_shentsize: 40
******************* e_shnum: 7
**************** e_shstrndx: 1
***
***
*** Elf64_Shdr:
******************* sh_name: 0 ""
******************* sh_type: 0 (SHT_NULL)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 0
******************* sh_size: 0
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 0
**************** sh_entsize: 0
***
*** Elf64_Shdr:
******************* sh_name: 1 ".shstrtab"
******************* sh_type: 3 (SHT_STRTAB)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 7fe41400
******************* sh_size: 48
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 0
**************** sh_entsize: 0
**************************** .shstrtab
**************************** .note.Xen
**************************** .xen_prstatus
**************************** .xen_shared_info
**************************** .xen_pages
**************************** .xen_p2m
***
*** Elf64_Shdr:
******************* sh_name: b ".note.Xen"
******************* sh_type: 7 (SHT_NOTE)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 200
******************* sh_size: 568
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 0
**************** sh_entsize: 0
******************** namesz: 4
********************* descz: 0
********************** type: 2000000 (XEN_ELFNOTE_DUMPCORE_NONE)
********************** name: Xen
**************************** (empty)
******************** namesz: 4
********************* descz: 32
********************** type: 2000001 (XEN_ELFNOTE_DUMPCORE_HEADER)
********************** name: Xen
**************************** 00000000f00febed 0000000000000007
**************************** 000000000007f640 0000000000001000
******************** namesz: 4
********************* descz: 1280
********************** type: 2000002 (XEN_ELFNOTE_DUMPCORE_XEN_VERSION)
***
**************************** 0000000000000004 0000000000000000
**************************** ff003463722d312e 2400000000000008
**************************** 7372657620636367 2e342e34206e6f69
**************************** 746e756255282033 2d332e342e342075
**************************** 3575746e75627534 ffff82c480002029
**************************** 0000000000000001 0000000001aa0660
**************************** ffff006567646562 0000000000000001
**************************** 006d6f632e69736c ffff880000000001
**************************** ffff82c480367c0e ffff82c400000100
**************************** 20677541206e7553 34343a3930203120
**************************** 205444502039323a ffff820030313032
**************************** 2d302e332d6e6578 782034365f363878
**************************** 782d302e332d6e65 68207032335f3638
**************************** 782d302e332d6d76 76682032335f3638
**************************** 38782d302e332d6d 7668207032335f36
**************************** 38782d302e332d6d 0000002034365f36
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 0000000000000000 0000000000000000
**************************** 616c696176616e75 7820343600656c62
**************************** 782d302e332d6e65 68207032335f3638
**************************** 782d302e332d6d76 76682032335f3638
**************************** 38782d302e332d6d 7668207032335f36
**************************** ffff800000000000 0000000000001000
******************** namesz: 4
********************* descz: 8
********************** type: 2000003 (XEN_ELFNOTE_DUMPCORE_FORMAT_VERSION)
********************** name: Xen
**************************** 0000000000000001
***
*** Elf64_Shdr:
******************* sh_type: 1 (SHT_PROGBITS)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 768
******************* sh_size: 8d50
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 8
**************** sh_entsize: 1430
***
*** Elf64_Shdr:
******************* sh_name: 23 ".xen_shared_info"
******************* sh_type: 1 (SHT_PROGBITS)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 94b8
******************* sh_size: 1000
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 8
**************** sh_entsize: 1000
***
*** Elf64_Shdr:
******************* sh_name: 34 ".xen_pages"
******************* sh_type: 1 (SHT_PROGBITS)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: b000
******************* sh_size: 7f640000
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 1000
**************** sh_entsize: 1000
***
*** Elf64_Shdr:
******************* sh_name: 3f ".xen_p2m"
******************* sh_type: 1 (SHT_PROGBITS)
****************** sh_flags: 0
******************* sh_addr: 0
***************** sh_offset: 7f64b000
******************* sh_size: 7f6400
******************* sh_link: 0
******************* sh_info: 0
************** sh_addralign: 8
**************** sh_entsize: 10
***
*** crash: pv_init_ops exists: ARCH_PVOPS
*** gdb /boot/vmlinux-2.6.32.21-1
*** GNU gdb (GDB) 7.0
*** Copyright (C) 2009 Free Software Foundation, Inc.
*** License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
*** This is free software: you are free to change and redistribute it.
*** There is NO WARRANTY, to the extent permitted by law.* Type "show copying"
*** and "show warranty" for details.
*** This GDB was configured as "x86_64-unknown-linux-gnu"...
*** GETBUF(248 -> 0)
***** GETBUF(1500 -> 1)
***
***** FREEBUF(1)
*** FREEBUF(0)
*** <readmem: ffffffff81614800, KVADDR, "kernel_config_data", 32768, (ROE), 2fed090>
******* addr: ffffffff81614800* paddr: 1614800* cnt: 2048
*** GETBUF(248 -> 0)
*** FREEBUF(0)
*** MEMBER_OFFSET(vcpu_guest_context, ctrlreg): 4984
*** ctrlreg[0]: 80050033
*** ctrlreg[1]: d5742000
*** ctrlreg[2]: 0
*** ctrlreg[3]: d5743000
*** ctrlreg[4]: 2660
*** ctrlreg[5]: 0
*** ctrlreg[6]: 0
*** ctrlreg[7]: 0
*** crash: cannot find mfn 874307 (0xd5743) in page index
***
*** crash: cannot read/find cr3 page
***

*
*

*
*
*

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

tom anderson 09-13-2010 04:52 PM
(pvops 2.6.32.21) crash: cannot read/find cr3 page
 
*


> Date: Mon, 13 Sep 2010 11:59:44 -0400
> From: anderson@redhat.com
> To: crash-utility@redhat.com
> Subject: Re: [Crash-utility] (pvops 2.6.32.21) crash: cannot read/find cr3 page
>
> ----- "tom anderson" <xentoma@hotmail.com > wrote:
>
> > if I use a pvops domU kernel version 2.6.32.18 crash works fine. However if I
> > use a pvops domU kernel version 2.6.32.21 I get the error messages:
> >
> > crash: cannot find mfn 874307 (0xd5743) in page index
> > crash: cannot read/find cr3 page
> >
> > Any suggestions as to what is wrong?
>
> Hi Tom,
>
> I can't really give you specific suggestions as to what is wrong,
> but at least tell what the crash utility is encountering.
>
> I suppose there's good news and bad news concerning this issue,
> the good news being that it worked OK with 2.6.32.18, which is
> fairly close to your failing 2.6.32.21. Since I've done very little
> with Xen support since Red Hat dropped Xen development beyond our
> RHEL5 2.6.18-era release, it's always good to hear that it actually
> still worked with a 2.6.32.18 kernel. I imagine eventually something
> will break in the future, and at that time I may likely require outside
> assistance to keep Xen support in place.
>
> Anyway, that all being said, in your failure case, here are the issues
> at hand. The header shows this:
>
> xc_core:
> header:
> xch_magic: f00febed (XC_CORE_MAGIC)
> xch_nr_vcpus: 7
> xch_nr_pages: 521792 (0x7f640)
> xch_ctxt_offset: 1896 (0x768)
> xch_index_offset: 2137305088 (0x7f64b000)
> xch_pages_offset: 45056 (0xb000)
> elf_class: ELFCLASS64
> elf_strtab_offset: 2145653760 (0x7fe41400)
> format_version: 0000000000000001
> shared_info_offset: 38072 (0x94b8)
>
> The "xch_nr_pages" indicates that the domU vmlinux kernel has 521792
> pseudo-physical pages assigned to it, where those pseudo-physical pages
> are backed by the Xen hypervisor by machine pages, which are the "real"
> physical pages. And so when the crash utility needs to access a
> pseudo-physical page used by a domU kernel, that pseudo-physical page
> needs to be translated to the actual machine physical page that backs it,
> and then that physical page needs to be found in the dumpfile. The PFN
> (page frame number) of the pseudo-physical pages are call "pfns" and the
> PFN of the machine pages are called "mfns" or "gmfns".
>
> To match a pfn with its corresponding mfn, the kdump operation dumps an
> array of pfn-to-mfn pairs in the vmcore's ".xen_p2m" section, this taken from
> http://www.sfr-fresh.com/unix/misc/xen-4.0.1.tar.gz:a/xen-4.0.1/docs/misc/dump-core-format.txt
>
> ".xen_p2m" section
> name ".xen_p2m"
> type SHT_PROGBITS
> structure array of struct xen_dumpcore_p2m
> struct xen_dumpcore_p2m {
> uint64_t pfn;
> uint64_t gmfn;
> };
>
> description
> This elements represents the frame number of the page
> in .xen_pages section.
> pfn: guest-specific pseudo-physical frame number
> gmfn: machine physical frame number
> The size of arrays is stored in xch_nr_pages member of header
> note descriptor in .note.Xen note section.
> The entryies are stored in pfn-ascending order.
> This section must exist when the domain is non auto
> translated physmap mode. Currently x86 paravirtualized domain.
>
> The "pfn" value associated with the "gmfn" value, is in turn used
> as an index into an array of actual pages in the dumpfile, which is
> found at the "xch_pages_offset" at 45056 (0xb000).
>
> The start of the index array is found in the dumpfile at the "xch_index_offset"
> at 2137305088 (0x7f64b000), and ends at the "elf_strtab_offset" at 2145653760
> (0x7fe41400). Accordingly, if you subtract 2137305088 from 2145653760,
> the array of xen_dumpcore_p2m structures is 8348672 bytes, which when
> divided by the size of the data structure (16), it equals the value of
> "xch_nr_pages", or 521792.
>
> Anyway, the very first read attempt requires the crash utility to do a
> one-time-only recreation of the kernel's "p2m_top" array (pvops kernels only),
> and in so doing needs to first read the page found in the hypervisor's cr3
> register, which contains a machine address:
>
> <readmem: ffffffff81614800, KVADDR, "kernel_config_data", 32768, (ROE), 2fed090>
> addr: ffffffff81614800 paddr: 1614800 cnt: 2048
> GETBUF(248 -> 0)
> FREEBUF(0)
> MEMBER_OFFSET(vcpu_guest_context, ctrlreg): 4984
> ctrlreg[0]: 80050033
> ctrlreg[1]: d5742000
> ctrlreg[2]: 0
> ctrlreg[3]: d5743000
> ctrlreg[4]: 2660
> ctrlreg[5]: 0
> ctrlreg[6]: 0
> ctrlreg[7]: 0
> crash: cannot find mfn 874307 (0xd5743) in page index
>
> crash: cannot read/find cr3 page
>
> It contained a machine address of d5743000, which when shifted-right equates
> to an PFN (or "mfn") of 874307 (0xd5743). It then walked through the index
> array of xen_dumpcore_p2m structures in the dumpfile, looking for the one that
> contains that "gmfn" value.
>
> But for whatever reason, it could not find it. That being the
> case, there's no way it can continue.
>
> I can't really help much more than that. The function that
> walks through the array is xc_core_mfn_to_page() in xendump.c.
> It prints the "cannot find mfn ..." message, and returns back
> to the x86_64_pvops_xendump_p2m_create() function in x86_64.c,
> which prints the final, fatal, "cannot read/find cr3 page"
> message.
>
> If you capture the same type of debug output with the earlier
> kernel, you should see it get to the point above and continue
> on from there.
>
> Dave
>
> --
> Crash-utility mailing list
> Crash-utility@redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility

*

Dave,

*

Thanks for your response and providing such detailed information relating to the issue at hand.

*

-Thomas


*

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility


All times are GMT. The time now is 08:45 PM.

VBulletin, Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.