Issues with ps -a/vtop
 

Hello All,
Â*
I ran into this problem were ps –a/vtop can’t access user space.Â* Looking at the change log it appears a fix was put in as 4.0-3.7.Â* But I’m seeing this issue with 5.1.3Â*
crash> ps -a 23591
PID: 23591Â* TASK: ffff8103813be7a0Â* CPU: 5Â*Â* COMMAND: "python"
ps: cannot access user stack address: 7fff7d89a75f
Â*
crash> vtop 7fff7d89a75f
VIRTUALÂ*Â*Â*Â* PHYSICAL
7fff7d89a75fÂ* (not accessible)
Â*
This is on a RHEL 5.4 with kernel 2.6.18-194.11.3.el5.Â* What information will needed to debug this? Or am I using these commands incorrectly?
Â*
Thanks in advance for any help you can give me.
Â*
Steven Soulen
Â*

Unless otherwise indicated, this message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product or service, an official confirmation of any transaction, or as an official statement of the entity sending this message. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Issues with ps -a/vtop
 

----- Original Message -----
> Hello All,
>
> I ran into this problem were ps –a/vtop can’t access user space.
> Looking at the change log it appears a fix was put in as 4.0-3.7. But
> I’m seeing this issue with 5.1.3
>
> crash> ps -a 23591
>
> PID: 23591 TASK: ffff8103813be7a0 CPU: 5 COMMAND: "python"
> ps: cannot access user stack address: 7fff7d89a75f
>
> crash> vtop 7fff7d89a75f
> VIRTUAL PHYSICAL
> 7fff7d89a75f (not accessible)
>
> This is on a RHEL 5.4 with kernel 2.6.18-194.11.3.el5. What
> information will needed to debug this? Or am I using these commands
> incorrectly?
>
> Thanks in advance for any help you can give me.
>
> Steven Soulen

Hi Steven,

If the vmcore was post-processed by makedumpfile -- which had filtered
out user-space pages -- then the attempted read of a user stack address
by the "ps -a" command would cause a failure like yours did.

But the vtop command should be able to do a translation down to the
PTE of the "filtered-out" user space page. When "vtop" fails with
the "(not accessible)" message, it means that the virtual address
doesn't fit into any of the current context's VMAs. So when you did
the "vtop 7fff7d89a75f", were you in pid 23591's context? In other
words, you would have to do either this:

crash> set 23591
...
crash> vtop 7fff7d89a75f
...

or alternatively, this:

crash> vtop -c 23591 7fff7d89a75f

Dave


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Issues with ps -a/vtop
 

Hi Dave,
Â*
Thanks for the quick response.Â* That was the issue I was having. I am now able to use vtop.Â*
Â*
crash> vtop -c 23619 7fff7efa16a6
VIRTUALÂ*Â*Â*Â* PHYSICAL
7fff7efa16a6Â* 201d126a6
Â*
Â*Â* PML: 2a948d7f8 => 3f031a067
Â*Â* PUD: 3f031afe8 => 1ed481067
Â*Â* PMD: 1ed481fb8 => 2c9af2067
Â*Â* PTE: 2c9af2d08 => 201d12005
Â* PAGE: 201d12000
Â*
Â*Â* PTEÂ*Â*Â*Â* PHYSICALÂ*Â* FLAGS
201d12005Â* 201d12000Â* (PRESENT|USER)
Â*
Â*Â*Â*Â*Â* VMAÂ*Â*Â*Â*Â*Â*Â*Â*Â*Â* STARTÂ*Â*Â*Â*Â*Â* ENDÂ*Â*Â*Â* FLAGS FILE
ffff81041195dce8 7fff7efa0000 7fff7efa2000 100173
Â*
Â*Â*Â*Â*Â* PAGEÂ*Â*Â*Â*Â*Â*Â* PHYSICALÂ*Â*Â*Â*Â* MAPPINGÂ*Â*Â*Â*Â*Â* INDEX CNT FLAGS
ffff810107065bf0 201d12000 ffff8101eee276f1 7fffffffeÂ* 1 200100000000278
Â*
So how do I get at the information that is proved by ps –a?Â* What I’m really after is the arguments of the command.
Â*
Thanks again.
Â*
Steven Soulen

Unless otherwise indicated, this message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product or service, an official confirmation of any transaction, or as an official statement of the entity sending this message. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Issues with ps -a/vtop
 

----- Original Message -----
> Hi Dave,
>
> Thanks for the quick response. That was the issue I was having. I am
> now able to use vtop.
>
> crash> vtop -c 23619 7fff7efa16a6
> VIRTUAL PHYSICAL
> 7fff7efa16a6 201d126a6

> PML: 2a948d7f8 => 3f031a067
> PUD: 3f031afe8 => 1ed481067
> PMD: 1ed481fb8 => 2c9af2067
> PTE: 2c9af2d08 => 201d12005
> PAGE: 201d12000
>
> PTE PHYSICAL FLAGS
> 201d12005 201d12000 (PRESENT|USER)

> VMA START END FLAGS FILE
> ffff81041195dce8 7fff7efa0000 7fff7efa2000 100173
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> ffff810107065bf0 201d12000 ffff8101eee276f1 7fffffffe 1 200100000000278

> So how do I get at the information that is proved by ps –a? What I’m
> really after is the arguments of the command.

If the user stack page containing the information needed by "ps -a"
is not in the dumpfile, then you just can't get it. I'm presuming that
your dumpfile has had its user-pages filtered out. Or less likely but
possible, that user stack page happened to be swapped out at the time
the crash occurred. But if you do a "ps -a" and all user tasks generate
the "cannot access user stack" messages, then they must have been filtered.

Note that if it's a compressed kdump dumpfile, then you can see what's
been filtered out, like this RHEL5 vmcore example, which was post-processed
with "makedumpfile -c -d31":

crash> help -n | grep -e " flags" -e dump_level
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE _CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FRE E)
crash>

If the "-c" (compressed) flag was not used, then the resultant vmcore
would still be in ELF format, but there's no readily available indication
of what types of pages (if any) have been filtered out. The only thing
you would typically see is that "help -n" would show a very large number
of PT_LOAD segments if it was filtered, whereas if there's just a handful
of PT_LOAD segments, then it's probably a full dump.

Dave

> Thanks again.
> Steven Soulen


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Issues with ps -a/vtop
 

Hi Dave,
Â*
The user pages have been filtered out.Â* Is this documented anywhere?
Â*
On a side note does anyone have suggestions for documentation/books I should read for debugging crash dumps?Â* (I’ve already read the whitepaper on using crash)
Â*
Thanks again
Â*
Steven Soulen

Unless otherwise indicated, this message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product or service, an official confirmation of any transaction, or as an official statement of the entity sending this message. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Issues with ps -a/vtop
 

----- Original Message -----
> Hi Dave,
>
>
>
> The user pages have been filtered out. Is this documented anywhere?

The filtering process is documented by the makedumpfile command itself
by entering "makedumpfile --help". (I don't think there is a man page
for it). And the use of makedumpfile by the kexec-kdump facility is
documented in /usr/share/doc/kexec-tools-2.0.0/kexec-kdump-howto.txt.
Both are contained within the RHEL kexec-tools package.

> On a side note does anyone have suggestions for documentation/books I
> should read for debugging crash dumps? (I’ve already read the
> whitepaper on using crash)
>
> Thanks again
> Steven Soulen

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility