FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Crash Utility

 
 
LinkBack Thread Tools
 
Old 01-25-2011, 07:53 PM
Bob Montgomery
 
Default RFC: Improving crash's search speed

I like using the search command in crash, but noticed that it takes much
longer than my own dump search tools.

As an example, on a level 31 (lots of exclusions) dump from a 4 GB
x86_64 system, this search command in (benchmarked with a command file)
finds these results and takes almost 2 minutes:

$ cat cmdfile2
search -k 0xffff88012491b240

$ time crash-5.1.1 kernel_link dump.201101161845 <cmdfile2
...

ffff88012491b280: ffff88012491b240
ffff880126dcca18: ffff88012491b240
ffff880126dcca20: ffff88012491b240
ffff880126dcca30: ffff88012491b240
ffff880126dcca88: ffff88012491b240
ffff880126eca148: ffff88012491b240
ffffc9000468d148: ffff88012491b240

real 1m52.834s
user 1m50.571s
sys 0m2.220s

When you watch it search, the first 6 results come out in a few seconds,
then nothing happens for a long time.

The first six are coming from searching the identity mapped region which
covers every page in the dump. Note the forms of the addresses for the
first six hits.

The majority of the search time is spent going through the kernel
vmalloc address range and checking to see if pages are mapped to any of
those addresses. Any page searched through these addresses should have
already been searched in the identity mapped search.

So, for the last hit: (ffffc9000468d148: ffff88012491b240), converting
to physical, and then back to identity-mapped virtual gives:

crash-5.1.1> vtop 0xffffc9000468d148
VIRTUAL PHYSICAL
ffffc9000468d148 126eca148
...

And:
crash-5.1.1> ptov 0x126eca148
VIRTUAL PHYSICAL
ffff880126eca148 126eca148

And so the hit at 0xffffc9000468d148 was already caught by the earlier
hit in the identity-mapped range:
ffff880126eca148: ffff88012491b240

If you don't want to wait, you can find the vmalloc_start_addr from
"help -m" and use it to restrict the search range:

$ cat cmdfile2a
search -k -e 0xffffc90000000000 0xffff88012491b240

$ time crash-5.1.1 dump.201101161845 kernel_link <cmdfile2a

...
ffff88012491b280: ffff88012491b240
ffff880126dcca18: ffff88012491b240
ffff880126dcca20: ffff88012491b240
ffff880126dcca30: ffff88012491b240
ffff880126dcca88: ffff88012491b240
ffff880126eca148: ffff88012491b240

real 0m4.243s
user 0m4.088s
sys 0m0.156s

This command finishes with the first 6 hits in 4 seconds.

Once you have those hits, if you have to know if any virtual mappings
exist, you can use kmem on the physical address:

crash-5.1.1> vtop 0xffff880126eca148
VIRTUAL PHYSICAL
ffff880126eca148 126eca148
...

crash-5.1.1> kmem -v 126eca148
VM_STRUCT ADDRESS RANGE SIZE
ffff8801277f8180 ffffc90004682000 - 1052672

Which shows the virtual range that contains the mapping for the page.
Then this command takes no time:

crash-5.1.1> search -k -s ffffc90004682000 -e ffffc90004783000
ffff88012491b240
ffffc9000468d148: ffff88012491b240



I think the drastic reduction of search time from 2 minutes to 4 seconds
is interesting enough to warrant a shortcut.

The attached patch implements the -K option that is the same as doing
"-k -e <vmalloc_start_addr>".

Comments?
Bob Montgomery











--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-25-2011, 08:50 PM
Dave Anderson
 
Default RFC: Improving crash's search speed

----- Original Message -----
> I like using the search command in crash, but noticed that it takes
> much
> longer than my own dump search tools.
>
> As an example, on a level 31 (lots of exclusions) dump from a 4 GB
> x86_64 system, this search command in (benchmarked with a command file)
> finds these results and takes almost 2 minutes:
>
> $ cat cmdfile2
> search -k 0xffff88012491b240
>
> $ time crash-5.1.1 kernel_link dump.201101161845 <cmdfile2
> ...
>
> ffff88012491b280: ffff88012491b240
> ffff880126dcca18: ffff88012491b240
> ffff880126dcca20: ffff88012491b240
> ffff880126dcca30: ffff88012491b240
> ffff880126dcca88: ffff88012491b240
> ffff880126eca148: ffff88012491b240
> ffffc9000468d148: ffff88012491b240
>
> real 1m52.834s
> user 1m50.571s
> sys 0m2.220s
>
> When you watch it search, the first 6 results come out in a few seconds,
> then nothing happens for a long time.
>
> The first six are coming from searching the identity mapped region which
> covers every page in the dump. Note the forms of the addresses for the
> first six hits.
>
> The majority of the search time is spent going through the kernel
> vmalloc address range and checking to see if pages are mapped to any of
> those addresses. Any page searched through these addresses should have
> already been searched in the identity mapped search.
>
> So, for the last hit: (ffffc9000468d148: ffff88012491b240), converting
> to physical, and then back to identity-mapped virtual gives:
>
> crash-5.1.1> vtop 0xffffc9000468d148
> VIRTUAL PHYSICAL
> ffffc9000468d148 126eca148
> ...
>
> And:
> crash-5.1.1> ptov 0x126eca148
> VIRTUAL PHYSICAL
> ffff880126eca148 126eca148
>
> And so the hit at 0xffffc9000468d148 was already caught by the earlier
> hit in the identity-mapped range:
> ffff880126eca148: ffff88012491b240
>
> If you don't want to wait, you can find the vmalloc_start_addr from
> "help -m" and use it to restrict the search range:
>
> $ cat cmdfile2a
> search -k -e 0xffffc90000000000 0xffff88012491b240
>
> $ time crash-5.1.1 dump.201101161845 kernel_link <cmdfile2a
>
> ...
> ffff88012491b280: ffff88012491b240
> ffff880126dcca18: ffff88012491b240
> ffff880126dcca20: ffff88012491b240
> ffff880126dcca30: ffff88012491b240
> ffff880126dcca88: ffff88012491b240
> ffff880126eca148: ffff88012491b240
>
> real 0m4.243s
> user 0m4.088s
> sys 0m0.156s
>
> This command finishes with the first 6 hits in 4 seconds.
>
> Once you have those hits, if you have to know if any virtual mappings
> exist, you can use kmem on the physical address:
>
> crash-5.1.1> vtop 0xffff880126eca148
> VIRTUAL PHYSICAL
> ffff880126eca148 126eca148
> ...
>
> crash-5.1.1> kmem -v 126eca148
> VM_STRUCT ADDRESS RANGE SIZE
> ffff8801277f8180 ffffc90004682000 - 1052672
>
> Which shows the virtual range that contains the mapping for the page.
> Then this command takes no time:
>
> crash-5.1.1> search -k -s ffffc90004682000 -e ffffc90004783000
> ffff88012491b240
> ffffc9000468d148: ffff88012491b240
>
>
>
> I think the drastic reduction of search time from 2 minutes to 4
> seconds
> is interesting enough to warrant a shortcut.
>
> The attached patch implements the -K option that is the same as doing
> "-k -e <vmalloc_start_addr>".
>
> Comments?
> Bob Montgomery

I like the idea...

But it won't work on ia64 machines, where the vmalloc space is contained
within the region starting at a000000000000000, which is below the unity-mapped
region starting at e000000000000000.

I put in a debug statement to show the "start" and "end" values passed
to search():

crash> mach
MACHINE TYPE: ia64
MEMORY SIZE: 15.7 GB
CPUS: 2
PROCESSOR SPEED: 1594 Mhz
HZ: 1000
PAGE SIZE: 16384
KERNEL STACK SIZE: 32768
KERNEL CACHED REGION: e000000000000000
KERNEL UNCACHED REGION: c000000000000000
KERNEL VMALLOC REGION: a000000000000000
USER DATA/STACK REGION: 8000000000000000
USER DATA/STACK REGION: 6000000000000000
USER TEXT REGION: 4000000000000000
USER SHARED MEMORY REGION: 2000000000000000
USER IA32 EMULATION REGION: 0000000000000000
crash> search -k deadbeef
start: a000000100000000 end: ffffffffffffffff
... [ cut ] ...
crash> search -K deadbeef
start: a000000100000000 end: a000000200000000
... [ cut ] ...
crash>

So it looks like it might need a hack for ia64 anyway, which
already has a few. I don't think any of the other arches have
their vmalloc regions below their unity-mapped regions.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-25-2011, 10:01 PM
Bob Montgomery
 
Default RFC: Improving crash's search speed

On Tue, 2011-01-25 at 21:50 +0000, Dave Anderson wrote:

> >
> > The attached patch implements the -K option that is the same as doing
> > "-k -e <vmalloc_start_addr>".
> >
> > Comments?
> > Bob Montgomery
>
> I like the idea...
>
> But it won't work on ia64 machines, where the vmalloc space is contained
> within the region starting at a000000000000000, which is below the unity-mapped
> region starting at e000000000000000.
...
>
> So it looks like it might need a hack for ia64 anyway, which
> already has a few. I don't think any of the other arches have
> their vmalloc regions below their unity-mapped regions.
>
> Dave

So -K should really be defined as "search the identity-mapped region",
maybe? Perhaps a mach-specific utility routine to return start and end
for that region which become the defaults for -K search??

My own tools are really "search physical pages", but I didn't see that
mechanism in crash (though it might be there?).

It's been a while since ia64 was my main architecture :-)

Bob Montgomery

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-26-2011, 01:47 PM
Dave Anderson
 
Default RFC: Improving crash's search speed

----- Original Message -----
> On Tue, 2011-01-25 at 21:50 +0000, Dave Anderson wrote:
>
> > >
> > > The attached patch implements the -K option that is the same as doing
> > > "-k -e <vmalloc_start_addr>".
> > >
> > > Comments?
> > > Bob Montgomery
> >
> > I like the idea...
> >
> > But it won't work on ia64 machines, where the vmalloc space is contained
> > within the region starting at a000000000000000, which is below the unity-mapped
> > region starting at e000000000000000.
> ...
> >
> > So it looks like it might need a hack for ia64 anyway, which
> > already has a few. I don't think any of the other arches have
> > their vmalloc regions below their unity-mapped regions.
> >
> > Dave
>
> So -K should really be defined as "search the identity-mapped region",
> maybe? Perhaps a mach-specific utility routine to return start and end
> for that region which become the defaults for -K search??

Yeah, I think so. In fact, in looking at the ia64 anomoly, I note that
the similarly-mapped kernel text/data region in x86_64 is being skipped
entirely:

crash> mach
MACHINE TYPE: x86_64
MEMORY SIZE: 1 GB
CPUS: 8
PROCESSOR SPEED: 1995 Mhz
HZ: 1000
PAGE SIZE: 4096
KERNEL VIRTUAL BASE: ffff810000000000
KERNEL VMALLOC BASE: ffffc20000000000
KERNEL START MAP: ffffffff80000000
KERNEL MODULES BASE: ffffffff88000000
KERNEL STACK SIZE: 8192
crash>

The x86_64 search starts at the unity-mapped region at ffff810000000000,
then jumps to the vmalloc base region at ffffc20000000000, and then when
it reaches the end of that section, it calls next_vmalloc_vaddr(), which
skips to the modules part of of the vmalloc region at ffffffff88000000.
So it leaps over the kernel text/data mapped area at ffffffff80000000.
The kernel text/data mapped region, similar to the vmalloc region, is
"double-mapped", so you would see any search matches in that memory, but
they would be shown by their unity-mapped reference only. But that's not
the intent, so it's a bug.

> My own tools are really "search physical pages", but I didn't see that
> mechanism in crash (though it might be there?).

No it's not, but it probably should be. That would be easy to shoe-horn
in there since all of the search addresses (user or kernel) are first
turned into physical addresses before being read (except for the
Xen hypervisor). So yeah, I think a "-p" argument would be pretty
useful.

>
> It's been a while since ia64 was my main architecture :-)

Yeah, since RHEL6 dropped it, it's beginning to leak out of my brain cells,
but I remember running into the oddity of its mapped kernel and vmalloc
region location.

Anyway, I'm thinking now that "search" should be reworked to add these
new variants:

search -K unity-mapped kernel only
search -V vmalloc memory only
search -M kernel text/data mapped region (arch-dependent)
search -p physical memory

and leave the current mechanism as it is now:

search -u user-space of current context
search -k all of kernel virtual space

where "search -k" would be equivalent to "search -KVM".

Let me tinker around with this -- thanks for the suggestion
(and the inadvertant x86_64 bug report).

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-26-2011, 06:42 PM
Bob Montgomery
 
Default RFC: Improving crash's search speed

On Wed, 2011-01-26 at 14:47 +0000, Dave Anderson wrote:

>
> Anyway, I'm thinking now that "search" should be reworked to add these
> new variants:
>
> search -K unity-mapped kernel only
> search -V vmalloc memory only
> search -M kernel text/data mapped region (arch-dependent)
> search -p physical memory
>
> and leave the current mechanism as it is now:
>
> search -u user-space of current context
> search -k all of kernel virtual space
>
> where "search -k" would be equivalent to "search -KVM".
>
> Let me tinker around with this -- thanks for the suggestion
> (and the inadvertant x86_64 bug report).

Well, the real reason I started here was that I'm trying to implement my
string searcher, and when I tested the first prototype in crash, it took
forever compared to my physical page searcher.

(Basically a bunch of strncmp's starting on each byte, with extra
searches at the start of the page for "last half or more" and at the end
for "first half or more", since it's hard to track the virtual page
layout to search for strings across page boundaries.)

Today I'm trying to find out why parse_line() is messing up when given
more than one string in ""'s.



Bob Montgomery



--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-26-2011, 07:29 PM
Dave Anderson
 
Default RFC: Improving crash's search speed

----- Original Message -----
> On Wed, 2011-01-26 at 14:47 +0000, Dave Anderson wrote:
>
> >
> > Anyway, I'm thinking now that "search" should be reworked to add
> > these
> > new variants:
> >
> > search -K unity-mapped kernel only
> > search -V vmalloc memory only
> > search -M kernel text/data mapped region (arch-dependent)
> > search -p physical memory
> >
> > and leave the current mechanism as it is now:
> >
> > search -u user-space of current context
> > search -k all of kernel virtual space
> >
> > where "search -k" would be equivalent to "search -KVM".
> >
> > Let me tinker around with this -- thanks for the suggestion
> > (and the inadvertant x86_64 bug report).
>
> Well, the real reason I started here was that I'm trying to implement my
> string searcher, and when I tested the first prototype in crash, it took
> forever compared to my physical page searcher.
>
> (Basically a bunch of strncmp's starting on each byte, with extra
> searches at the start of the page for "last half or more" and at the end
> for "first half or more", since it's hard to track the virtual page
> layout to search for strings across page boundaries.)

Hmmm, string search -- that's pretty cool... I'll let you work on that.

I've pretty much got -p working OK, and next I'll plug in the -KVM restrictors.

> Today I'm trying to find out why parse_line() is messing up when given
> more than one string in ""'s.

Not sure I understand how you can do that, but don't let me interfere...

Thanks,
Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-26-2011, 09:08 PM
Bob Montgomery
 
Default RFC: Improving crash's search speed

On Wed, 2011-01-26 at 20:29 +0000, Dave Anderson wrote:

>
> I've pretty much got -p working OK, and next I'll plug in the -KVM restrictors.

By the way, thanks for running with this.


>
> > Today I'm trying to find out why parse_line() is messing up when given
> > more than one string in ""'s.
>
> Not sure I understand how you can do that, but don't let me interfere...

The syntax for the search command:
search [-s start | -k | -u] [-e end | -l length] [-m mask] value ...

allows multiple values for simultaneous search. For example:

crash-5.1.1str> search -k 00007ffffd401e28 00007f29a0cbeb9f
ffff88011e889f70: 7ffffd401e28
ffff88011e889fd8: 7f29a0cbeb9f

I thought I'd use a similar syntax for strings (-c for "chars", looking
for a better letter, maybe -S):

crash-5.1.1str> search -k -c "Memory value expected" "No such file or"

The current version of parse_line in tools.c has a problem with repeated
strings, so that it delivers the following in the value[] array:

(gdb) p value[0]
$1 = 0xb74a04 "Memory value expected"
(gdb) p value[1]
$2 = 0xb74a1b ""No"
(gdb) p value[2]
$3 = 0xb74a1f "such"
(gdb) p value[3]
$4 = 0xb74a24 "file"
(gdb) p value[4]
$5 = 0xb74a29 "or""

With the attached patch, it delivers:

(gdb) p value[0]
$1 = 0xb77a24 "Memory value expected"
(gdb) p value[1]
$2 = 0xb77a3c "No such file or"

I don't think it messes anything else up...

Bob Montgomery




--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 
Old 01-27-2011, 07:24 PM
Dave Anderson
 
Default RFC: Improving crash's search speed

----- Original Message -----
> On Wed, 2011-01-26 at 20:29 +0000, Dave Anderson wrote:
>
> >
> > I've pretty much got -p working OK, and next I'll plug in the -KVM restrictors.
>
> By the way, thanks for running with this.
>
>
> >
> > > Today I'm trying to find out why parse_line() is messing up when given
> > > more than one string in ""'s.
> >
> > Not sure I understand how you can do that, but don't let me interfere...
>
> The syntax for the search command:
> search [-s start | -k | -u] [-e end | -l length] [-m mask] value ...
>
> allows multiple values for simultaneous search. For example:
>
> crash-5.1.1str> search -k 00007ffffd401e28 00007f29a0cbeb9f
> ffff88011e889f70: 7ffffd401e28
> ffff88011e889fd8: 7f29a0cbeb9f
>
> I thought I'd use a similar syntax for strings (-c for "chars", looking
> for a better letter, maybe -S):
>
> crash-5.1.1str> search -k -c "Memory value expected" "No such file or"
>
> The current version of parse_line in tools.c has a problem with repeated
> strings, so that it delivers the following in the value[] array:
>
> (gdb) p value[0]
> $1 = 0xb74a04 "Memory value expected"
> (gdb) p value[1]
> $2 = 0xb74a1b ""No"
> (gdb) p value[2]
> $3 = 0xb74a1f "such"
> (gdb) p value[3]
> $4 = 0xb74a24 "file"
> (gdb) p value[4]
> $5 = 0xb74a29 "or""
>
> With the attached patch, it delivers:
>
> (gdb) p value[0]
> $1 = 0xb77a24 "Memory value expected"
> (gdb) p value[1]
> $2 = 0xb77a3c "No such file or"
>
> I don't think it messes anything else up...

I don't believe so either -- the patch makes good sense. Thanks
for tracking that down. Strange that it would only "break up" the
2nd, 4th, 6th, etc. string arguments. I didn't bother trying to
understand why that happens...)

Queued for the next release.

Thanks,
Dave


--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
 

Thread Tools




All times are GMT. The time now is 07:05 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org