Hi there,

I worked on a few patches to the help the userspace perf tools do
better at loading debug symbols -- this is particularly of interest on
Ubuntu and Linaro where separated debug symbols in .ddebs are
typically not loaded even when present. Note that even with the
patches, the contents of .ddebs are still not cached by perf along
with the other profiled binaries, so you will still lose the symbols
if you uninstall/upgrade your ddebs in the meantime. This should get
fixed in the future.

Due to mailserver issues, Arnaldo kindly helped merge these patches
for me. The patches are now merged to the linux-2.6-tip master
branch.


You can find details of the commits here:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/608775


[ Nico: WIll says he's going to pull these patches into his linaro
branch, if it's more convenient for you to pull from there. ]

Covering notes explaining the changes follow.

Cheers
---Dave

On Mon, 2010-08-02 at 10:55 +0100, Dave Martin wrote:
> Hi there,
>
> I worked on a few patches to the help the userspace perf tools do
> better at loading debug symbols -- this is particularly of interest on
> Ubuntu and Linaro where separated debug symbols in .ddebs are
> typically not loaded even when present. Note that even with the
> patches, the contents of .ddebs are still not cached by perf along
> with the other profiled binaries, so you will still lose the symbols
> if you uninstall/upgrade your ddebs in the meantime. This should get
> fixed in the future.
>
> Due to mailserver issues, Arnaldo kindly helped merge these patches
> for me. The patches are now merged to the linux-2.6-tip master
> branch.
>
>
> You can find details of the commits here:
>
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/608775

Thanks for passing this along! I am interested in figuring out whether
we should backport the patches to Ubuntu Maverick.

I took a look at the first two patches that deal with the cwd issue.
Although it fixes a real bug, I'm not sure it's worthwhile to pull it
into the Ubuntu Maverick kernel because it's rather esoteric. For those
unaware, the bug is triggered by starting a perf record while inside a
directory along the path to the executable being recorded. For example:

/bin # perf record ls
/bin # pert report

The report will work but it won't find the debug symbols
in /usr/lib/debug/bin/ls. Moving to a different directory, like /tmp,
while recording is a work around.

Since the bug is likely never to be seen except when running perf inside
system directories (which doesn't seem like a good usage scenario), I'm
inclined to recommend not backporting this for Ubuntu.

However, I still don't fully understand the issue surrounding the
buildid cache bug fix. Can you provide more details on the issue this is
fixing? It may be something we want to consider for Maverick, but I need
more info to know what's going on.

Thanks,

-- Chase


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team

Hi there,

On Mon, Aug 2, 2010 at 10:10 PM, Chase Douglas
<chase.douglas@canonical.com> wrote:
> On Mon, 2010-08-02 at 10:55 +0100, Dave Martin wrote:

[...]

>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/608775
>
> Thanks for passing this along! I am interested in figuring out whether
> we should backport the patches to Ubuntu Maverick.

Certainly would be nice to have, if there are developers who'd like to
use the tools.
The perf tools look like a good asset to have, but unpatched, they are
not very usable on Ubuntu yet (see below).

The patches only affect the userspace perf tools, so there's no risk
of destabilising the kernel etc. by applying the patches.

>
> I took a look at the first two patches that deal with the cwd issue.
> Although it fixes a real bug, I'm not sure it's worthwhile to pull it
> into the Ubuntu Maverick kernel because it's rather esoteric. For those

[...]

> Since the bug is likely never to be seen except when running perf inside
> system directories (which doesn't seem like a good usage scenario), I'm
> inclined to recommend not backporting this for Ubuntu.

Indeed--- I hit this a lot when trying to debug the perf tools but
for most people it will be much more uncommon. When profiling an app
from inside its own build tree, the debug will usually not have been
separated out already so it should work fine.

So it seems reasonable to allow this change to propagate in its own
time via mainline.

> However, I still don't fully understand the issue surrounding the
> buildid cache bug fix. Can you provide more details on the issue this is
> fixing? It may be something we want to consider for Maverick, but I need
> more info to know what's going on.

perf cleverly caches binaries during perf record, to ensure that later
analysis definitely uses the right symbols etc., even if binaries were
rebuilt / upgraded / removed in the meantime.

Unfortunately, it doesn't cleverly cache separate debug images yet
(i.e., ddebs contents). So when searching for symbols during perf
report and friends, it hits in the cache and uses the stripped images
which were observed by perf record. The result is that perf report
will only see symbols from .dynsym even if the ddebs are available ---
usually inadequate/misleading for profiling purposes.

The patch provides a workaround which looks in locations like
/usr/lib/debug anyway, if no matching image is found in the cache with
a non-empty .symtab. This should make the tools signifcantly more
usable on Ubuntu. Without the patch, the workaround is to pass
--no-buildid-cache to perf record, and to make sure there's no stale
cache content when running the other tools (to keep deleting ~/.debug
is the easiest way to ensure this -- perf buildid-cache is currently
buggy)



Unfortunately, I hit another bug which causes code symbol locations to
get skewed in non-PIE executable images --- see follow-ups (starting
[PATCH 0/2] perf: symbol offset breakage with separated debug, to lkml
and kernel-team@ubuntu). I believe there is no workaround for this
other than applying the patch... or building everything as PIE ;P

Cheers
---Dave

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team