FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Kernel Team

 
 
LinkBack Thread Tools
 
Old 05-25-2010, 07:58 PM
Chase Douglas
 
Default Tracing configuration review

On Tue, 2010-05-25 at 15:46 -0400, Steven Rostedt wrote:
> On Tue, 2010-05-25 at 15:31 -0400, Chase Douglas wrote:
> > Hi all,
> >
> > I'm going through our Ubuntu kernel configuration for our next release
> > to ensure we have all the trace options enabled that may be useful. I
> > have a few questions about what tracer options we should have enabled.
> >
> > Our guiding principle in regards to these options is: if an option can
> > be turned on and has no performance impact unless explicitly enabled on
> > the kernel command line or at runtime, we are happy to enable it.
> > Secondarily, we don't want to enable options that are headed for
> > deprecation.
> >
> > The following options are what I am looking to set for our x86
> > configurations. I've only included those that I am not 100% sure of.
> > Comments are what I could gather from documentation and Kconfig, but
> > they may not be accurate:
> >
> > # CONFIG_IRQSOFF_TRACER is not set (performance impact by default)
>
> Correct, keep that off.
>
> > # CONFIG_SYSPROF_TRACER is not set (don't know much about this)
>
> Neither do I ;-)
>
>
> > # CONFIG_SCHED_TRACER is not set (headed for deprecation?)
>
> Although it is headed for deprecation, I think it still gets set by
> other tracers, since it has the code to initiate the comm reader.
>
> > CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)
>
> Correct
>
> > CONFIG_BOOT_TRACER=y (no performance impact by default)
>
> But this tracer is pretty useless. It gives no more information than
> debug_initcalls.
>
> > CONFIG_KSYM_TRACER=y (no performance impact by default)
>
> Yep
>
> > # CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)
>
> I would set this if you already have the function tracer. It gives no
> more overhead than that, and it is very useful.
>
> > # CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)
>
> Don't know.
>
> > CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)
> >
> > Lastly, what options are safe for performance when
> > HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me
> > through what's in Documentation/trace/* and the Kconfig entries what
> > options could cause a performance decrease due to the inability to
> > dynamically enable and disable tracing at runtime.
>
> HAVE_DYNAMIC_FTRACE affects the function tracer. If you do not have
> that, then do not enable FUNCTION_TRACER or anything that depends on it.
>
> Also note, FUNCTION_TRACER depends on FRAME_POINTERS. Your millage may
> vary with that. If you already have FRAME_POINTERS on, and the arch
> supports DYNAMIC_FTRACE, then its fine to have FUNCTION_TRACER and all
> those that are built on top (STACK_TRACER, FUNCTION_GRAPH, etc)

(I would have snipped much of the above, but since I've added CC for the
Ubuntu list I wanted to leave it as is for this first reply)

After enabling KSYM_TRACER, I was presented with PROFILE_KSYM_TRACER.
This is also "Say N if unsure," so I could use some guidance on whether
we should turn it on as well.

Thanks,

-- Chase


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 05-25-2010, 08:20 PM
Frederic Weisbecker
 
Default Tracing configuration review

On Tue, May 25, 2010 at 03:58:18PM -0400, Chase Douglas wrote:
> On Tue, 2010-05-25 at 15:46 -0400, Steven Rostedt wrote:
> > On Tue, 2010-05-25 at 15:31 -0400, Chase Douglas wrote:
> > > Hi all,
> > >
> > > I'm going through our Ubuntu kernel configuration for our next release
> > > to ensure we have all the trace options enabled that may be useful. I
> > > have a few questions about what tracer options we should have enabled.
> > >
> > > Our guiding principle in regards to these options is: if an option can
> > > be turned on and has no performance impact unless explicitly enabled on
> > > the kernel command line or at runtime, we are happy to enable it.
> > > Secondarily, we don't want to enable options that are headed for
> > > deprecation.
> > >
> > > The following options are what I am looking to set for our x86
> > > configurations. I've only included those that I am not 100% sure of.
> > > Comments are what I could gather from documentation and Kconfig, but
> > > they may not be accurate:
> > >
> > > # CONFIG_IRQSOFF_TRACER is not set (performance impact by default)
> >
> > Correct, keep that off.
> >
> > > # CONFIG_SYSPROF_TRACER is not set (don't know much about this)
> >
> > Neither do I ;-)
> >
> >
> > > # CONFIG_SCHED_TRACER is not set (headed for deprecation?)
> >
> > Although it is headed for deprecation, I think it still gets set by
> > other tracers, since it has the code to initiate the comm reader.
> >
> > > CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)
> >
> > Correct
> >
> > > CONFIG_BOOT_TRACER=y (no performance impact by default)
> >
> > But this tracer is pretty useless. It gives no more information than
> > debug_initcalls.
> >
> > > CONFIG_KSYM_TRACER=y (no performance impact by default)
> >
> > Yep
> >
> > > # CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)
> >
> > I would set this if you already have the function tracer. It gives no
> > more overhead than that, and it is very useful.
> >
> > > # CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)
> >
> > Don't know.
> >
> > > CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)
> > >
> > > Lastly, what options are safe for performance when
> > > HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me
> > > through what's in Documentation/trace/* and the Kconfig entries what
> > > options could cause a performance decrease due to the inability to
> > > dynamically enable and disable tracing at runtime.
> >
> > HAVE_DYNAMIC_FTRACE affects the function tracer. If you do not have
> > that, then do not enable FUNCTION_TRACER or anything that depends on it.
> >
> > Also note, FUNCTION_TRACER depends on FRAME_POINTERS. Your millage may
> > vary with that. If you already have FRAME_POINTERS on, and the arch
> > supports DYNAMIC_FTRACE, then its fine to have FUNCTION_TRACER and all
> > those that are built on top (STACK_TRACER, FUNCTION_GRAPH, etc)
>
> (I would have snipped much of the above, but since I've added CC for the
> Ubuntu list I wanted to leave it as is for this first reply)
>
> After enabling KSYM_TRACER, I was presented with PROFILE_KSYM_TRACER.
> This is also "Say N if unsure," so I could use some guidance on whether
> we should turn it on as well.



KSYM_TRACER traces memory accesses (using breakpoints) and display each
of these in the trace/trace_pipe files. PROFILE_KSYM_TRACER produces
basic statistics on top of these traces.

But both are deprecated really, the perf tools interface is much
better for this job.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 05-25-2010, 09:09 PM
Chase Douglas
 
Default Tracing configuration review

On Tue, 2010-05-25 at 22:13 +0200, Frederic Weisbecker wrote:
> On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:
> > The following options are what I am looking to set for our x86
> > configurations. I've only included those that I am not 100% sure of.
> > Comments are what I could gather from documentation and Kconfig, but
> > they may not be accurate:

<snip>

> > # CONFIG_SCHED_TRACER is not set (headed for deprecation?)
>
>
> We want to deprecate it in the long term, but for now we
> don't have any replacement. Cool for RT latency tracing.

I thought that the functionality is the same as what you get by:

echo 1 > (debufs)/tracing/events/sched/enable

> > CONFIG_KSYM_TRACER=y (no performance impact by default)
>
>
> IMO, it is deprecated. The perf interface is much more powerful and flexible.
> Prasad, do you agree if I remove this ftrace plugin?

If there isn't any use in enabling it due to perf's features, then we
can turn it off. However, if there's any use to be gained by this over
perf's features, then I'd prefer to leave it on. Thoughts?


> > CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)
>
>
> In the way for deprecation.

Is this like the KMEM_TRACER where trace events have superseded it?

Thanks,

-- Chase


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 05-25-2010, 11:06 PM
Frederic Weisbecker
 
Default Tracing configuration review

On Tue, May 25, 2010 at 05:09:59PM -0400, Chase Douglas wrote:
> On Tue, 2010-05-25 at 22:13 +0200, Frederic Weisbecker wrote:
> > On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:
> > > The following options are what I am looking to set for our x86
> > > configurations. I've only included those that I am not 100% sure of.
> > > Comments are what I could gather from documentation and Kconfig, but
> > > they may not be accurate:
>
> <snip>
>
> > > # CONFIG_SCHED_TRACER is not set (headed for deprecation?)
> >
> >
> > We want to deprecate it in the long term, but for now we
> > don't have any replacement. Cool for RT latency tracing.
>
> I thought that the functionality is the same as what you get by:
>
> echo 1 > (debufs)/tracing/events/sched/enable



No, enabling every sched events will simply dump every events related
to the scheduler. It's then up to the user to make sense of these
traces through post-processing.

The wakeup tracer hooks the scheduler events for the specific
purpose of tracing the scheduler latencies: it measures the time
between a task is woken up and its actual scheduling to a cpu.
If you have the function tracer built, you'll also have a function
trace of everything that happened in-between.

So the wakeup tracer brings a kind of brain on top of the sched
events, but for very specific purposes.


> > > CONFIG_KSYM_TRACER=y (no performance impact by default)
> >
> >
> > IMO, it is deprecated. The perf interface is much more powerful and flexible.
> > Prasad, do you agree if I remove this ftrace plugin?
>
> If there isn't any use in enabling it due to perf's features, then we
> can turn it off. However, if there's any use to be gained by this over
> perf's features, then I'd prefer to leave it on. Thoughts?



No, perf does much more:

- stacktraces recording
- "top" alike view with perf top
- stat with perf stat, etc...
- userspace memory accesses


Here is a quick example:

$ cat test.c
int var;

void func_c(void)
{
var++;
}

void func_b(void)
{
func_c();
}


void func_a(void)
{
func_c();
}


int main(int argc, char **argv)
{
int i;

for (i = 0; i < 1000; i++)
if (i % 2)
func_a();
else
func_b();

return 0;
}
//end test.c

$ gcc test.c -fno-omit-frame-pointer -o test

$ readelf -s test | grep var
74: 000000000060102c 4 OBJECT GLOBAL DEFAULT 25 var

$ perf record -g -c 1 -e mem:0x000000000060102c:w ./test
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.069 MB perf.data (~3020 samples) ]

$ perf report

# Events: 1K cycles
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ......
#
99.90% test test [.] func_c
|
--- func_c
|
|--49.95%-- func_a
| |
| |--99.60%-- main
| | __libc_start_main
| --0.40%-- [...]
|
|--49.85%-- func_b
| main
| |
| |--99.60%-- __libc_start_main
| --0.40%-- [...]
--0.20%-- [...]


To sum up, there is nothing the ksym tracer does that perf can't.

Well, may be perf doesn't offer the time ordered view of memory
accesses, I must confess. Although this is still something we can
easily provide if people want it.



> > > CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)
> >
> >
> > In the way for deprecation.
>
> Is this like the KMEM_TRACER where trace events have superseded it?


It's a bit more complicated. This is a tracer that is able to produce
statistics on top of workqueue events. You'll get the number of events
queued and executed per workqueues. This gives some clues about their
load. Past patches brought the ground to get the average/max time of
execution, the works that took most time to complete, etc... But
they didn't make it because they were growing too much the size
and complexity of the code while a post processing in userspace would
be better suited for that.

So the current version only displays the very basic informations
of the number of works queued and executed.

This is something we could replace with a script in perf tools
that analyse the workqueue events, but I'm not even sure it's
worth now that the new cmwq workqueues may come upstream.
The workqueue tracing code was there to analyse the latencies
induced by works that block, which wouldn't be a problem anymore
with cmwq.

So, what I think I'm going to do is to remove this workqueue
statistics code from the kernel and if people complain for
the loss, I'll write this script for perf as a replacement.

So you can expect CONFIG_WORKQUEUE_TRACER will be removed
for 2.6.36 or so.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 05-27-2010, 11:20 AM
"K.Prasad"
 
Default Tracing configuration review

On Wed, May 26, 2010 at 01:06:57AM +0200, Frederic Weisbecker wrote:
> On Tue, May 25, 2010 at 05:09:59PM -0400, Chase Douglas wrote:
> > On Tue, 2010-05-25 at 22:13 +0200, Frederic Weisbecker wrote:
> > > On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:

<snipped>

> > > IMO, it is deprecated. The perf interface is much more powerful and flexible.
> > > Prasad, do you agree if I remove this ftrace plugin?

Sure, go ahead.

> >
> > If there isn't any use in enabling it due to perf's features, then we
> > can turn it off. However, if there's any use to be gained by this over
> > perf's features, then I'd prefer to leave it on. Thoughts?
>
>
>
> No, perf does much more:
>
> - stacktraces recording
> - "top" alike view with perf top
> - stat with perf stat, etc...
> - userspace memory accesses
>
>
> Here is a quick example:
>
> $ cat test.c
> int var;
>
> void func_c(void)
> {
> var++;
> }
>
> void func_b(void)
> {
> func_c();
> }
>
>
> void func_a(void)
> {
> func_c();
> }
>
>
> int main(int argc, char **argv)
> {
> int i;
>
> for (i = 0; i < 1000; i++)
> if (i % 2)
> func_a();
> else
> func_b();
>
> return 0;
> }
> //end test.c
>
> $ gcc test.c -fno-omit-frame-pointer -o test
>
> $ readelf -s test | grep var
> 74: 000000000060102c 4 OBJECT GLOBAL DEFAULT 25 var
>
> $ perf record -g -c 1 -e mem:0x000000000060102c:w ./test
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.069 MB perf.data (~3020 samples) ]
>
> $ perf report
>
> # Events: 1K cycles
> #
> # Overhead Command Shared Object Symbol
> # ........ ....... ................. ......
> #
> 99.90% test test [.] func_c
> |
> --- func_c
> |
> |--49.95%-- func_a
> | |
> | |--99.60%-- main
> | | __libc_start_main
> | --0.40%-- [...]
> |
> |--49.85%-- func_b
> | main
> | |
> | |--99.60%-- __libc_start_main
> | --0.40%-- [...]
> --0.20%-- [...]
>
>
> To sum up, there is nothing the ksym tracer does that perf can't.
>

I second Frederic's opinion on this.

Thanks,
K.Prasad

> Well, may be perf doesn't offer the time ordered view of memory
> accesses, I must confess. Although this is still something we can
> easily provide if people want it.
>

--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 05-27-2010, 10:15 PM
Frederic Weisbecker
 
Default Tracing configuration review

On Thu, May 27, 2010 at 04:50:44PM +0530, K.Prasad wrote:
> On Wed, May 26, 2010 at 01:06:57AM +0200, Frederic Weisbecker wrote:
> > On Tue, May 25, 2010 at 05:09:59PM -0400, Chase Douglas wrote:
> > > On Tue, 2010-05-25 at 22:13 +0200, Frederic Weisbecker wrote:
> > > > On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:
>
> <snipped>
>
> > > > IMO, it is deprecated. The perf interface is much more powerful and flexible.
> > > > Prasad, do you agree if I remove this ftrace plugin?
>
> Sure, go ahead.



Great, I'll do that soon.

Thanks.


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 06-08-2010, 10:00 PM
Chase Douglas
 
Default Tracing configuration review

On Tue, 2010-06-08 at 10:35 -0700, Randy Dunlap wrote:
> Hi Chase,
>
> Would you mind sharing your final tracing config options?

Sure.

Our source tree for Maverick encompasses all our major supported
platforms in one git branch. Thus, we have one common config file and a
bunch of arch and flavour (generic, server, virtual, pae) specific
config files. One note about the common configs: if a config option
would not exist because an option in a specific config file is set a
certain way, then you can ignore it for that specific arch/flavour. For
example, the common config has CONFIG_DYNAMIC_FTRACE=y, but the armel
config has "CONFIG_FUNCTION_TRACER is not set" so the former would not
have any affect on armel builds.

You can find the source branch at
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-maverick.git;a=summary

$ grep -Ir "TRAC[EI]" debian.master/config
./amd64/config.common.amd64:CONFIG_FUNCTION_TRACER=y
./amd64/config.common.amd64:CONFIG_STACK_TRACER=y
./i386/config.common.i386:CONFIG_FUNCTION_TRACER=y
./i386/config.common.i386:CONFIG_STACK_TRACER=y
./armel/config.common.armel:# CONFIG_FUNCTION_TRACER is not set
./armel/config.common.armel:# CONFIG_STACK_TRACER is not set
./config.common.ubuntu:# CONFIG_BACKTRACE_SELF_TEST is not set
./config.common.ubuntu:CONFIG_BLK_DEV_IO_TRACE=y
./config.common.ubuntu:# CONFIG_BOOT_TRACER is not set
./config.common.ubuntu:CONFIG_CAN_PM_TRACE=y
./config.common.ubuntu:CONFIG_CAPI_TRACE=y
./config.common.ubuntu:CONFIG_CONTEXT_SWITCH_TRACER= y
./config.common.ubuntu:CONFIG_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_EVENT_TRACING=y
./config.common.ubuntu:CONFIG_FTRACE=y
./config.common.ubuntu:CONFIG_FTRACE_MCOUNT_RECORD=y
./config.common.ubuntu:CONFIG_FTRACE_NMI_ENTER=y
./config.common.ubuntu:# CONFIG_FTRACE_STARTUP_TEST is not set
./config.common.ubuntu:CONFIG_FTRACE_SYSCALLS=y
./config.common.ubuntu:CONFIG_FUNCTION_GRAPH_TRACER= y
./config.common.ubuntu:CONFIG_GENERIC_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_ARCH_TRACEHOOK=y
./config.common.ubuntu:CONFIG_HAVE_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_MCOUNT_REC ORD=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_NMI_ENTER= y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_GRAPH_TR ACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACE_MC OUNT_TEST=y
./config.common.ubuntu:CONFIG_HAVE_MMIOTRACE_SUPPORT =y
./config.common.ubuntu:CONFIG_HAVE_SYSCALL_TRACEPOIN TS=y
./config.common.ubuntu:# CONFIG_IRQSOFF_TRACER is not set
./config.common.ubuntu:CONFIG_IWLWIFI_DEVICE_TRACING =y
./config.common.ubuntu:# CONFIG_KMEMTRACE is not set
./config.common.ubuntu:# CONFIG_KSYM_TRACER is not set
./config.common.ubuntu:CONFIG_MMIOTRACE=y
./config.common.ubuntu:# CONFIG_MMIOTRACE_TEST is not set
./config.common.ubuntu:CONFIG_NETFILTER_XT_TARGET_TR ACE=m
./config.common.ubuntu:CONFIG_NOP_TRACER=y
./config.common.ubuntu:CONFIG_PM_TRACE=y
./config.common.ubuntu:CONFIG_PM_TRACE_RTC=y
./config.common.ubuntu:# CONFIG_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_SCHED_TRACER=y
./config.common.ubuntu:# CONFIG_SCSI_IPR_TRACE is not set
./config.common.ubuntu:CONFIG_STACKTRACE=y
./config.common.ubuntu:CONFIG_STACKTRACE_SUPPORT=y
./config.common.ubuntu:# CONFIG_SYSPROF_TRACER is not set
./config.common.ubuntu:CONFIG_TRACEPOINTS=y
./config.common.ubuntu:CONFIG_TRACER_MAX_TRACE=y
./config.common.ubuntu:CONFIG_TRACE_IRQFLAGS_SUPPORT =y
./config.common.ubuntu:CONFIG_TRACING=y
./config.common.ubuntu:CONFIG_TRACING_SUPPORT=y
./config.common.ubuntu:# CONFIG_TREE_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_USER_STACKTRACE_SUPPOR T=y
./config.common.ubuntu:# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
./config.common.ubuntu:# CONFIG_WORKQUEUE_TRACER is not set

Some of the above is not FTRACE. I also snipped the matches in
config.common.ports because they are slightly different due to what the
ports architectures support, but they should be pretty similar.

Please let me know if anything stands out at you as incorrect.

Thanks,

-- Chase


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 
Old 06-14-2010, 02:41 AM
Chase Douglas
 
Default Tracing configuration review

On Fri, 2010-06-11 at 14:51 -0700, Randy Dunlap wrote:
> I'm a bit surprised by one (family) of kconfig symbols here:
>
> PM_TRACE=y, PM_TRACE_RTC=y
>
> That enables low-level kernel developer debugging code.
>
> Other than that, the rest make sense to me.

We try to focus a lot on the laptop use case, and so we do a bit of work
on suspend/resume bugs. For those unaware, these options allow one to
save the very last PM event point inside the RTC. If a suspend dies we
may be able to get some good information on where it died using this
tool. The functionality has to be manually enabled at runtime, otherwise
the RTC would get scribbled over during every resume. Since it doesn't
have any impact in the general use case, we leave it enabled by default.
We already build too many test kernels for other types of bugs .

Thanks,

-- Chase


--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
 

Thread Tools




All times are GMT. The time now is 10:50 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org