FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian ISP

 
 
LinkBack Thread Tools
 
Old 08-26-2008, 02:45 PM
Marek Podmaka
 
Default sys cpu usage

Hello all,

What is normal sys CPU usage in %? I have DualCore Xeon with
hyperthreading, so 4 logical CPUs as reported by kernel (2.6.24.3),
but the "sys" usage is about 25% (or 97% on scale when max is 400%).
It grows proportionally with normal (user and nice) cpu usage, but
seems too high for me. Server is running etch with php+mysql.

Or is it normal? How can I see what causes it?

Here is graph from munin: http://92.240.244.190/cpu-week.png


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 03:18 PM
Michael Loftis
 
Default sys cpu usage

It really depends on what you're doing and how. Specifics about your
configuration can make a world of difference. On our web servers, where
their primary filesystem is on NFS, the NICs do UDP/TCP checksum
offloading, and they mostly push large packets I see up to about 15% or so
on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so four CPUs).
But that's peak, average is more like 7-8% at around 70 hits/second, and
we've a pretty high overhead too. So 25%+ sounds off.


As far as finding out where it's spending time, well, there aren't many
options there with Linux. Solaris has dtrace, and with that you can do
profiling. OS/X has Shark and a couple others (and they're dtrace under the
hood). But I've never found a good way of profiling the Linux kernel.


--On August 26, 2008 4:45:38 PM +0200 Marek Podmaka
<marki@marki-online.net> wrote:



Hello all,

What is normal sys CPU usage in %? I have DualCore Xeon with
hyperthreading, so 4 logical CPUs as reported by kernel (2.6.24.3),
but the "sys" usage is about 25% (or 97% on scale when max is 400%).
It grows proportionally with normal (user and nice) cpu usage, but
seems too high for me. Server is running etch with php+mysql.

Or is it normal? How can I see what causes it?

Here is graph from munin: http://92.240.244.190/cpu-week.png


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact
listmaster@lists.debian.org







--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 03:48 PM
blorb
 
Default sys cpu usage

well, if it's within the mysql, try mytop but if it's something within the apahce or php i'm not sure myself...
but try logging while working


tail -f /var/log/whatever.log (change whatever to stuff you think making the troubles)

Shahar Fermon.


On Tue, Aug 26, 2008 at 6:18 PM, Michael Loftis <mloftis@modwest.com> wrote:

It really depends on what you're doing and how. *Specifics about your configuration can make a world of difference. *On our web servers, where their primary filesystem is on NFS, the NICs do UDP/TCP checksum offloading, and they mostly push large packets I see up to about 15% or so on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so four CPUs). But that's peak, average is more like 7-8% at around 70 hits/second, and we've a pretty high overhead too. *So 25%+ sounds off.




As far as finding out where it's spending time, well, there aren't many options there with Linux. *Solaris has dtrace, and with that you can do profiling. OS/X has Shark and a couple others (and they're dtrace under the hood). *But I've never found a good way of profiling the Linux kernel.




--On August 26, 2008 4:45:38 PM +0200 Marek Podmaka <marki@marki-online.net> wrote:




Hello all,



What is normal sys CPU usage in %? I have DualCore Xeon with

hyperthreading, so 4 logical CPUs as reported by kernel (2.6.24.3),

but the "sys" usage is about 25% (or 97% on scale when max is 400%).

It grows proportionally with normal (user and nice) cpu usage, but

seems too high for me. Server is running etch with php+mysql.



Or is it normal? How can I see what causes it?



Here is graph from munin: http://92.240.244.190/cpu-week.png





--

*bYE, Marki





--

To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact

listmaster@lists.debian.org














--

Michael Loftis

Modwest Operations Manager

Powerful, Affordable Web Hosting





--

To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org





--
"I thought what I'd do was, I'd pretend I was one of those deaf-mutes".
 
Old 08-26-2008, 03:51 PM
Michael Loftis
 
Default sys cpu usage

--On August 26, 2008 6:48:35 PM +0300 blorb <shaharf@gmail.com> wrote:



well, if it's within the mysql, try mytop but if it's something within
the apahce or php i'm not sure myself...
but try logging while working

tail -f /var/log/whatever.log (change whatever to stuff you think making
the troubles)


%sys is kernel CPU usage though. mytop/etc can identify within that
specific userspace app. One can use strace and profiling utilities to
profile userspace code runtime and the interface into the kernel, which can
give you a better idea, but I don't know of any way to get function level
profiling (time spent information) inside the kernel, just the strace
information which shows you as you transition to/from kernel space for eg
open, read, write, close, select, etc.




Shahar Fermon.



On Tue, Aug 26, 2008 at 6:18 PM, Michael Loftis <mloftis@modwest.com>
wrote:

It really depends on what you're doing and how. Specifics about your
configuration can make a world of difference. On our web servers, where
their primary filesystem is on NFS, the NICs do UDP/TCP checksum
offloading, and they mostly push large packets I see up to about 15% or
so on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so four
CPUs). But that's peak, average is more like 7-8% at around 70
hits/second, and we've a pretty high overhead too. So 25%+ sounds off.

As far as finding out where it's spending time, well, there aren't many
options there with Linux. Solaris has dtrace, and with that you can do
profiling. OS/X has Shark and a couple others (and they're dtrace under
the hood). But I've never found a good way of profiling the Linux kernel.

--On August 26, 2008 4:45:38 PM +0200 Marek Podmaka
<marki@marki-online.net> wrote:


Hello all,

What is normal sys CPU usage in %? I have DualCore Xeon with
hyperthreading, so 4 logical CPUs as reported by kernel (2.6.24.3),
but the "sys" usage is about 25% (or 97% on scale when max is 400%).
It grows proportionally with normal (user and nice) cpu usage, but
seems too high for me. Server is running etch with php+mysql.

Or is it normal? How can I see what causes it?

Here is graph from munin: http://92.240.244.190/cpu-week.png


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact
listmaster@lists.debian.org







--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact
listmaster@lists.debian.org





--
"I thought what I'd do was, I'd pretend I was one of those deaf-mutes".





--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 04:51 PM
Marcin Sochacki
 
Default sys cpu usage

oprofile is a Linux tool for kernel profiling.

Regarding the original issue with high load, it's worth noting that USER
CPU usage is also very high and corresponding to the SYS usage. I think
that simply the web application you're using is not optimised enough, or you
just have a lot of traffic.

Marcin

--
+---------------------------------------+
| -o) http://wanted.eu.org/
| / Message void if penguin violated
+ _\_V Don't mess with the penguin


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 06:14 PM
Marek Podmaka
 
Default sys cpu usage

Hello,

Tuesday, August 26, 2008, 17:18:01, Michael Loftis wrote:
> their primary filesystem is on NFS, the NICs do UDP/TCP checksum
> offloading, and they mostly push large packets I see up to about 15%
> or so on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so
> four CPUs). But that's peak, average is more like 7-8% at around 70
> hits/second, and we've a pretty high overhead too. So 25%+ sounds

How is TCP checksum offload affecting the SYS usage? It is the chip on
network card which is doing the stuff and not kernel, right? Or does
it show in the sys cpu usage? (On this server, only checksumming
offload is supported.)


Tuesday, August 26, 2008, 17:48:35, blorb wrote:
> well, if it's within the mysql, try mytop but if it's something
> within the apahce or php i'm not sure myself...
> but try logging while working

Tuesday, August 26, 2008, 17:52:00, Joel Merrick wrote:
> Take a look at apache-top
> You need the server-status page enabled and it has to be apache 2.
> Also the mysql slow query log can be useful

Thanks for the tips, but I'm already using all this (mytop, slow log
and apache's status page) for performance tuning, but that wouldn't
help in case I need to track the system cpu usage.


Tuesday, August 26, 2008, 18:51:43, Marcin Sochacki wrote:
> oprofile is a Linux tool for kernel profiling.
> Regarding the original issue with high load, it's worth noting that USER
> CPU usage is also very high and corresponding to the SYS usage. I think
> that simply the web application you're using is not optimised enough, or you
> just have a lot of traffic.

Oh yes, I was playing with oprofile some time ago, so I will try it.

The total CPU load is high now and mysql will be migrated to new
server next month. I just published the graph so that someone will be
able to tell if the SYS usage is too high compared to USER or not. And
yes, there's lot of traffic - about 40 hits/s for apache/php and another
100-120 hits/s for images using lighttpd (but 95% of them are served
from mem-cache.


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 06:54 PM
Michael Loftis
 
Default sys cpu usage

--On August 26, 2008 8:14:50 PM +0200 Marek Podmaka
<marki@marki-online.net> wrote:



Hello,

Tuesday, August 26, 2008, 17:18:01, Michael Loftis wrote:

their primary filesystem is on NFS, the NICs do UDP/TCP checksum
offloading, and they mostly push large packets I see up to about 15%
or so on dual core 2Ghz processors (Xeon 5130s not hyperthreaded, so
four CPUs). But that's peak, average is more like 7-8% at around 70
hits/second, and we've a pretty high overhead too. So 25%+ sounds


How is TCP checksum offload affecting the SYS usage? It is the chip on
network card which is doing the stuff and not kernel, right? Or does
it show in the sys cpu usage? (On this server, only checksumming
offload is supported.)


Depends on what is supported on your card, use ethtool -k <dev> to find out
and to change settings. On our systems we're doing checksum and TSO. The
offload doesn't show at all in the CPU usage stats. The CPU basically
skips computing checksums, and for TSO, segmentation, and just sends big
packets to the DMA ring buffers, the card then picks it up, does any
segmentation (replicating/affixing IP headers), computes checksums, and
pushes it out over the wire. This frees up a decent amount of CPU at high
packet rates or high line rates. The result is lower %sys CPU, thus more
CPU available for userspace (and system) operations. UDP checksum
offloading is actually pretty important for us since we're using NFS as
primary storage.


What kernel are you running BTW? And debian version? Kernels prior to 2.6
didn't do well on high interrupt loads P4, Xeon, and Core2's because of
large stack frames. Somewhere in 2.6 they made a change that allows the
kernel to export an optimal path for entering/leaving the kernel routines
without necessarily saving the entire stack frame (sysenter/sysexit calling
convention), I can't recall if that shows up as %sys, or just high overall
CPU usage. Try using vmstat too, it'll tell you your number of context
switches.


Someone else mentioned oprofile to profile kernel CPU usage too.




Tuesday, August 26, 2008, 17:48:35, blorb wrote:

well, if it's within the mysql, try mytop but if it's something
within the apahce or php i'm not sure myself...
but try logging while working


Tuesday, August 26, 2008, 17:52:00, Joel Merrick wrote:

Take a look at apache-top
You need the server-status page enabled and it has to be apache 2.
Also the mysql slow query log can be useful


Thanks for the tips, but I'm already using all this (mytop, slow log
and apache's status page) for performance tuning, but that wouldn't
help in case I need to track the system cpu usage.


Tuesday, August 26, 2008, 18:51:43, Marcin Sochacki wrote:

oprofile is a Linux tool for kernel profiling.
Regarding the original issue with high load, it's worth noting that USER
CPU usage is also very high and corresponding to the SYS usage. I think
that simply the web application you're using is not optimised enough, or
you just have a lot of traffic.


Oh yes, I was playing with oprofile some time ago, so I will try it.

The total CPU load is high now and mysql will be migrated to new
server next month. I just published the graph so that someone will be
able to tell if the SYS usage is too high compared to USER or not. And
yes, there's lot of traffic - about 40 hits/s for apache/php and another
100-120 hits/s for images using lighttpd (but 95% of them are served
from mem-cache.


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact
listmaster@lists.debian.org







--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-26-2008, 10:00 PM
Marek Podmaka
 
Default sys cpu usage

Hello,

Tuesday, August 26, 2008, 20:54:22, Michael Loftis wrote:

> What kernel are you running BTW? And debian version? Kernels prior to 2.6
> didn't do well on high interrupt loads P4, Xeon, and Core2's because of

I'm running 2.6.24.3 on debian etch.

> Someone else mentioned oprofile to profile kernel CPU usage too.

I tried oprofile, but the results are not very clear for me. If I
understand it well, everything which is vmlinux is counted as sys cpu,
right?

bill:~:# opreport | head
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.65 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
GLOBAL_POWER_E...|
samples| %|
------------------
142167514 31.3228 vmlinux
97946264 21.5798 mysqld
73468449 16.1868 libc-2.3.6.so
68429064 15.0765 libphp5.so
20384681 4.4912 perl


bill:~:# opreport -l | head -30
warning: /oprofile could not be found.
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.65 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
warning: /etc/ld.so.cache is not in a usable binary format.
warning: /usr/lib/locale/locale-archive is not in a usable binary format.
warning: /var/log/rrd/eth0.rrd is not in a usable binary format.
samples % image name app name symbol name
95243636 21.5689 mysqld mysqld (no symbols)
71545755 16.2023 libc-2.3.6.so libc-2.3.6.so (no symbols)
66640228 15.0914 libphp5.so libphp5.so (no symbols)
20330122 4.6040 perl perl (no symbols)
11724900 2.6552 vmlinux vmlinux port_inb
10713862 2.4263 vmlinux vmlinux __copy_user_intel
10572065 2.3942 vmlinux vmlinux __copy_to_user_ll
8592559 1.9459 vmlinux vmlinux system_call
6383308 1.4456 apache-ssl apache-ssl (no symbols)
5983763 1.3551 vmlinux vmlinux kunmap_atomic
5311477 1.2028 vmlinux vmlinux __d_lookup
5127355 1.1611 libz.so.1.2.3 libz.so.1.2.3 (no symbols)
4969862 1.1255 ZendOptimizer.so ZendOptimizer.so (no symbols)
4453856 1.0086 libpthread-2.3.6.so libpthread-2.3.6.so __read_nocancel
4441345 1.0058 oprofiled oprofiled (no symbols)


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 08:16 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org