Hi Bryn,

On Thu, Jul 12, 2012 at 03:34:56PM +0100, Bryn M. Reeves wrote:
>
> It's possible that you have processes spending brief periods in
> D-state - short enough that they are hard to spot in top with the
> default sample period but long enough to count toward the load average
> (it's possible there's some other explanation though but that's what
> I'd try to rule out first).
>
> You could use tools like blktrace, iotop and latencytop to try to
> investigate further or if you're willing to install systemtap you
> could use the "sleepingBeauties.stp" script to look for processes
> spending >10ms in this state (and dump their stacks when they do).
>


With latencytop I see that the large latency processes are usually
caused by

1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
2. `Waiting for buffer IO to complete': jbd2/dm-*
3. `SCSI disk ioctl': hddtemp

The wait times are of the order of 60ms - 200ms.

`btrace /dev/sda' gives me the following:

CPU0 (8,0):
Reads Queued: 1,572, 22,536KiB Writes Queued: 5,211, 32,112KiB
Read Dispatches: 1,612, 22,896KiB Write Dispatches: 3,956, 39,424KiB
Reads Requeued: 2 Writes Requeued: 11
Reads Completed: 2,156, 27,844KiB Writes Completed: 7,028, 69,784KiB
Read Merges: 0, 0KiB Write Merges: 1,568, 6,280KiB
Read depth: 98 Write depth: 33
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 136, 0KiB PC Write Disp.: 0, 0KiB
PC Reads Req.: 22 PC Writes Req.: 0
PC Reads Compl.: 90 PC Writes Compl.: 0
IO unplugs: 1,224 Timer unplugs: 35
CPU1 (8,0):
Reads Queued: 107, 1,272KiB Writes Queued: 1,162, 6,768KiB
Read Dispatches: 95, 1,140KiB Write Dispatches: 244, 4,708KiB
Reads Requeued: 0 Writes Requeued: 2
Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB
Read Merges: 0, 0KiB Write Merges: 612, 2,448KiB
Read depth: 98 Write depth: 33
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 28, 0KiB PC Write Disp.: 0,
0KiB
PC Reads Req.: 0 PC Writes Req.: 0
PC Reads Compl.: 0 PC Writes Compl.: 0
IO unplugs: 264 Timer unplugs: 0
CPU2 (8,0):
Reads Queued: 249, 2,284KiB Writes Queued: 4,206, 26,020KiB
Read Dispatches: 238, 2,220KiB Write Dispatches: 1,463, 22,116KiB
Reads Requeued: 0 Writes Requeued: 1
Reads Completed: 0, 0KiB Writes Completed: 0,
0KiB
Read Merges: 0, 0KiB Write Merges: 2,214, 8,952KiB
Read depth: 98 Write depth: 33
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 21, 0KiB PC Write Disp.: 0,
0KiB
PC Reads Req.: 0 PC Writes Req.: 0
PC Reads Compl.: 0 PC Writes Compl.: 0
IO unplugs: 736 Timer unplugs: 9
CPU3 (8,0):
Reads Queued: 247, 1,752KiB Writes Queued: 448, 4,884KiB
Read Dispatches: 214, 1,588KiB Write Dispatches: 210, 3,536KiB
Reads Requeued: 1 Writes Requeued: 0
Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB
Read Merges: 18, 72KiB Write Merges: 184, 736KiB
Read depth: 98 Write depth: 33
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 23, 0KiB PC Write Disp.: 0, 0KiB
PC Reads Req.: 0 PC Writes Req.: 0
PC Reads Compl.: 0 PC Writes Compl.: 0
IO unplugs: 171 Timer unplugs: 1

Total (8,0):
Reads Queued: 2,175, 27,844KiB Writes Queued: 11,027, 69,784KiB
Read Dispatches: 2,159, 27,844KiB Write Dispatches: 5,873, 69,784KiB
Reads Requeued: 3 Writes Requeued: 14
Reads Completed: 2,156, 27,844KiB Writes Completed: 7,028, 69,784KiB
Read Merges: 18, 72KiB Write Merges: 4,578, 18,416KiB
PC Reads Queued: 0, 0KiB PC Writes Queued: 0, 0KiB
PC Read Disp.: 208, 0KiB PC Write Disp.: 0, 0KiB
PC Reads Req.: 22 PC Writes Req.: 0
PC Reads Compl.: 90 PC Writes Compl.: 0
IO unplugs: 2,395 Timer unplugs: 45

Throughput (R/W): 6KiB/s / 16KiB/s
Events (8,0): 147,334 entries
Skips: 0 forward (0 - 0.0%)

iotop doesn't show anything in particular other than once or twice one
of the desktop apps like Firefox showing up and going away almost
immediately.

I don't quiet know how to interpret any of the above output though.

PS: I will run the systemtap script later tonight.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 12.07.2012, Suvayu Ali wrote:

> 1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
> 2. `Waiting for buffer IO to complete': jbd2/dm-*
[....]

Fsync is expensive, but that's the way it is.

If you're using cfq as your scheduler, try this in rc.local:

echo "32" > /sys/block/sda/queue/iosched/quantum
echo "0" > /sys/block/sda/queue/iosched/slice_idle
echo "1" > /sys/block/sda/queue/iosched/low_latency
echo "51200" > /sys/block/sda/queue/nr_requests

Together with this in /etc/sysctl.conf:

vm.dirty_ratio = 10
vm.dirty_background_ratio = 5

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Hi Heinz,

Sorry for the late response.

On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl <htd@fritha.org> wrote:
> On 12.07.2012, Suvayu Ali wrote:
>
>> 1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
>> 2. `Waiting for buffer IO to complete': jbd2/dm-*
> [....]
>
> Fsync is expensive, but that's the way it is.
>
> If you're using cfq as your scheduler, try this in rc.local:
>
> echo "32" > /sys/block/sda/queue/iosched/quantum
> echo "0" > /sys/block/sda/queue/iosched/slice_idle
> echo "1" > /sys/block/sda/queue/iosched/low_latency
> echo "51200" > /sys/block/sda/queue/nr_requests
>
> Together with this in /etc/sysctl.conf:
>
> vm.dirty_ratio = 10
> vm.dirty_background_ratio = 5
>

I tend not to try things that I don't understand. Could please outline
briefly what the above suggestions do? I would like to understand before
I try them out.

Thanks,

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

2012/7/15 suvayu ali <fatkasuvayu+linux@gmail.com>:
> Hi Heinz,
>
> Sorry for the late response.
>
> On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl <htd@fritha.org> wrote:
>> On 12.07.2012, Suvayu Ali wrote:
(...)
>> If you're using cfq as your scheduler, try this in rc.local:
>>
>> echo "32" > /sys/block/sda/queue/iosched/quantum
>> echo "0" > /sys/block/sda/queue/iosched/slice_idle
>> echo "1" > /sys/block/sda/queue/iosched/low_latency
>> echo "51200" > /sys/block/sda/queue/nr_requests
>>
>> Together with this in /etc/sysctl.conf:
>>
>> vm.dirty_ratio = 10
>> vm.dirty_background_ratio = 5
>>
>
> I tend not to try things that I don't understand. Could please outline
> briefly what the above suggestions do? I would like to understand before
> I try them out.
>
> Thanks,
>

When vm.dirty_ratio percent of total system memory is taken up by
dirty pages (data waiting to be saved to disk), the process which is
generating disk writes will itself start writing out dirty data. It
means, that program will be made to stop using system buffers to hide
cost of writes, and to write the data to the disk.

When vm.dirty_background_ratio of total system memory is taken up by
dirty pages, the pdflush background writeback daemon will start
writing out dirty data.

If you keep these lower, the system will try to prevent accumulation
of large amounts of data to write. Therefore, when sync comes, you
will not have to wait for the accumulated data to be written to disk.

vm tunables are explained here:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Hi Paweł and Heinz,

On Sun, Jul 15, 2012 at 03:55:45PM +0200, Paweł Brodacki wrote:
> 2012/7/15 suvayu ali <fatkasuvayu+linux@gmail.com>:
> > Hi Heinz,
> >
> > Sorry for the late response.
> >
> > On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl <htd@fritha.org> wrote:
> >> On 12.07.2012, Suvayu Ali wrote:
> (...)
> >> If you're using cfq as your scheduler, try this in rc.local:
> >>
> >> echo "32" > /sys/block/sda/queue/iosched/quantum
> >> echo "0" > /sys/block/sda/queue/iosched/slice_idle
> >> echo "1" > /sys/block/sda/queue/iosched/low_latency
> >> echo "51200" > /sys/block/sda/queue/nr_requests
> >>
> >> Together with this in /etc/sysctl.conf:
> >>
> >> vm.dirty_ratio = 10
> >> vm.dirty_background_ratio = 5
> >>
> >
> > I tend not to try things that I don't understand. Could please outline
> > briefly what the above suggestions do? I would like to understand before
> > I try them out.
> >
> > Thanks,
> >
>
> When vm.dirty_ratio percent of total system memory is taken up by
> dirty pages (data waiting to be saved to disk), the process which is
> generating disk writes will itself start writing out dirty data. It
> means, that program will be made to stop using system buffers to hide
> cost of writes, and to write the data to the disk.
>
> When vm.dirty_background_ratio of total system memory is taken up by
> dirty pages, the pdflush background writeback daemon will start
> writing out dirty data.
>
> If you keep these lower, the system will try to prevent accumulation
> of large amounts of data to write. Therefore, when sync comes, you
> will not have to wait for the accumulated data to be written to disk.
>
> vm tunables are explained here:
> https://www.kernel.org/doc/Documentation/sysctl/vm.txt.

Thanks for the nice explanation. But sadly this doesn't help either.
Earlier the load average on login would be ~ 1-1.5. With these changes,
it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
example now my laptop has been running for 50 minutes and load is
hovering over 1.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 07/15/2012 01:27 PM, Suvayu Ali wrote:

Thanks for the nice explanation. But sadly this doesn't help either.
Earlier the load average on login would be ~ 1-1.5. With these changes,
it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
example now my laptop has been running for 50 minutes and load is
hovering over 1.


OK, you've eliminated disk IO as the cause of your high load.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On Sun, Jul 15, 2012 at 4:27 PM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> Thanks for the nice explanation. But sadly this doesn't help either.
> Earlier the load average on login would be ~ 1-1.5. With these changes,
> it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
> example now my laptop has been running for 50 minutes and load is
> hovering over 1.

A co-worker has a similar problem with a new Lenovo ThinkPad -- it
turns out something about the audio chip was taking a lot more cpu
cycles than it should be -- I'll try to dig out some more details from
him.

--
Jared Smith
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On Sun, Jul 15, 2012 at 07:00:23PM -0400, Jared K. Smith wrote:
> On Sun, Jul 15, 2012 at 4:27 PM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> > Thanks for the nice explanation. But sadly this doesn't help either.
> > Earlier the load average on login would be ~ 1-1.5. With these changes,
> > it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
> > example now my laptop has been running for 50 minutes and load is
> > hovering over 1.
>
> A co-worker has a similar problem with a new Lenovo ThinkPad -- it
> turns out something about the audio chip was taking a lot more cpu
> cycles than it should be -- I'll try to dig out some more details from
> him.
>

Thanks a lot Jared.

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

On 07/16/2012 12:00 AM, Jared K. Smith wrote:
> A co-worker has a similar problem with a new Lenovo ThinkPad -- it
> turns out something about the audio chip was taking a lot more cpu
> cycles than it should be -- I'll try to dig out some more details from
> him.

Fascinating. If so, how come those cycles weren't being accounted for?
Hmmmm....

Andrew.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Hello everyone,

On Thu, Jul 12, 2012 at 01:00:34PM +0200, Suvayu Ali wrote:
> Hi,
>
> I have a ThinkPad (Core i7, 2.67GHz) with F17 and I'm seeing high system
> loads persistently. Right from boot up I see loads of 1, and often it
> goes up to 2 or more.
>
> Things I have looked at:
> 1. I'm using Xfce, so the desktop environment is also light.
> 2. I looked at the output of `systemctl list-units' and don't see any
> unnecessary services enabled.
> 3. top only says currently Xorg is the most active process with 1% CPU
> activity. From the activity summary of atop since boot, I see it's
> Xorg with 3% CPU usage. And while idling all my 4 logical cores are
> throttled down and are at about 1%.
>
> So my question is since CPU usage is already so low, what could be
> driving up the system load? This is bugging me since I also see a
> general sluggishness compared to a much faster system early this year.
>

To answer myself; after I upgraded to kernel 3.4.6-2.fc17.x86_64
yesterday, I haven't seen any issues with load. My average load when
browsing, reading email, or other regular desktop activities is ~ 0.22.

Cheers,

--
Suvayu

Open source is the future. It sets us free.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org