Bug#686847: KVM guest slows down and finally hangs on I/O
 

Package: linux-2.6
Version: 2.6.32-45
Severity: normal

Linux vmXXXX.cs16.seeweb.it 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux

This is a fully up to date squeeze system running as a guest on a RHEL
6.2 host. 1 CPU, 1 GB of RAM, virtio net and disk (using XFS).
We have hundreds of similar guests but only this one shows this problem.

The system suddenly slows down and then all processes hang. The getty on
the virtual console is responsive to input, but it hangs when it tries
to fork+exec login.

The workload is unmemorable, apparently mostly light apache/PHP.

The kernel logs this same stack trace for all (?) processes:

INFO: task apache2:12623 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
apache2 D 0000000000000000 0 12623 12619 0x00000000
ffffffff814611f0 0000000000000086 0000000000000292 ffffffff8118ac27
ffff88002faf8000 000000108102cdc4 000000000000f9e0 ffff880003113fd8
0000000000015780 0000000000015780 ffff88002faf8000 ffff88002faf82f8
Call Trace:
[<ffffffff8118ac27>] ? cfq_set_request+0x2af/0x319
[<ffffffff8102cdc4>] ? pvclock_clocksource_read+0x3a/0x8b
[<ffffffff812fb3fa>] ? io_schedule+0x73/0xb7
[<ffffffff8117f6e9>] ? get_request_wait+0xf0/0x188
[<ffffffff81064f92>] ? autoremove_wake_function+0x0/0x2e
[<ffffffff8117fa78>] ? __make_request+0x2f7/0x428
[<ffffffff81191f63>] ? radix_tree_tag_clear+0x93/0xf1
[<ffffffff8117e247>] ? generic_make_request+0x299/0x2f9
[<ffffffff81192109>] ? radix_tree_tag_set+0x65/0xc6
[<ffffffff8117e37d>] ? submit_bio+0xd6/0xf2
[<ffffffff810ba9a9>] ? test_set_page_writeback+0xe0/0xef
[<ffffffff810d8be8>] ? swap_writepage+0x9b/0xa5
[<ffffffff810be529>] ? shrink_page_list+0x375/0x623
[<ffffffff810bea8b>] ? shrink_active_list+0x2b4/0x2d9
[<ffffffff810bef0c>] ? shrink_list+0x45c/0x767
[<ffffffff812fb320>] ? thread_return+0x79/0xe0
[<ffffffff810bf497>] ? shrink_zone+0x280/0x342
[<ffffffff812fb4ab>] ? _cond_resched+0x24/0x2f
[<ffffffff810bf69a>] ? shrink_slab+0x141/0x153
[<ffffffff810c055e>] ? try_to_free_pages+0x232/0x38e
[<ffffffff810bd553>] ? isolate_pages_global+0x0/0x20f
[<ffffffff810ba5ec>] ? __alloc_pages_nodemask+0x3d4/0x5fc
[<ffffffff810d8e00>] ? read_swap_cache_async+0x5d/0xf3
[<ffffffff810d8eed>] ? swapin_readahead+0x57/0x98
[<ffffffff810ccdfd>] ? handle_mm_fault+0x47f/0x80f
[<ffffffff812feb96>] ? do_page_fault+0x2e0/0x2fc
[<ffffffff812fca35>] ? page_fault+0x25/0x30

--
ciao,
Marco

Bug#686847: KVM guest slows down and finally hangs on I/O
 

On Sep 06, Marco d'Itri <md@linux.it> wrote:

> The system suddenly slows down and then all processes hang. The getty on
> the virtual console is responsive to input, but it hangs when it tries
> to fork+exec login.
After installing the backported wheezy kernel I have found out that this
is related to VM pressure: now the system is trashing hard but is
responsive and does not freezes, and the OOM killer started killing
some processes.

--
ciao,
Marco