FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Kernel

 
 
LinkBack Thread Tools
 
Old 05-02-2011, 02:50 PM
Sebastian Hofmann
 
Default Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

Package: xen-linux-system-2.6.32-5-xen-amd64
Version: 2.6.32-31
Severity: critical
Justification: breaks the whole system

Hi,

I have 64 bit xen kernel from squeeze installed on a dual xeon maschine. Usually everything runs fine until it comes to heavy load on a domU with high I/O and memory consumption.
This causes the dom0 to run out of memory and to kill several processes (see log below). As a consequence of this, the whole system becomes unusable.

I tried several things like assign dedicated memory to dom0, disable balloning, increase scheduler domain weights and assigned dedicated CPUs to dom0 as described in
http://wiki.xensource.com/xenwiki/XenBestPractices but had no success.

I think a domU should never break the whole system, so this might be a bug. Please let me know if you need further information.

Thanks
Sebastian


May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0
May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1
May 2 16:05:26 hercules kernel: [ 1768.319889] Call Trace:
May 2 16:05:26 hercules kernel: [ 1768.319899] [<ffffffff810b7104>] ? oom_kill_process+0x7f/0x23f
May 2 16:05:26 hercules kernel: [ 1768.319903] [<ffffffff810b7628>] ? __out_of_memory+0x12a/0x141
May 2 16:05:26 hercules kernel: [ 1768.319906] [<ffffffff810b777f>] ? out_of_memory+0x140/0x172
May 2 16:05:26 hercules kernel: [ 1768.319912] [<ffffffff81042ace>] ? pick_next_task_fair+0xca/0xd6
May 2 16:05:26 hercules kernel: [ 1768.319916] [<ffffffff810bb520>] ? __alloc_pages_nodemask+0x4e5/0x5f5
May 2 16:05:26 hercules kernel: [ 1768.319922] [<ffffffff810d9c04>] ? read_swap_cache_async+0x5d/0xf3
May 2 16:05:27 hercules kernel: [ 1768.319926] [<ffffffff810d9d23>] ? swapin_readahead+0x89/0x98
May 2 16:05:27 hercules kernel: [ 1768.319931] [<ffffffff8100c2f1>] ? __raw_callee_save_xen_pte_val+0x11/0x1e
May 2 16:05:27 hercules kernel: [ 1768.319935] [<ffffffff810cdc18>] ? handle_mm_fault+0x47f/0x80f
May 2 16:05:27 hercules kernel: [ 1768.319940] [<ffffffff8106d41f>] ? ktime_get_ts+0x68/0xb2
May 2 16:05:27 hercules kernel: [ 1768.319945] [<ffffffff8130f616>] ? do_page_fault+0x2e0/0x2fc
May 2 16:05:27 hercules kernel: [ 1768.319951] [<ffffffff8130d4b5>] ? page_fault+0x25/0x30
May 2 16:05:27 hercules kernel: [ 1768.319953] Mem-Info:
May 2 16:05:27 hercules kernel: [ 1768.319955] Node 0 DMA per-cpu:
May 2 16:05:27 hercules kernel: [ 1768.319958] CPU 0: hi: 0, btch: 1 usd: 0
May 2 16:05:27 hercules kernel: [ 1768.319960] CPU 1: hi: 0, btch: 1 usd: 0
May 2 16:05:27 hercules kernel: [ 1768.319961] Node 0 DMA32 per-cpu:
May 2 16:05:27 hercules kernel: [ 1768.319964] CPU 0: hi: 186, btch: 31 usd: 18
May 2 16:05:27 hercules kernel: [ 1768.319966] CPU 1: hi: 186, btch: 31 usd: 0
May 2 16:05:28 hercules kernel: [ 1768.319971] active_anon:1 inactive_anon:25 isolated_anon:0
May 2 16:05:28 hercules kernel: [ 1768.319972] active_file:15047 inactive_file:196012 isolated_file:128
May 2 16:05:28 hercules kernel: [ 1768.319973] unevictable:5 dirty:196071 writeback:143 unstable:0
May 2 16:05:28 hercules kernel: [ 1768.319974] free:2030 slab_reclaimable:14633 slab_unreclaimable:4593
May 2 16:05:28 hercules kernel: [ 1768.319975] mapped:3592 shmem:0 pagetables:957 bounce:0
May 2 16:05:28 hercules kernel: [ 1768.319977] Node 0 DMA free:4016kB min:52kB low:64kB high:76kB active_anon:0kB inactive_anon:0kB active_file:196kB inactive_file:9112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13828kB mlocked:0kB dirty:9128kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:592kB slab_unreclaimable:60kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:10656 all_unreclaimable? no
May 2 16:05:28 hercules kernel: [ 1768.319986] lowmem_reserve[]: 0 994 994 994
May 2 16:05:28 hercules kernel: [ 1768.319991] Node 0 DMA32 free:4104kB min:4004kB low:5004kB high:6004kB active_anon:4kB inactive_anon:100kB active_file:59992kB inactive_file:774936kB unevictable:20kB isolated(anon):0kB isolated(file):512kB present:1018080kB mlocked:20kB dirty:775156kB writeback:572kB mapped:14368kB shmem:0kB slab_reclaimable:57940kB slab_unreclaimable:18312kB kernel_stack:1376kB pagetables:3828kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:441765 all_unreclaimable? no
May 2 16:05:28 hercules kernel: [ 1768.320001] lowmem_reserve[]: 0 0 0 0
May 2 16:05:28 hercules kernel: [ 1768.320005] Node 0 DMA: 4*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4016kB
May 2 16:05:28 hercules kernel: [ 1768.320019] Node 0 DMA32: 28*4kB 265*8kB 115*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4104kB
May 2 16:05:28 hercules kernel: [ 1768.320030] 211215 total pagecache pages
May 2 16:05:28 hercules kernel: [ 1768.320032] 25 pages in swap cache
May 2 16:05:28 hercules kernel: [ 1768.320034] Swap cache stats: add 19348, delete 19323, find 514/1053
May 2 16:05:28 hercules kernel: [ 1768.320036] Free swap = 1894804kB
May 2 16:05:28 hercules kernel: [ 1768.320037] Total swap = 1959800kB
May 2 16:05:28 hercules kernel: [ 1768.322969] 262144 pages RAM
May 2 16:05:28 hercules kernel: [ 1768.322972] 22887 pages reserved
May 2 16:05:28 hercules kernel: [ 1768.322973] 218032 pages shared
May 2 16:05:28 hercules kernel: [ 1768.322974] 27151 pages non-shared
May 2 16:05:28 hercules kernel: [ 1768.322978] Out of memory: kill process 2640 (sshd) score 21829 or a child
May 2 16:05:28 hercules kernel: [ 1768.323109] Killed process 2656 (bash)



-- System Information:
Debian Release: 6.0.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages xen-linux-system-2.6.32-5-xen-amd64 depends on:
ii linux-image-2.6.32-5-xen-amd6 2.6.32-31 Linux 2.6.32 for 64-bit PCs, Xen d
ii xen-hypervisor-4.0-amd64 [xen 4.0.1-2 The Xen Hypervisor on AMD64

xen-linux-system-2.6.32-5-xen-amd64 recommends no packages.

xen-linux-system-2.6.32-5-xen-amd64 suggests no packages.


-- no debconf information



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110502145050.4202.47150.reportbug@hercules.sourc epark.local">http://lists.debian.org/20110502145050.4202.47150.reportbug@hercules.sourc epark.local
 
Old 05-02-2011, 05:35 PM
Ben Hutchings
 
Default Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote:
> Package: xen-linux-system-2.6.32-5-xen-amd64
> Version: 2.6.32-31
> Severity: critical
> Justification: breaks the whole system
>
> Hi,
>
> I have 64 bit xen kernel from squeeze installed on a dual xeon
> maschine. Usually everything runs fine until it comes to heavy load on
> a domU with high I/O and memory consumption.
> This causes the dom0 to run out of memory and to kill several
> processes (see log below). As a consequence of this, the whole system
> becomes unusable.
>
> I tried several things like assign dedicated memory to dom0, disable
> balloning, increase scheduler domain weights and assigned dedicated
> CPUs to dom0 as described in
> http://wiki.xensource.com/xenwiki/XenBestPractices but had no success.
>
> I think a domU should never break the whole system, so this might be a
> bug. Please let me know if you need further information.
[...]

How much physical RAM is in the system and how much memory do you assign
to each domain?

Ben.

--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
 
Old 05-03-2011, 07:42 AM
Sebastian Hofmann
 
Default Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

Hi,

The maschine has 12G of RAM. For dom0 I tried several configurations -
4G with balloning enabled, at least 1G statically assigned. DomUs have
different memory settings.

xm info | grep mem:

total_memory : 12224
free_memory : 1589
node_to_memory : node0:1077
node_to_dma32_mem : node0:1074
xen_commandline : placeholder dom0_mem=1024M dom0_max_vcpus=2
dom0_vcpus_pin


dom0 free -m:

total used free shared buffers cached
Mem: 1019 1008 10 0 249 528
-/+ buffers/cache: 231 788
Swap: 1913 29 1884


xm list:

Name ID Mem VCPUs State
Time(s)
Domain-0 0 1019 2 r-----
12087.5
alfresco 1 3096 1 -b----
57.0
destiny 3 384 1 -b----
919.5
hudson32 4 384 1 -b----
165.2
hudson64 5 1536 1 -b----
2697.1
ms2cistage 7 2048 1 -b----
10.5
ms2istage 6 2048 1 -b----
19.1


Thanks
Sebastian


Am 02.05.2011 19:35, schrieb Ben Hutchings:
> On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote:
>> Package: xen-linux-system-2.6.32-5-xen-amd64
>> Version: 2.6.32-31
>> Severity: critical
>> Justification: breaks the whole system
>>
>> Hi,
>>
>> I have 64 bit xen kernel from squeeze installed on a dual xeon
>> maschine. Usually everything runs fine until it comes to heavy load on
>> a domU with high I/O and memory consumption.
>> This causes the dom0 to run out of memory and to kill several
>> processes (see log below). As a consequence of this, the whole system
>> becomes unusable.
>>
>> I tried several things like assign dedicated memory to dom0, disable
>> balloning, increase scheduler domain weights and assigned dedicated
>> CPUs to dom0 as described in
>> http://wiki.xensource.com/xenwiki/XenBestPractices but had no success.
>>
>> I think a domU should never break the whole system, so this might be a
>> bug. Please let me know if you need further information.
> [...]
>
> How much physical RAM is in the system and how much memory do you assign
> to each domain?
>
> Ben.
>

--
--------------------------------------
M.Sc. Sebastian Hofmann

SOURCEPARK GmbH
Hohenzollerndamm 150 Haus 7a
14199 Berlin

Tel: +49 (0)30/398 068 30
Fax: +49 (0)30/398 068 39
e-mail: sebastian.hofmann@sourcepark.de
www: www.sourcepark.de
--------------------------------------
SOURCEPARK GmbH
Sitz der Gesellschaft: Berlin / Amtsgericht Charlottenburg
HRB 80254
Geschäftsführung: Matthias Barmeier, Harald Dürr
--------------------------------------

Wichtiger Hinweis: Die vorgenannten Angaben werden jeder E-Mail
automatisch hinzugefügt und lassen keine Rückschlüsse auf den
Rechtscharakter der E-Mail zu.

Diese E-Mail kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten. Wenn Sie nicht der richtige Adressat
sind oder diese E-Mail irrtümlich erhalten haben, informieren
Sie bitte sofort den Absender und vernichten Sie diese E-Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser E-Mail ist nicht gestattet.



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DBFB1E2.6080203@sourcepark.de">http://lists.debian.org/4DBFB1E2.6080203@sourcepark.de
 
Old 05-12-2011, 08:20 AM
Ian Campbell
 
Default Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

Hi Sebastian,

Thanks for your report.

On Mon, 2011-05-02 at 16:50 +0200, Sebastian Hofmann wrote:
> Package: xen-linux-system-2.6.32-5-xen-amd64
> Version: 2.6.32-31
> Severity: critical
> Justification: breaks the whole system
>
> Hi,
>
> I have 64 bit xen kernel from squeeze installed on a dual xeon
> maschine. Usually everything runs fine until it comes to heavy load on
> a domU with high I/O and memory consumption.
> This causes the dom0 to run out of memory and to kill several
> processes (see log below). As a consequence of this, the whole system
> becomes unusable.
>
> I tried several things like assign dedicated memory to dom0, disable
> balloning, increase scheduler domain weights and assigned dedicated
> CPUs to dom0 as described in
> http://wiki.xensource.com/xenwiki/XenBestPractices but had no success.
>
> I think a domU should never break the whole system, so this might be a
> bug. Please let me know if you need further information.
>
> Thanks
> Sebastian
>
>
> May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
> May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0
> May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1

Am I right that nrpe is part of nagios? (it's probably just the unluck
process so tells us nothing really)

What sort of load are the domUs experiencing? i.e. CPU, network, disk
etc.

What does your storage stack look like? (are you using LVM, iSCSI, DRDB,
SW RAID, filesystems etc).

Are you running anything interesting in domain 0 other than the Xen
toolstack, nagios, sshd etc?

What does /proc/meminfo look like after a fresh boot?

Lastly, please can you provide a dmesg log of the initial bootup.

Cheers,
Ian.

--
Ian Campbell
Current Noise: Behemoth - Alas, Lord Is Upon Me

To stay young requires unceasing cultivation of the ability to unlearn
old falsehoods.
-- Lazarus Long, "Time Enough For Love"




--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1305188425.26692.471.camel@zakaz.uk.xensource.com" >http://lists.debian.org/1305188425.26692.471.camel@zakaz.uk.xensource.com
 
Old 05-19-2011, 03:05 PM
Sebastian Hofmann
 
Default Bug#625217: xen-linux-system-2.6.32-5-xen-amd64: Heavy load on domU causes dom0 to run out of memory

Hi Ian,

Thanks for your reply.

In the meantime I found out that I forgot to change the device name of
the swap devices in fstab of the domUs when upgrading to XEN 4.0. So in
fact all domUs had no swap space at all.

I think this might be the problem that a domU ran out of memory...

However, the dom0 always had a swap space, so the question is still why
the dom0 also runs out of memory.

>> May 2 16:05:26 hercules kernel: [ 1768.319877] nrpe invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
>> May 2 16:05:26 hercules kernel: [ 1768.319883] nrpe cpuset=/ mems_allowed=0
>> May 2 16:05:26 hercules kernel: [ 1768.319886] Pid: 2118, comm: nrpe Not tainted 2.6.32-5-xen-amd64 #1
>
> Am I right that nrpe is part of nagios? (it's probably just the unluck
> process so tells us nothing really)

You are right - this log message was just an example of what happens.

> What sort of load are the domUs experiencing? i.e. CPU, network, disk
> etc.

CPU and memory consumption

> What does your storage stack look like? (are you using LVM, iSCSI, DRDB,
> SW RAID, filesystems etc).

SW Raid

> Are you running anything interesting in domain 0 other than the Xen
> toolstack, nagios, sshd etc?

nagios nrpe, sshd, fail2ban-server, bacula-fd, rsyncd, postfix

> What does /proc/meminfo look like after a fresh boot?
>
> Lastly, please can you provide a dmesg log of the initial bootup.

I will provide the requested information as soon as I can reboot the
machine.

We also will do some tests with high load again on the domUs with
enabled swap space to see if the problem still exists.

> Cheers,
> Ian.

Thanks
Sebastian



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4DD531B5.2040601@sourcepark.de">http://lists.debian.org/4DD531B5.2040601@sourcepark.de
 

Thread Tools




All times are GMT. The time now is 08:16 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org