FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 05-17-2012, 07:08 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

Hello,

In general I am in the habit of turning off memory overcommit because I
believe it's a bad thing in a multi-user environment. This was never a
problem on rhel5 systems, but on rhel6, I am having issues. When I try
to set overcommit_memory=2, my system locks up. It basically behaves as
if the memory is all used up... I see the same behavior on centos6 or
rhel6. Following is some output from each platform.

# -- RHEL6
# uname -a
Linux joker 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64
x86_64 x86_64 GNU/Linux

#lsb_release -a
LSB Version:
:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarchrinting-4.0-amd64rinting-4.0-noarch
Distributor ID: RedHatEnterpriseWorkstation
Description: Red Hat Enterprise Linux Workstation release 6.2 (Santiago)
Release: 6.2
Codename: Santiago

# free
total used free shared buffers cached
Mem: 2052176 234828 1817348 0 15352 112852
-/+ buffers/cache: 106624 1945552
Swap: 2052088 0 2052088

# sysctl -a |grep commit
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.nr_overcommit_hugepages = 0
# sysctl -w vm.overcommit_memory=2
vm.overcommit_memory = 2
# ls
-bash: fork: Cannot allocate memory
#

#--- CENTOS6 --------------------------
# uname
-a
Linux joker 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64
x86_64 x86_64 GNU/Linux

# lsb_release -a
LSB Version:
:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarchrinting-4.0-amd64rinting-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.2 (Final)
Release: 6.2
Codename: Final

# free
total used free shared buffers cached
Mem: 2052176 378668 1673508 0 19472 252576
-/+ buffers/cache: 106620 1945556
Swap: 2052088 0 2052088

# sysctl -a |grep commit
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.nr_overcommit_hugepages = 0
# sysctl -w vm.overcommit_memory=2
vm.overcommit_memory = 2
[root@joker ~]# ls
-bash: fork: Cannot allocate memory

One last point. If I set the overcommit values in /etc/sysctl.conf and
then reboot, the values get set correctly on boot and everything seems
fine. In addition I can then change the value of overcommit_memory to 0
and back to 2 with out any ill affects.

Searches for issues with setting overcommit_memory=2 haven't turned up
anything useful..


Thanks.
--
-MichaelC
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 12:15 PM
John Stanley
 
Default Setting overcommit_memory=2 kills system

On Thu, 2012-05-17 at 13:08 -0600, Michael Coffman wrote:

> vm.overcommit_ratio = 50
^^

Try changing the commit ratio to a greater value (yes you can go over
100).
Then sysctl -w vm.overcommit_memory=2

Just because one machine fails gracefully does not mean the next will.
But really buy some ram for real. OR solve the real problem.



John

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 01:46 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 6:15 AM, John Stanley <john.stanley@elslc.com>wrote:

> On Thu, 2012-05-17 at 13:08 -0600, Michael Coffman wrote:
>
> > vm.overcommit_ratio = 50
> ^^
>
> Try changing the commit ratio to a greater value (yes you can go over
> 100).
> Then sysctl -w vm.overcommit_memory=2
>
>
OK.


> Just because one machine fails gracefully does not mean the next will.
>

I don't even know what the above means.


> But really buy some ram for real. OR solve the real problem.
>
>

Well that's just ridiculous. 'Real problem'?. If I run a command to
change a kernel value, I assume that my system will not be rendered useless
because it now believes all the memory is consumed. It seems like a
problem to me no matter how much memory the machine has.



>
>
> John
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
-MichaelC
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 01:50 PM
Les Mikesell
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 8:46 AM, Michael Coffman
<michael.coffman@avagotech.com> wrote:
>
>> Just because one machine fails gracefully does not mean the next will.
>>
>
> I don't even know what the above means.
>
>
>> But really buy some ram for real. OR solve the real problem.
>>
>>
>
> Well that's just ridiculous. *'Real problem'?. * *If I run a command to
> change a kernel value, I assume that my system will not be rendered useless
> because it now believes all the memory is consumed. * It seems like a
> problem to me no matter how much memory the machine has.

If you are already overcommitted, what do you expect to happen when
you say not to allow that? The kernel doesn't have a really good way
to handle that situation (or any other OOM condition for that
matter...).

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 04:24 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 7:50 AM, Les Mikesell <lesmikesell@gmail.com> wrote:

> On Fri, May 18, 2012 at 8:46 AM, Michael Coffman
> <michael.coffman@avagotech.com> wrote:
> >
> >> Just because one machine fails gracefully does not mean the next will.
> >>
> >
> > I don't even know what the above means.
> >
> >
> >> But really buy some ram for real. OR solve the real problem.
> >>
> >>
> >
> > Well that's just ridiculous. 'Real problem'?. If I run a command to
> > change a kernel value, I assume that my system will not be rendered
> useless
> > because it now believes all the memory is consumed. It seems like a
> > problem to me no matter how much memory the machine has.
>
> If you are already overcommitted, what do you expect to happen when
> you say not to allow that? The kernel doesn't have a really good way
> to handle that situation (or any other OOM condition for that
> matter...).
>
>
OK. So I was confused because free shows I have plenty of memory and I am
only now noticing that Committed_AS shows a very large value. This is
largely an idle system:

total used free shared buffers cached
Mem: 2052176 951648 1100528 0 147580 626096
-/+ buffers/cache: 177972 1874204
Swap: 2052088 0 2052088

So my real question should have been why would the Committed_AS value be so
large?
Committed_AS: 137197248820 kB

On the exact same hardware with fresh build of centos5u4 and overcommit
turned on:
Committed_AS: 125716 kB



--
> Les Mikesell
> lesmikesell@gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
-MichaelC
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 04:39 PM
Les Mikesell
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 11:24 AM, Michael Coffman
<michael.coffman@avagotech.com> wrote:
>
>> If you are already overcommitted, what do you expect to happen when
>> you say not to allow that? * The kernel doesn't have a really good way
>> to handle that situation (or any other OOM condition for that
>> matter...).
>>
>>
> OK. *So I was confused because free shows I have plenty of memory and I am
> only now noticing that Committed_AS shows a very large value. * This is
> largely an idle system:
>
> * * * * * * total * * * used * * * free * * shared * *buffers * * cached
> Mem: * * * 2052176 * * 951648 * *1100528 * * * * *0 * * 147580 * * 626096
> -/+ buffers/cache: * * 177972 * *1874204
> Swap: * * *2052088 * * * * *0 * *2052088
>
> So my real question should have been why would the Committed_AS value be so
> large?
> Committed_AS: * 137197248820 kB
>
> On the exact same hardware with fresh build of centos5u4 and overcommit
> turned on:
> Committed_AS: * 125716 kB

I think that means an application has malloc()'d but not actually used
a huge amount of memory. It used to be a common practice to test the
amount of memory available by requesting more until the request
failed, so it could be meaningless. If overcommit is turned off the
app gets an error when it expects and backs off. The Committed_AS is
an estimate of how much RAM/swap you would be likely to need if the
requested storage is actually used.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 05:03 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 10:39 AM, Les Mikesell <lesmikesell@gmail.com>wrote:

> On Fri, May 18, 2012 at 11:24 AM, Michael Coffman
> <michael.coffman@avagotech.com> wrote:
> >
> >> If you are already overcommitted, what do you expect to happen when
> >> you say not to allow that? The kernel doesn't have a really good way
> >> to handle that situation (or any other OOM condition for that
> >> matter...).
> >>
> >>
> > OK. So I was confused because free shows I have plenty of memory and I
> am
> > only now noticing that Committed_AS shows a very large value. This is
> > largely an idle system:
> >
> > total used free shared buffers cached
> > Mem: 2052176 951648 1100528 0 147580 626096
> > -/+ buffers/cache: 177972 1874204
> > Swap: 2052088 0 2052088
> >
> > So my real question should have been why would the Committed_AS value be
> so
> > large?
> > Committed_AS: 137197248820 kB
> >
> > On the exact same hardware with fresh build of centos5u4 and overcommit
> > turned on:
> > Committed_AS: 125716 kB
>
> I think that means an application has malloc()'d but not actually used
> a huge amount of memory. It used to be a common practice to test the
> amount of memory available by requesting more until the request
> failed, so it could be meaningless. If overcommit is turned off the
> app gets an error when it expects and backs off. The Committed_AS is
> an estimate of how much RAM/swap you would be likely to need if the
> requested storage is actually used.
>
>
I get how overcommit works and it's affect on malloc. I intentionally turn
it off so malloc will return a correct value when a process can't get
enough memory. What I currently don't understand is the large value of
Committed_AS on my centos6/rhel6 systems.

I just did a fresh build of centos6u2 and there is very little running and
the value is:
Committed_AS: 136382202700 kB

There is almost nothing running.... Filtering out kernel processes here
is the list of running processes

$ps -e -o uid,pid,ppid,rsz,vsz,cmd |grep -v [ UID
PID PPID RSZ VSZ CMD
0 1 0 1608 23636 /sbin/init
0 493 1 1156 11112 /sbin/udevd -d
0 1072 493 1168 11108 /sbin/udevd -d
0 1073 493 1164 11108 /sbin/udevd -d
0 1147 1 828 27688 auditd
0 1172 1 1472 252980 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
32 1215 1 1004 19084 rpcbind
29 1233 1 1212 25292 rpc.statd
81 1269 1 940 25676 dbus-daemon --system
70 1281 1280 524 31952 avahi-daemon: chroot helper
0 1291 1 3420 92432 cupsd -C /etc/cups/cupsd.conf
0 1316 1 624 4132 /usr/sbin/acpid
68 1325 1 4272 29572 hald
0 1326 1325 1336 18156 hald-runner
0 1365 1326 1268 20272 hald-addon-input: Listening on
/dev/input/event3 /dev/input/event0 /dev/input/event1
68 1369 1326 1164 17856 hald-addon-acpi: listening on acpid socket
/var/run/acpid.socket
0 1389 1 1428 120820 /usr/sbin/ypbind
0 1439 1 280 10524 rpc.rquotad
0 1455 1 808 21464 rpc.mountd
0 1472 1 520 6300 /usr/sbin/mcelog --daemon
0 1492 1 1216 64068 /usr/sbin/sshd
0 1500 1 1088 24236 xinetd -stayalive -pidfile
/var/run/xinetd.pid
38 1508 1 1740 34448 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
0 1516 1 368 8356 rpc.rstatd
0 1525 1 6220 57364 /usr/sbin/amd -F /etc/amd.conf
0 1608 1 3304 80768 /usr/libexec/postfix/master
0 1629 1 884 9216 abrt-dump-oops -d /var/spool/abrt -rwx
/var/log/messages
0 1638 1 1304 22000 /usr/sbin/abrtd
0 1646 1 1236 20416 crond
0 2032 1 468 21492 /usr/sbin/atd
0 2292 1 4564 100224 /usr/sbin/snmpd -LS 4 d -Lf /dev/null -p
/var/run/snmpd.pid -a
89 2305 1608 3260 80848 pickup -l -t fifo -u
89 2306 1608 3304 80916 qmgr -l -t fifo -u
99 2425 1 4052 136382123488
/opt/ictools/64bit/synopsys-aserver/bin/AServer
0 2433 1 580 4116 /sbin/mingetty /dev/tty1
0 2435 1 584 4116 /sbin/mingetty /dev/tty2
0 2437 1 580 4116 /sbin/mingetty /dev/tty3
0 2439 1 580 4116 /sbin/mingetty /dev/tty4
0 2441 1 580 4116 /sbin/mingetty /dev/tty5
0 2443 1 580 4116 /sbin/mingetty /dev/tty6
0 3001 1492 3888 99900 sshd: rootm@pts/0
0 3004 3001 3692 13320 -bash
0 3277 3004 920 11316 ps -e -o uid,pid,ppid,rsz,vsz,cmd



> --
> Les Mikesell
> lesmikesell@gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
-MichaelC
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 05:09 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 10:39 AM, Les Mikesell <lesmikesell@gmail.com>wrote:

> On Fri, May 18, 2012 at 11:24 AM, Michael Coffman
> <michael.coffman@avagotech.com> wrote:
> >
> >> If you are already overcommitted, what do you expect to happen when
> >> you say not to allow that? The kernel doesn't have a really good way
> >> to handle that situation (or any other OOM condition for that
> >> matter...).
> >>
> >>
> > OK. So I was confused because free shows I have plenty of memory and I
> am
> > only now noticing that Committed_AS shows a very large value. This is
> > largely an idle system:
> >
> > total used free shared buffers cached
> > Mem: 2052176 951648 1100528 0 147580 626096
> > -/+ buffers/cache: 177972 1874204
> > Swap: 2052088 0 2052088
> >
> > So my real question should have been why would the Committed_AS value be
> so
> > large?
> > Committed_AS: 137197248820 kB
> >
> > On the exact same hardware with fresh build of centos5u4 and overcommit
> > turned on:
> > Committed_AS: 125716 kB
>
> I think that means an application has malloc()'d but not actually used
> a huge amount of memory. It used to be a common practice to test the
> amount of memory available by requesting more until the request
> failed, so it could be meaningless. If overcommit is turned off the
> app gets an error when it expects and backs off. The Committed_AS is
> an estimate of how much RAM/swap you would be likely to need if the
> requested storage is actually used.
>
>
Hmm. I replied and included a PS listing and go the following error:

Symantec Mail Security detected prohibited content in a message sent from
your address (SYM:13316346581192243434)

My reply -

I get how overcommit works and it's affect on malloc. I intentionally turn
it off so malloc will return a correct value when a process can't get
enough memory. What I currently don't understand is the large value of
Committed_AS on my centos6/rhel6 systems.

I just did a fresh build of centos6u2 and there is very little running and
the value is:
Committed_AS: 136382202700 kB

There is almost nothing running.... Filtering out kernel processes here
is the list of running processes

See attached ps listing if interested...

--
> Les Mikesell
> lesmikesell@gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
-MichaelC
$ps -e -o uid,pid,ppid,rsz,vsz,cmd |grep -v [ UID PID PPID RSZ VSZ CMD
0 1 0 1608 23636 /sbin/init
0 493 1 1156 11112 /sbin/udevd -d
0 1072 493 1168 11108 /sbin/udevd -d
0 1073 493 1164 11108 /sbin/udevd -d
0 1147 1 828 27688 auditd
0 1172 1 1472 252980 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
32 1215 1 1004 19084 rpcbind
29 1233 1 1212 25292 rpc.statd
81 1269 1 940 25676 dbus-daemon --system
70 1281 1280 524 31952 avahi-daemon: chroot helper
0 1291 1 3420 92432 cupsd -C /etc/cups/cupsd.conf
0 1316 1 624 4132 /usr/sbin/acpid
68 1325 1 4272 29572 hald
0 1326 1325 1336 18156 hald-runner
0 1365 1326 1268 20272 hald-addon-input: Listening on /dev/input/event3 /dev/input/event0 /dev/input/event1
68 1369 1326 1164 17856 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
0 1389 1 1428 120820 /usr/sbin/ypbind
0 1439 1 280 10524 rpc.rquotad
0 1455 1 808 21464 rpc.mountd
0 1472 1 520 6300 /usr/sbin/mcelog --daemon
0 1492 1 1216 64068 /usr/sbin/sshd
0 1500 1 1088 24236 xinetd -stayalive -pidfile /var/run/xinetd.pid
38 1508 1 1740 34448 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
0 1516 1 368 8356 rpc.rstatd
0 1525 1 6220 57364 /usr/sbin/amd -F /etc/amd.conf
0 1608 1 3304 80768 /usr/libexec/postfix/master
0 1629 1 884 9216 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
0 1638 1 1304 22000 /usr/sbin/abrtd
0 1646 1 1236 20416 crond
0 2032 1 468 21492 /usr/sbin/atd
0 2292 1 4564 100224 /usr/sbin/snmpd -LS 4 d -Lf /dev/null -p /var/run/snmpd.pid -a
89 2305 1608 3260 80848 pickup -l -t fifo -u
89 2306 1608 3304 80916 qmgr -l -t fifo -u
99 2425 1 4052 136382123488 /opt/ictools/64bit/synopsys-aserver/bin/AServer
0 2433 1 580 4116 /sbin/mingetty /dev/tty1
0 2435 1 584 4116 /sbin/mingetty /dev/tty2
0 2437 1 580 4116 /sbin/mingetty /dev/tty3
0 2439 1 580 4116 /sbin/mingetty /dev/tty4
0 2441 1 580 4116 /sbin/mingetty /dev/tty5
0 2443 1 580 4116 /sbin/mingetty /dev/tty6
0 3001 1492 3888 99900 sshd: rootm@pts/0
0 3004 3001 3692 13320 -bash
0 3277 3004 920 11316 ps -e -o uid,pid,ppid,rsz,vsz,cmd
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 05:13 PM
Michael Coffman
 
Default Setting overcommit_memory=2 kills system

On Fri, May 18, 2012 at 10:39 AM, Les Mikesell <lesmikesell@gmail.com>wrote:

> On Fri, May 18, 2012 at 11:24 AM, Michael Coffman
> <michael.coffman@avagotech.com> wrote:
> >
> >> If you are already overcommitted, what do you expect to happen when
> >> you say not to allow that? The kernel doesn't have a really good way
> >> to handle that situation (or any other OOM condition for that
> >> matter...).
> >>
> >>
> > OK. So I was confused because free shows I have plenty of memory and I
> am
> > only now noticing that Committed_AS shows a very large value. This is
> > largely an idle system:
> >
> > total used free shared buffers cached
> > Mem: 2052176 951648 1100528 0 147580 626096
> > -/+ buffers/cache: 177972 1874204
> > Swap: 2052088 0 2052088
> >
> > So my real question should have been why would the Committed_AS value be
> so
> > large?
> > Committed_AS: 137197248820 kB
> >
> > On the exact same hardware with fresh build of centos5u4 and overcommit
> > turned on:
> > Committed_AS: 125716 kB
>
> I think that means an application has malloc()'d but not actually used
> a huge amount of memory. It used to be a common practice to test the
> amount of memory available by requesting more until the request
> failed, so it could be meaningless. If overcommit is turned off the
> app gets an error when it expects and backs off. The Committed_AS is
> an estimate of how much RAM/swap you would be likely to need if the
> requested storage is actually used.
>
> I get how overcommit works and it's affect on malloc. I intentionally
turn it off so malloc will return a correct value when a process can't get
enough memory. What I currently don't understand is the large value of
Committed_AS on my centos6/rhel6 systems.

I just did a fresh build of centos6u2 and there is very little running and
the value is:
Committed_AS: 136382202700 kB

There is almost nothing running.... Filtering out kernel processes here
is the list of running processes

I tried to post the ps listing, both as inline text and as an attached
ps.txt file and the message was rejected by Symantec

--
> Les Mikesell
> lesmikesell@gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
-MichaelC
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-18-2012, 05:23 PM
John R Pierce
 
Default Setting overcommit_memory=2 kills system

On 05/18/12 10:09 AM, Michael Coffman wrote:
> 99 2425 1 4052 136382123488 /opt/ictools/64bit/synopsys-aserver/bin/AServer
^^^^^^^^^^

well, I dunno what that process is, but thats a HUGE vsize.



--
john r pierce N 37, W 122
santa cruz ca mid-left coast

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 09:35 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org