FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 07-25-2012, 01:36 PM
Alan McKay
 
Default Manual OOM killing?

Hey guys and gals,

Yesterday I had one of my scientists kill one of my servers when his
program ran amok and gobbled up all the memory, or forked too many
processes, or I'm just not exactly sure what to be honest.

Is there something I can run manually in cron to look for rampant
programs and kill them? I know that may be hard to discern but I
could also include a list if "known good" programs not to kill, as
well as a list of "known suspect" user IDs

Anyone ever done this? Searching the list on "OOM" does not bring up much.

thanks
-Alan

--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 01:46 PM
Daniel De Marco
 
Default Manual OOM killing?

* Alan McKay <alan.mckay@gmail.com> [07/25/2012 09:36]:
> Yesterday I had one of my scientists kill one of my servers when his
> program ran amok and gobbled up all the memory, or forked too many
> processes, or I'm just not exactly sure what to be honest.
>
> Is there something I can run manually in cron to look for rampant
> programs and kill them? I know that may be hard to discern but I
> could also include a list if "known good" programs not to kill, as
> well as a list of "known suspect" user IDs
>
> Anyone ever done this? Searching the list on "OOM" does not bring up much.

I would look into disabling memory overcommit.

On our compute cluster nodes we use the following sysctls:
# disable overcommit (allocation_limit=swap + overcommit_ratio/100 * phys_mem)
vm.overcommit_ratio=70
vm.overcommit_memory=2

This will prevent programs to allocate more than allocation_limit of
memory. In this way the program that tries to allocate the memory that
would put the memory usage over the limit will be prevented to do so.
The malloc will return NULL.

For the forking too many processes you can look into man limits.conf
where you can limit the maximum number of processes per user.

Daniel.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 01:49 PM
 
Default Manual OOM killing?

Alan McKay wrote:
> Hey guys and gals,
>
> Yesterday I had one of my scientists kill one of my servers when his
> program ran amok and gobbled up all the memory, or forked too many
> processes, or I'm just not exactly sure what to be honest.
>
> Is there something I can run manually in cron to look for rampant
> programs and kill them? I know that may be hard to discern but I
> could also include a list if "known good" programs not to kill, as
> well as a list of "known suspect" user IDs
>
> Anyone ever done this? Searching the list on "OOM" does not bring up
> much.

Yeah, we've had that a few times, even on 64 core systems with a
ridiculous amount of memory. One thing we did was to tell them to limit
the number of cores they were using in the parallel processing threads.
There is some kind of limit you can set up - I forget exactly what it is,
but I'm not sure it will limit memory usage.

Someone suggested yesterday, in another context, giving 'em a VM of their
own. Doing that, you can limit how many cores and how much memory they
have, and then if they crash, it's more their problem than yours.

Or have them buy another server and make it a cluster. We use the torque
package.

mark

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 04:08 PM
Alan McKay
 
Default Manual OOM killing?

Interesting stuff - thanks again guys. Looks like I can get what I
need right here ...

--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 04:10 PM
John Doe
 
Default Manual OOM killing?

From: Alan McKay <alan.mckay@gmail.com>

> Yesterday I had one of my scientists kill one of my servers when his
> program ran amok and gobbled up all the memory, or forked too many
> processes, or I'm just not exactly sure what to be honest.
>
> Is there something I can run manually in cron to look for rampant
> programs and kill them?* I know that may be hard to discern but I
> could also include a list if "known good" programs not to kill, as
> well as a list of "known suspect" user IDs

What about setting limits...?
/etc/security/limits.conf

JD
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 05:24 PM
Warren Young
 
Default Manual OOM killing?

On 7/25/2012 7:36 AM, Alan McKay wrote:
>
> Is there something I can run manually in cron to look for rampant
> programs and kill them?

You appear to be under the impression that you have a technical problem.

What you actually have is a people problem.

Go now, and kneel at the feet of the Bastard Operator From Hell
(http://bofh.ntk.net/) to learn how to deal with such matters.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 07-25-2012, 07:02 PM
Alan McKay
 
Default Manual OOM killing?

On Wed, Jul 25, 2012 at 1:24 PM, Warren Young <warren@etr-usa.com> wrote:
> Go now, and kneel at the feet of the Bastard Operator From Hell
> (http://bofh.ntk.net/) to learn how to deal with such matters.

Well I was looking for my LART ... seem to have misplaced it over the years :-)

--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 10:38 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org