FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 01-12-2011, 05:59 PM
Kaddeh
 
Default Web Server Memory Issues

So, I have run into an interesting problem while building out a web server for a client which I haven't come across before and I was hoping that the list would be a good way for me to find the answer.
*
A little beckground on the systems:
P4 @ 3.0Ghz
2GB PC2 4200
2x 250GB drives in RAID1
*
The system configurations are default for the most part with the server running MySQL and Apache.*
*
The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it.

*
At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue.
*
Anything that would help at this point would be much appreciated.
*
Cheers
*
Kad
 
Old 01-12-2011, 08:37 PM
Jarry
 
Default Web Server Memory Issues

On 12. 1. 2011 19:59, Kaddeh wrote:


P4 @ 3.0Ghz
2GB PC2 4200
2x 250GB drives in RAID1
The system configurations are default for the most part with the server
running MySQL and Apache.
The problem that I am running into at this point, however is that the
machine seems to run out of memory and will segfault either apache or
mysql when does so, when apache segfaults, it is a recoverable error,
when mysql does it, mysql can't recover short of restarting it.
At this point, I have found a soft fix by running a cron job every 6
hours or so to clear the cached memory, which seems to be the problem,
however, I would like to find a more permanent fix to this issue.


First of all, find what is causing that excessive memory usage.
I think 2GB should be enough for moderate web with apache+mysql.

Second, use some monitoring software. Personally I'm using
"monit" and I am very satisfied with it. It can monitor processes
(if it is running, answering requests, etc), resources (disk,
memory, swap, cpu, i/o), files (content, permissions, checksums),
remote hosts (with some basic protocol checks i.e. http, ssh,
smtp, ftp, mysql, ntp, dns...), it can inform you about problems
(mail, log) and you can define rules what to do in case of anomalies
(i.e. if mysql is using to much memory, it will be restarted).

It can start/restart processes if they die (happened to me once
with sshd on server which was ~50 miles away from me). You can
put monit in inittab, so in case monit itself dies it is restarted
automatically. Etc, etc.

Jarry

--
__________________________________________________ _____________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.
 
Old 01-12-2011, 09:47 PM
Kaddeh
 
Default Web Server Memory Issues

Jarry,

Thanks for the monitoring advice, I am checking out monit right now.

In terms of what is the root cause of the issue, I have narrowed it down to either write caching of a SQL cache issue.

First, addressing the SQL issue and why I think that that could be one of the causes.* The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that.


The write caching hteroy just comes up because I can clear the cached memory down to 14mb cached using 'sync && echo 3 > /proc/sys/vm/dump_cache'

Cheers

Kad


On Wed, Jan 12, 2011 at 1:37 PM, Jarry <mr.jarry@gmail.com> wrote:

On 12. 1. 2011 19:59, Kaddeh wrote:




P4 @ 3.0Ghz

2GB PC2 4200

2x 250GB drives in RAID1

The system configurations are default for the most part with the server

running MySQL and Apache.

The problem that I am running into at this point, however is that the

machine seems to run out of memory and will segfault either apache or

mysql when does so, when apache segfaults, it is a recoverable error,

when mysql does it, mysql can't recover short of restarting it.

At this point, I have found a soft fix by running a cron job every 6

hours or so to clear the cached memory, which seems to be the problem,

however, I would like to find a more permanent fix to this issue.




First of all, find what is causing that excessive memory usage.

I think 2GB should be enough for moderate web with apache+mysql.



Second, use some monitoring software. Personally I'm using

"monit" and I am very satisfied with it. It can monitor processes

(if it is running, answering requests, etc), resources (disk,

memory, swap, cpu, i/o), files (content, permissions, checksums),

remote hosts (with some basic protocol checks i.e. http, ssh,

smtp, ftp, mysql, ntp, dns...), it can inform you about problems

(mail, log) and you can define rules what to do in case of anomalies

(i.e. if mysql is using to much memory, it will be restarted).



It can start/restart processes if they die (happened to me once

with sshd on server which was ~50 miles away from me). You can

put monit in inittab, so in case monit itself dies it is restarted

automatically. Etc, etc.



Jarry



--

__________________________________________________ _____________

This mailbox accepts e-mails only from selected mailing-lists!

Everything else is considered to be spam and therefore deleted.
 
Old 01-12-2011, 11:08 PM
Matthew Summers
 
Default Web Server Memory Issues

On Wed, Jan 12, 2011 at 4:47 PM, Kaddeh <kaddeh@gmail.com> wrote:
> Jarry,
>
> Thanks for the monitoring advice, I am checking out monit right now.
>
> In terms of what is the root cause of the issue, I have narrowed it down to
> either write caching of a SQL cache issue.
>
> First, addressing the SQL issue and why I think that that could be one of
> the causes.* The entire site, for the most part is all in one giant DB
> (~9GB) a significant part of that is a 3gb table full of raw image data
> (yes, I know that this is a REALLY bad idea to do, but I didn't design the
> site, I just did a migration to off-site) that being said, there could be a
> problem with that.
>
> The write caching hteroy just comes up because I can clear the cached memory
> down to 14mb cached using 'sync && echo 3 > /proc/sys/vm/dump_cache'
>
> Cheers
>
> Kad
>
> On Wed, Jan 12, 2011 at 1:37 PM, Jarry <mr.jarry@gmail.com> wrote:
>>
>> On 12. 1. 2011 19:59, Kaddeh wrote:
>>
>>> P4 @ 3.0Ghz
>>> 2GB PC2 4200
>>> 2x 250GB drives in RAID1
>>> The system configurations are default for the most part with the server
>>> running MySQL and Apache.
>>> The problem that I am running into at this point, however is that the
>>> machine seems to run out of memory and will segfault either apache or
>>> mysql when does so, when apache segfaults, it is a recoverable error,
>>> when mysql does it, mysql can't recover short of restarting it.
>>> At this point, I have found a soft fix by running a cron job every 6
>>> hours or so to clear the cached memory, which seems to be the problem,
>>> however, I would like to find a more permanent fix to this issue.
>>
>> First of all, find what is causing that excessive memory usage.
>> I think 2GB should be enough for moderate web with apache+mysql.
>>
>> Second, use some monitoring software. Personally I'm using
>> "monit" and I am very satisfied with it. It can monitor processes
>> (if it is running, answering requests, etc), resources (disk,
>> memory, swap, cpu, i/o), files (content, permissions, checksums),
>> remote hosts (with some basic protocol checks i.e. http, ssh,
>> smtp, ftp, mysql, ntp, dns...), it can inform you about problems
>> (mail, log) and you can define rules what to do in case of anomalies
>> (i.e. if mysql is using to much memory, it will be restarted).
>>
>> It can start/restart processes if they die (happened to me once
>> with sshd on server which was ~50 miles away from me). You can
>> put monit in inittab, so in case monit itself dies it is restarted
>> automatically. Etc, etc.
>>
>> Jarry
>>
>> --
>> __________________________________________________ _____________
>> This mailbox accepts e-mails only from selected mailing-lists!
>> Everything else is considered to be spam and therefore deleted.
>>
>
>

So, a few questions:

What apache MPM are you using? You can control the number of processes
or threads in that file. The default is something like 200 processes
or threads (depending on MPM), so that could cause issues.

What does your my.cnf look like? MySQL makes it pretty easy to
regulate memory usage in my.cnf.

What sort of webapp is this, PHP, python, perl, ...?

That should be a good start.
Cheers
--
Matthew W. Summers
 
Old 01-12-2011, 11:24 PM
Kaddeh
 
Default Web Server Memory Issues

Matthew,

Default settings for both my.cnf and httpd.conf are defaults, however, I would assume that a restart of a service would clear up the memory that was used by child processes.
The only things that are really different in my.cnf is the base stuff like bin-log and such for doing DB replication.

As for the webapp itself, it is PHP, but that is literally to make the MySQL connections to pull down the pages in the database (literally, entire pages of html in columns).

Cheers

Kad


On Wed, Jan 12, 2011 at 4:08 PM, Matthew Summers <quantumsummers@gentoo.org> wrote:

On Wed, Jan 12, 2011 at 4:47 PM, Kaddeh <kaddeh@gmail.com> wrote:

> Jarry,

>

> Thanks for the monitoring advice, I am checking out monit right now.

>

> In terms of what is the root cause of the issue, I have narrowed it down to

> either write caching of a SQL cache issue.

>

> First, addressing the SQL issue and why I think that that could be one of

> the causes.* The entire site, for the most part is all in one giant DB

> (~9GB) a significant part of that is a 3gb table full of raw image data

> (yes, I know that this is a REALLY bad idea to do, but I didn't design the

> site, I just did a migration to off-site) that being said, there could be a

> problem with that.

>

> The write caching hteroy just comes up because I can clear the cached memory

> down to 14mb cached using 'sync && echo 3 > /proc/sys/vm/dump_cache'

>

> Cheers

>

> Kad

>

> On Wed, Jan 12, 2011 at 1:37 PM, Jarry <mr.jarry@gmail.com> wrote:

>>

>> On 12. 1. 2011 19:59, Kaddeh wrote:

>>

>>> P4 @ 3.0Ghz

>>> 2GB PC2 4200

>>> 2x 250GB drives in RAID1

>>> The system configurations are default for the most part with the server

>>> running MySQL and Apache.

>>> The problem that I am running into at this point, however is that the

>>> machine seems to run out of memory and will segfault either apache or

>>> mysql when does so, when apache segfaults, it is a recoverable error,

>>> when mysql does it, mysql can't recover short of restarting it.

>>> At this point, I have found a soft fix by running a cron job every 6

>>> hours or so to clear the cached memory, which seems to be the problem,

>>> however, I would like to find a more permanent fix to this issue.

>>

>> First of all, find what is causing that excessive memory usage.

>> I think 2GB should be enough for moderate web with apache+mysql.

>>

>> Second, use some monitoring software. Personally I'm using

>> "monit" and I am very satisfied with it. It can monitor processes

>> (if it is running, answering requests, etc), resources (disk,

>> memory, swap, cpu, i/o), files (content, permissions, checksums),

>> remote hosts (with some basic protocol checks i.e. http, ssh,

>> smtp, ftp, mysql, ntp, dns...), it can inform you about problems

>> (mail, log) and you can define rules what to do in case of anomalies

>> (i.e. if mysql is using to much memory, it will be restarted).

>>

>> It can start/restart processes if they die (happened to me once

>> with sshd on server which was ~50 miles away from me). You can

>> put monit in inittab, so in case monit itself dies it is restarted

>> automatically. Etc, etc.

>>

>> Jarry

>>

>> --

>> __________________________________________________ _____________

>> This mailbox accepts e-mails only from selected mailing-lists!

>> Everything else is considered to be spam and therefore deleted.

>>

>

>



So, a few questions:



What apache MPM are you using? You can control the number of processes

or threads in that file. The default is something like 200 processes

or threads (depending on MPM), so that could cause issues.



What does your my.cnf look like? MySQL makes it pretty easy to

regulate memory usage in my.cnf.



What sort of webapp is this, PHP, python, perl, ...?



That should be a good start.

Cheers

--

Matthew W. Summers
 
Old 01-13-2011, 04:34 PM
Stroller
 
Default Web Server Memory Issues

On 12/1/2011, at 10:47pm, Kaddeh wrote:
> ...
> First, addressing the SQL issue and why I think that that could be one of
> the causes. The entire site, for the most part is all in one giant DB
> (~9GB) a significant part of that is a 3gb table full of raw image data
> (yes, I know that this is a REALLY bad idea to do, but I didn't design the
> site, I just did a migration to off-site) that being said, there could be a
> problem with that.

I think I may have heard database guys argue for keeping image data in the DB. I'm not sure that it's always a bad idea.

However: how much swap do you have?

If you have a 12GB swap file, perhaps you will no longer see this problem?

Stroller.
 
Old 01-13-2011, 04:59 PM
Kaddeh
 
Default Web Server Memory Issues

I have a standard 2x RAM swap size of 4gb.
The problem that I am seeing though is that the applications (MySQL and apache) are segfaulting -before- the system starts to swap, almost where they have an aversion to using swap.


Cheers

Kad

On Thu, Jan 13, 2011 at 9:34 AM, Stroller <stroller@stellar.eclipse.co.uk> wrote:



On 12/1/2011, at 10:47pm, Kaddeh wrote:

> ...

> First, addressing the SQL issue and why I think that that could be one of

> the causes. *The entire site, for the most part is all in one giant DB

> (~9GB) a significant part of that is a 3gb table full of raw image data

> (yes, I know that this is a REALLY bad idea to do, but I didn't design the

> site, I just did a migration to off-site) that being said, there could be a

> problem with that.



I think I may have heard database guys argue for keeping image data in the DB. I'm not sure that it's always a bad idea.



However: how much swap do you have?



If you have a 12GB swap file, perhaps you will no longer see this problem?



Stroller.
 
Old 01-13-2011, 06:56 PM
Bill Longman
 
Default Web Server Memory Issues

On 01/13/2011 09:59 AM, Kaddeh wrote:
> I have a standard 2x RAM swap size of 4gb.
> The problem that I am seeing though is that the applications (MySQL and
> apache) are segfaulting -before- the system starts to swap, almost where
> they have an aversion to using swap.

Are you running 32 bits?
 
Old 01-13-2011, 07:14 PM
kashani
 
Default Web Server Memory Issues

On 1/12/2011 10:59 AM, Kaddeh wrote:

So, I have run into an interesting problem while building out a web
server for a client which I haven't come across before and I was hoping
that the list would be a good way for me to find the answer.
A little beckground on the systems:
P4 @ 3.0Ghz
2GB PC2 4200
2x 250GB drives in RAID1
The system configurations are default for the most part with the server
running MySQL and Apache.
The problem that I am running into at this point, however is that the
machine seems to run out of memory and will segfault either apache or
mysql when does so, when apache segfaults, it is a recoverable error,
when mysql does it, mysql can't recover short of restarting it.
At this point, I have found a soft fix by running a cron job every 6
hours or so to clear the cached memory, which seems to be the problem,
however, I would like to find a more permanent fix to this issue.
Anything that would help at this point would be much appreciated.
Cheers
Kad


Overall I'd expect your Mysql is running slow, which causes Apache to
back up, which create more Apache children while your code blocks on the
db, which then uses all the RAM.


1. Assuming you're running prefork, Turn KeepAlives Off if you haven't
already. That'll reduce the number of Apache threads sitting around
doing nothing but using your RAM.


2. The default my.conf in Gentoo (and nearly all distros) is configured
to use 64MB. You should bump this up to 512MB total. The two settings I
would touch are the following and THEY ARE SEPARATE POOLS that do not
share configured memory with each other. Configure accordingly.

innodb_buffer_pool_size = 16M
key_buffer = 16M

Both variables are dynamic and can be set from with Mysql use set
variables key_buffer='10240000'; syntax.


Assuming you use Innodb tables I'd try 256MB for that setting and 128MB
for the key_buffer and see how it goes.


3. Mysql slow query log. Turn it on and look at it. Your db design
sounds sketchy at best and I'd be surprised if your weren't seeing a ton
of slow queries especially with no db tuning.


4. /tmp is how big? Make sure it's a couple of gigs so that Mysql can
build tmp tables in it. Again your db design is strange enough that you
might be generating large tmp tables that file /tmp (and / if you
haven't separated them) and causes Mysql problems. This is a fairly
common problem in my experience. The simplest solution is:

sudo mkdid -p /home/mysql
sudo chown -R mysql: /home/mysql
vi /etc/mysql/my.cnf and change to tmpdir = /home/mysql/
sudo /etc/init.d/mysql restart

Yes, tmpdir is *not* a dynamic variable so you will have to restart
Mysql to make this change.


kashani
 
Old 01-13-2011, 10:45 PM
Kaddeh
 
Default Web Server Memory Issues

yes, but that should have an effect on swap space.

Cheers

Kad

On Thu, Jan 13, 2011 at 11:56 AM, Bill Longman <bill.longman@gmail.com> wrote:

On 01/13/2011 09:59 AM, Kaddeh wrote:

> I have a standard 2x RAM swap size of 4gb.

> The problem that I am seeing though is that the applications (MySQL and

> apache) are segfaulting -before- the system starts to swap, almost where

> they have an aversion to using swap.



Are you running 32 bits?
 

Thread Tools




All times are GMT. The time now is 03:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org