Restart TG apps for high mem-usage
 

Mike McGrath wrote:

Bill Nottingham wrote:
Toshio Kuratomi (a.badger@gmail.com) said:
Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary. We could run
this via cron in alternate hours on each app server. Does this seem
like a good or bad idea to people?



It's a good idea if it's needed, but it's a bad idea that it is
needed. What's

wrong with TG that it leads to this situation?



I was wondering this myself, I know smolt recently had some major
changes to keep memory usage down. Which TG apps are having this issue
and how often? I know MM uses a lot of memory but, AFAIK, it was
determined that there's not much of a leak if there is one and that all
of that memory is actually used.


Looks like smolt was upgraded just before Thanksgiving so it could be
that we've plugged the leaks we had to deal with that inspired me to
write this. Would it be a good idea to have this in place anyways?
With it periodically checking, we would find out that we had problems
when cron emails us a notice that the script had to restart a process.
Without it, we'll be notified when nagios or a user tells us they're
getting timeouts.


I noticed that mirrormanager is currently at 761MB of RSS. If that's
steady-state for mm we'd want to bump the value the script checks for a
bit higher before deploying it or set different values per app.


-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Toshio Kuratomi wrote:

Mike McGrath wrote:

Bill Nottingham wrote:
Toshio Kuratomi (a.badger@gmail.com) said:
Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary. We could run
this via cron in alternate hours on each app server. Does this
seem like a good or bad idea to people?



It's a good idea if it's needed, but it's a bad idea that it is
needed. What's

wrong with TG that it leads to this situation?



I was wondering this myself, I know smolt recently had some major
changes to keep memory usage down. Which TG apps are having this
issue and how often? I know MM uses a lot of memory but, AFAIK, it
was determined that there's not much of a leak if there is one and
that all of that memory is actually used.


Looks like smolt was upgraded just before Thanksgiving so it could be
that we've plugged the leaks we had to deal with that inspired me to
write this. Would it be a good idea to have this in place anyways?
With it periodically checking, we would find out that we had problems
when cron emails us a notice that the script had to restart a process.
Without it, we'll be notified when nagios or a user tells us they're
getting timeouts.


I think its a good idea if for no other reason then allows us to more
actively monitor this stuff, we'll get notified when the app restarts.
+1 from me with the intention that, over time, we get fewer and fewer
restarts.


-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Mike McGrath wrote:

Toshio Kuratomi wrote:

Mike McGrath wrote:

Bill Nottingham wrote:

Toshio Kuratomi (a.badger@gmail.com) said:
Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary. We could run
this via cron in alternate hours on each app server. Does this
seem like a good or bad idea to people?



It's a good idea if it's needed, but it's a bad idea that it is
needed. What's

wrong with TG that it leads to this situation?



I was wondering this myself, I know smolt recently had some major
changes to keep memory usage down. Which TG apps are having this
issue and how often? I know MM uses a lot of memory but, AFAIK, it
was determined that there's not much of a leak if there is one and
that all of that memory is actually used.


Looks like smolt was upgraded just before Thanksgiving so it could be
that we've plugged the leaks we had to deal with that inspired me to
write this. Would it be a good idea to have this in place anyways?
With it periodically checking, we would find out that we had problems
when cron emails us a notice that the script had to restart a process.
Without it, we'll be notified when nagios or a user tells us they're
getting timeouts.


I think its a good idea if for no other reason then allows us to more
actively monitor this stuff, we'll get notified when the app restarts.
+1 from me with the intention that, over time, we get fewer and fewer
restarts.



Cool. I'll check it in and set up a cron job.

One further piece of information since I have output from testing this
on app3 yesterday:


AppName Uptime RSS 11/25 RSS 11/26
mirrormanager 2d4h 714336 962268
packagedb --- restarted 13h ago --
smolt 5d3h 299556 299556
transifex 5d3h 42744 42768

-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

On Sun, Nov 25, 2007 at 01:00:53PM -0800, Toshio Kuratomi wrote:
> Here's a short script to test our TG apps run via supervisor for excessive
> memory usage and restart them if necessary. We could run this via cron in
> alternate hours on each app server. Does this seem like a good or bad idea
> to people?

Probably not a bad idea; I think koji does something similar with
apache. However, I don't think we need this for bodhi, at least for the
moment. The only time bodhi's memory usage jumps is when it's pushing
updates -- so if we were to use this script for bodhi, it would have to check
if it is currently running mash. But for now, I'm not sure that it is necessary
seeing as how most of the time puppetd eats more memory than bodhi.

luke

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

On Mon, Nov 26, 2007 at 09:59:44AM -0800, Toshio Kuratomi wrote:
> Matt Domsch wrote:
>> On Sun, Nov 25, 2007 at 09:08:30PM -0800, Toshio Kuratomi wrote:
>>>> +1, but does it make sure all transactions are finished? I know smolt
>>>> does not have good transaction protection. If a transaction fails
>>>> halfway through, we might have a mess.
>>>>
>>> Not if the app doesn't. From a brief test, TG apps do not do this.
>>
>> MirrorManager doesn't use transactions, I never figured out how to get
>> them to work right. Advice welcome.
>>
> By not being able to get transactions working, do you mean explicit
> transactions or implicit transactions? I see that mirrormanager, bodhi,
> and noc (not running currently) are using a dburi that disables implicit
> transactions::
> mirrormanager-prod.cfg.erb:
> sqlobject.dburi="notrans_postgres://mirroradmin:
> <%= mirrorPassword %>@db2.fedora.phx.redhat.com/mirrormanager"
>
> If that was changed to::
> sqlobject.dburi="postgres://mirroradmin:[...]
>
> TurboGears would at least attempt to use an implicit transaction per http
> request which should protect the database from shutting down the
> application in the middle of processing a multi-table update. I don't know
> if that's the problem you're referring to, though.

Removing the notrans_postgres:// from bodhi's sqlobject.dburi causes
problems. Modifications don't seem to go through; I'm not sure if they
hit the DB or not. I remember encountering this issue early on in bodhi
development, and it was mitigated by calling hub.sync() all over the
place. I have since removed them, and use notrans_postgres, which has
been working fine since day 1 of our production instance. I'm not a
db guru, so I'm not sure which is better or worse. I'll have to
investigate this further.

luke

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Luke Macken wrote:

On Sun, Nov 25, 2007 at 01:00:53PM -0800, Toshio Kuratomi wrote:
Here's a short script to test our TG apps run via supervisor for excessive
memory usage and restart them if necessary. We could run this via cron in
alternate hours on each app server. Does this seem like a good or bad idea
to people?


Probably not a bad idea; I think koji does something similar with
apache. However, I don't think we need this for bodhi, at least for the
moment. The only time bodhi's memory usage jumps is when it's pushing
updates -- so if we were to use this script for bodhi, it would have to check
if it is currently running mash. But for now, I'm not sure that it is necessary
seeing as how most of the time puppetd eats more memory than bodhi.

Sounds good. I've taken both Bodhi and transifex out of the script for
now as neither one is load balanced (the idea being that the apps which
are load balanced should continue to serve requests off the other
instance while one is restarting).


I've also changed it to take a different memory limit for each app. It
currently has some generous guesses as to what the memory limit should
be. I'm running a cron that logs the rss of the apps on app3&4 and will
refine the limits after we have more data.


-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list