Restart TG apps for high mem-usage
 

Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary. We could run this
via cron in alternate hours on each app server. Does this seem like a
good or bad idea to people?


-Tosjio
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

sounds sane to me

On Nov 25, 2007 9:00 PM, Toshio Kuratomi <a.badger@gmail.com> wrote:
> Here's a short script to test our TG apps run via supervisor for
> excessive memory usage and restart them if necessary. We could run this
> via cron in alternate hours on each app server. Does this seem like a
> good or bad idea to people?
>
> -Tosjio
>
> _______________________________________________
> Fedora-infrastructure-list mailing list
> Fedora-infrastructure-list@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
>
>

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

On Nov 25, 2007 4:00 PM, Toshio Kuratomi <a.badger@gmail.com> wrote:
> Here's a short script to test our TG apps run via supervisor for
> excessive memory usage and restart them if necessary. We could run this
> via cron in alternate hours on each app server. Does this seem like a
> good or bad idea to people?
>
> -Tosjio

+1, but does it make sure all transactions are finished? I know smolt
does not have good transaction protection. If a transaction fails
halfway through, we might have a mess.

-Yaakov

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Yaakov Nemoy wrote:

On Nov 25, 2007 4:00 PM, Toshio Kuratomi <a.badger@gmail.com> wrote:

Here's a short script to test our TG apps run via supervisor for
excessive memory usage and restart them if necessary. We could run this
via cron in alternate hours on each app server. Does this seem like a
good or bad idea to people?

-Tosjio


+1, but does it make sure all transactions are finished? I know smolt
does not have good transaction protection. If a transaction fails
halfway through, we might have a mess.


Not if the app doesn't. From a brief test, TG apps do not do this.

The script is asking supervisor to shutdown the application. supervisor
sends a TERM to the TG app (we can configure it to send something other
than TERM if we want but I don't see any documentation that leads me to
believe it will be different with a HUP or QUIT). At that point it
looks like a TG app will immediately shutdown and rollback any current
transactions.


smolt is on shaky ground if it's not using transactions correctly... At
the beginning of the month when smolt was getting hit hard we did pretty
much this same thing except manually instead of via a script when we
noticed that smolt was giving timeouts and taking up 1G+ of RAM. I
think the current smolt code is using SQLAlchemy, correct? It's pretty
easy to use transactions so that you don't leave the db in an
inconsistent state with that configuration. Using the session's
implicit transaction flushed just before the return should do the safe
thing. You can look through the code later and find additional places
where you can safely flush the transaction if there's a need.


-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

On Nov 26, 2007 12:08 AM, Toshio Kuratomi <a.badger@gmail.com> wrote:
> Yaakov Nemoy wrote:
> > On Nov 25, 2007 4:00 PM, Toshio Kuratomi <a.badger@gmail.com> wrote:
> Not if the app doesn't. From a brief test, TG apps do not do this.
>
> The script is asking supervisor to shutdown the application. supervisor
> sends a TERM to the TG app (we can configure it to send something other
> than TERM if we want but I don't see any documentation that leads me to
> believe it will be different with a HUP or QUIT). At that point it
> looks like a TG app will immediately shutdown and rollback any current
> transactions.

It's got my vote then

> smolt is on shaky ground if it's not using transactions correctly... At
> the beginning of the month when smolt was getting hit hard we did pretty
> much this same thing except manually instead of via a script when we
> noticed that smolt was giving timeouts and taking up 1G+ of RAM. I
> think the current smolt code is using SQLAlchemy, correct? It's pretty
> easy to use transactions so that you don't leave the db in an
> inconsistent state with that configuration. Using the session's
> implicit transaction flushed just before the return should do the safe
> thing. You can look through the code later and find additional places
> where you can safely flush the transaction if there's a need.

We do use transactions where we can, but since most of the code is not
tested at all, let alone stress tested, i can't vouch for it doing The
Right Thing.

(Winter break is only a few weeks away)

-Yaakov

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

On Sun, Nov 25, 2007 at 09:08:30PM -0800, Toshio Kuratomi wrote:
> >+1, but does it make sure all transactions are finished? I know smolt
> >does not have good transaction protection. If a transaction fails
> >halfway through, we might have a mess.
> >
> Not if the app doesn't. From a brief test, TG apps do not do this.

MirrorManager doesn't use transactions, I never figured out how to get
them to work right. Advice welcome.

--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Matt Domsch wrote:

On Sun, Nov 25, 2007 at 09:08:30PM -0800, Toshio Kuratomi wrote:

+1, but does it make sure all transactions are finished? I know smolt
does not have good transaction protection. If a transaction fails
halfway through, we might have a mess.


Not if the app doesn't. From a brief test, TG apps do not do this.


MirrorManager doesn't use transactions, I never figured out how to get
them to work right. Advice welcome.

By not being able to get transactions working, do you mean explicit
transactions or implicit transactions? I see that mirrormanager, bodhi,
and noc (not running currently) are using a dburi that disables
implicit transactions::

mirrormanager-prod.cfg.erb:
sqlobject.dburi="notrans_postgres://mirroradmin:
<%= mirrorPassword %>@db2.fedora.phx.redhat.com/mirrormanager"

If that was changed to::
sqlobject.dburi="postgres://mirroradmin:[...]

TurboGears would at least attempt to use an implicit transaction per
http request which should protect the database from shutting down the
application in the middle of processing a multi-table update. I don't
know if that's the problem you're referring to, though.


-Toshio

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Toshio Kuratomi (a.badger@gmail.com) said:
> Here's a short script to test our TG apps run via supervisor for excessive
> memory usage and restart them if necessary. We could run this via cron in
> alternate hours on each app server. Does this seem like a good or bad idea
> to people?

It's a good idea if it's needed, but it's a bad idea that it is needed. What's
wrong with TG that it leads to this situation?

Bill

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

Restart TG apps for high mem-usage
 

Bill Nottingham wrote:
Toshio Kuratomi (a.badger@gmail.com) said:

Here's a short script to test our TG apps run via supervisor for excessive
memory usage and restart them if necessary. We could run this via cron in
alternate hours on each app server. Does this seem like a good or bad idea
to people?



It's a good idea if it's needed, but it's a bad idea that it is needed. What's
wrong with TG that it leads to this situation?



I was wondering this myself, I know smolt recently had some major
changes to keep memory usage down. Which TG apps are having this issue
and how often? I know MM uses a lot of memory but, AFAIK, it was
determined that there's not much of a leak if there is one and that all
of that memory is actually used.


-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list