FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 06-10-2008, 10:07 PM
"Sean Carolan"
 
Default Apache jserv monitoring?

In our environment we have many legacy application servers running
apache/jserv. There is a web server front end, then a couple of
load-balanced java servers on the backside. One of the problems we
are faced with is hung or stuck jvms. I have looked at the java
process with the ps command, and there are many times when URL(s) do
not respond, yet the java looks healthy, at least from the OS point of
view. The usual cure for this situation is to restart the JVM, then
the URLs come right back up.

Are any of you aware of tools for monitoring apache jserv, either from
localhost or by connecting to port 8008 over the network? I really
want to find out if there is a way to detect a "sick" JVM other than
getting a bunch of down URL alerts on my phone.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-10-2008, 10:16 PM
"nate"
 
Default Apache jserv monitoring?

Sean Carolan wrote:

> Are any of you aware of tools for monitoring apache jserv, either from
> localhost or by connecting to port 8008 over the network? I really
> want to find out if there is a way to detect a "sick" JVM other than
> getting a bunch of down URL alerts on my phone.

How about setting up a cron to monitor it and auto restart if it's not
responding?

wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to
restart jserv)

I'm not familiar with jserv but it looks like it serves a purpose similar
to tomcat, just a much older code base.

I've used similar scripts for tomcat in the past, such as this cron
*/5 * * * * /bin/grep -q "java.lang.OutOfMemoryError: Java heap space"
/logs/tomcat/web/1/run.latest.log 1>/dev/null 2>&1 && export
MYHOST=`hostname -s` && echo "Restarting Tomcat on $MYHOST -
OutOfMemoryError" | mail -s "[$MYHOST] Restarting Tomcat due to
OutOfMemoryError" my@email.address && /etc/init.d/appname restart
1>/dev/null 2>&1

When the app had a memory leak it helped me sleep better at night, as
the apps were restarting about 10x a day until the developers found the
source of the leak. Fortunately the app restarted reliably 99.9% of the
time. Another place I worked at before that had an app that was so poor
that you had to babysit it all the time, the number of things that could
go wrong was really too much work for us to automate it in a reasonable
amount of time.

Ideally the "load balancer" should be performing "health checks" against
the back end nodes and stop sending traffic to them if they are no
longer healthy. I'm not familiar with the load balancing method your
using, only with bigger load balancers like F5 BigIP.

nate

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-10-2008, 10:23 PM
"Vincent Knecht"
 
Default Apache jserv monitoring?

> In our environment we have many legacy application servers running
apache/jserv. There is a web server front end, then a couple of
load-balanced java servers on the backside. One of the problems we are
faced with is hung or stuck jvms. I have looked at the java process with
the ps command, and there are many times when URL(s) do not respond, yet the
java looks healthy, at least from the OS point of view. The usual cure for
this situation is to restart the JVM, then the URLs come right back up.
>
> Are any of you aware of tools for monitoring apache jserv, either from
localhost or by connecting to port 8008 over the network? I really want to
find out if there is a way to detect a "sick" JVM other than getting a bunch
of down URL alerts on my phone.

Hello,

I don't know jserv or about eventual specific jserv support, but Hyperic
might
be part of the answer. I know it can provide metrics about Tomcat and JVMs,
and application/server specific plugins can be written without too much
effort.
It's by far the most powerful/complete monitoring solution I've seen,
providing support out of the box or with plugins for a bunch of services and
applications.
Check
http://support.hyperic.com/confluence/display/hypcomm/HyperForge/#HyperFORGE-pluginforge
for existing plugins.
Perhaps what you want can be done with a JMX plugin ?

Really worth a try anyway...


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 01:19 AM
"Sean Carolan"
 
Default Apache jserv monitoring?

> How about setting up a cron to monitor it and auto restart if it's not
> responding?
>
> wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to
> restart jserv)

I tried pulling up port 8008 in a web browser, but it doesn't work
quite like that. Apache is configured with mod_jserv like this:

<IfModule mod_jserv.c>
ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root
ApjServAction .html /servlets/gnujsp
</IfModule>

Simply pulling up the URL in a web browser always returns an error:

Status: 400 Bad Request
Servlet-Error: Malformed data sent to JServ

> I'm not familiar with jserv but it looks like it serves a purpose similar
> to tomcat, just a much older code base.

Yes, it is very similar to tomcat. Both have a java process, with a
minimum and maximum heap size, etc. Unfortunately I don't know of an
easy way to test the java process to make sure it is healthy, short of
pulling up the front-end URL in a web browser. The problem here is, it
is possible that one URL could be down, but the other ones still
functioning correctly. We don't want to restart java just because of
a false positive on one URL.

> When the app had a memory leak it helped me sleep better at night, as
> the apps were restarting about 10x a day until the developers found the
> source of the leak. Fortunately the app restarted reliably 99.9% of the
> time.

We are not quite having to restart ten times a day, but there actually
is no chance of a code fix in my situation since this is deprecated
code, and the customer is not willing/able to upgrade. This is why we
want to simply auto-restart the JVM when the server gets into this
state.

> Ideally the "load balancer" should be performing "health checks" against
> the back end nodes and stop sending traffic to them if they are no
> longer healthy. I'm not familiar with the load balancing method your
> using, only with bigger load balancers like F5 BigIP.

We have such a load-balancing device but it sits in front of the web
server, not behind (the web servers are load balanced too). There is
an apache plugin that handles the jserv load balancing.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 01:27 AM
"Sean Carolan"
 
Default Apache jserv monitoring?

> Check
> http://support.hyperic.com/confluence/display/hypcomm/HyperForge/#HyperFORGE-pluginforge
> for existing plugins.
> Perhaps what you want can be done with a JMX plugin ?

Hyperic looks interesting, but anytime someone claims "Zero-Touch
Systems Management" I have to raise a skeptical eyebrow. I will take
a look at it and install the demo, but our company has already
invested in a different monitoring system. I was really hoping for
something simple, perhaps a status page, jserv management port, shell
script perhaps...
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 01:51 AM
Jeff
 
Default Apache jserv monitoring?

On Tue, Jun 10, 2008 at 5:07 PM, Sean Carolan <scarolan@gmail.com> wrote:
<snip>
> Are any of you aware of tools for monitoring apache jserv, either from
> localhost or by connecting to port 8008 over the network? I really
> want to find out if there is a way to detect a "sick" JVM other than
> getting a bunch of down URL alerts on my phone.

The Sun JDK provides tools for pulling info from a running JVM.

http://java.sun.com/javase/6/webnotes/trouble/other/tools6-Unix.html

Not jserv (time for an upgrade?) specific, but attaches directly to
the underlying JVM. The one I've played with is jmap. Of course, you
need to be running a recent JVM. I do not proclaim to be an expert
with these tools, but it may be of some use in your situation.

--
Jeff
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 03:14 AM
"nate"
 
Default Apache jserv monitoring?

Sean Carolan wrote:
>> How about setting up a cron to monitor it and auto restart if it's not
>> responding?
>>
>> wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to
>> restart jserv)
>
> I tried pulling up port 8008 in a web browser, but it doesn't work
> quite like that. Apache is configured with mod_jserv like this:
>
> <IfModule mod_jserv.c>
> ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root
> ApjServAction .html /servlets/gnujsp
> </IfModule>
>
> Simply pulling up the URL in a web browser always returns an error:
>
> Status: 400 Bad Request
> Servlet-Error: Malformed data sent to JServ
>

Sounds similar to the mod_jk connector in apache to connect to
tomcat. When I had to deal with this I setup a dedicated apache
instance on each system running tomcat whose sole purpose for
existence was for testing that connector.

So say setup an apache instance on another port, and have it
direct all traffic back to mod_jserv, then hit the apache
instance with http. It's not perfect but at least for me apache
was a lot more stable than tomcat especially in such a basic
configuration, so it worked well as a way to test the health
of the mod_jk connector.

nate

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 03:43 PM
"Sean Carolan"
 
Default Apache jserv monitoring?

> Sounds similar to the mod_jk connector in apache to connect to
> tomcat. When I had to deal with this I setup a dedicated apache
> instance on each system running tomcat whose sole purpose for
> existence was for testing that connector.
>
> So say setup an apache instance on another port, and have it
> direct all traffic back to mod_jserv, then hit the apache
> instance with http. It's not perfect but at least for me apache
> was a lot more stable than tomcat especially in such a basic
> configuration, so it worked well as a way to test the health
> of the mod_jk connector.

Excellent idea, Nate. I already have apache installed on all these
servers, so it should be fairly trivial to set up a local test site
just for this purpose. I'm going to test this out in our lab
environment today and see how it performs.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 04:10 PM
Ray Leventhal
 
Default Apache jserv monitoring?

Sean Carolan wrote:

In our environment we have many legacy application servers running
apache/jserv. There is a web server front end, then a couple of
load-balanced java servers on the backside. One of the problems we
are faced with is hung or stuck jvms. I have looked at the java
process with the ps command, and there are many times when URL(s) do
not respond, yet the java looks healthy, at least from the OS point of
view. The usual cure for this situation is to restart the JVM, then
the URLs come right back up.

Are any of you aware of tools for monitoring apache jserv, either from
localhost or by connecting to port 8008 over the network? I really
want to find out if there is a way to detect a "sick" JVM other than
getting a bunch of down URL alerts on my phone.

Being far from an expert, if restarting a service or script is what's
needed, you might find SIM[1] helpful


YMMV,
-R

[1]http://rfxnetworks.com/sim.php
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 06-11-2008, 04:27 PM
Les Mikesell
 
Default Apache jserv monitoring?

Sean Carolan wrote:

Sounds similar to the mod_jk connector in apache to connect to
tomcat. When I had to deal with this I setup a dedicated apache
instance on each system running tomcat whose sole purpose for
existence was for testing that connector.

So say setup an apache instance on another port, and have it
direct all traffic back to mod_jserv, then hit the apache
instance with http. It's not perfect but at least for me apache
was a lot more stable than tomcat especially in such a basic
configuration, so it worked well as a way to test the health
of the mod_jk connector.


Excellent idea, Nate. I already have apache installed on all these
servers, so it should be fairly trivial to set up a local test site
just for this purpose. I'm going to test this out in our lab
environment today and see how it performs.


If this is old enough to be running a Sun 1.4.x JVM and the web server
generates images, there was a bug that caused the JVM itself to leak
memory under certain circumstances. I've forgotten the details but it
was something like an unreleased file reference per image.


--
Les Mikesell
lesmikesell@gmail.com


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 04:13 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org