FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 10-28-2008, 08:30 PM
"David Miller"
 
Default Possibly OT: intermittently long response times when connecting to host

I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and
drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it -
2 hosts. The problem is that when I pull up a web page after not
having accessed it for ~15+ minutes it takes between 10-12s to load
fully. The 2nd page, 3rd page - and sometimes more than that - take
between 8-12 second. After that - decent performance, though not great
(0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes
back to slow. While it's been most apparent with httpd, I've noticed
similar slowness with the initial ssh session, etc.

I know what you're thinking. DNS problem, right? Yeah, me too. But for
the life of me I can't find a problem. This is at a colocation site
where we have other servers - all of those are just fine. We've got an
almost identical box internally - working just fine. I can do an
nslookup off of each of our different internal name servers and get
identical and immediate responses.

I've done tcpdumps (both on the client and the server). It shows the
request gets to the server immediately. I've started httpd using
strace and not seen anything glaringly wrong (though I'm not well
versed in the output I have enough C under my belt to have an idea
what's going on). Thinking it was code that the developer working on
the project I pulled all the custom drupal modules out and it's still
slow. Even created another virtual host with a stock drupal install
and it's slow (identical results). I moved the mysql server from local
to our central DB server. Load avg never gets above ~0.10 so I don't
think the box is stressed. Watching the ESX load average it looks good
too. I've tried a hell of a lot of other things and tried to include
all the relevant information so apologies in advance for anything I
might have left out.


Thanks,
Dave

Relevant info:

uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP
Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux

relevant portion of httpd.conf:
NameVirtualHost *:80

<VirtualHost *:80>
DocumentRoot /path/to/drupal-current
ServerName publicname.domainname.com
ErrorLog /path/to/logserror_log
CustomLog /path/to/logs/access_log common
</VirtualHost>

/etc/hosts:
127.0.0.1 localhost.localdomain localhost internal-name
12.34.56.78 publicname.domainname.com
12.34.56.79 db-server.domainname.com db-server
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-28-2008, 08:36 PM
Craig White
 
Default Possibly OT: intermittently long response times when connecting to host

On Tue, 2008-10-28 at 16:30 -0500, David Miller wrote:
> I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and
> drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it -
> 2 hosts. The problem is that when I pull up a web page after not
> having accessed it for ~15+ minutes it takes between 10-12s to load
> fully. The 2nd page, 3rd page - and sometimes more than that - take
> between 8-12 second. After that - decent performance, though not great
> (0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes
> back to slow. While it's been most apparent with httpd, I've noticed
> similar slowness with the initial ssh session, etc.
>
> I know what you're thinking. DNS problem, right? Yeah, me too. But for
> the life of me I can't find a problem. This is at a colocation site
> where we have other servers - all of those are just fine. We've got an
> almost identical box internally - working just fine. I can do an
> nslookup off of each of our different internal name servers and get
> identical and immediate responses.
>
> I've done tcpdumps (both on the client and the server). It shows the
> request gets to the server immediately. I've started httpd using
> strace and not seen anything glaringly wrong (though I'm not well
> versed in the output I have enough C under my belt to have an idea
> what's going on). Thinking it was code that the developer working on
> the project I pulled all the custom drupal modules out and it's still
> slow. Even created another virtual host with a stock drupal install
> and it's slow (identical results). I moved the mysql server from local
> to our central DB server. Load avg never gets above ~0.10 so I don't
> think the box is stressed. Watching the ESX load average it looks good
> too. I've tried a hell of a lot of other things and tried to include
> all the relevant information so apologies in advance for anything I
> might have left out.
>
>
> Thanks,
> Dave
>
> Relevant info:
>
> uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP
> Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux
>
> relevant portion of httpd.conf:
> NameVirtualHost *:80
>
> <VirtualHost *:80>
> DocumentRoot /path/to/drupal-current
> ServerName publicname.domainname.com
> ErrorLog /path/to/logserror_log
> CustomLog /path/to/logs/access_log common
> </VirtualHost>
>
> /etc/hosts:
> 127.0.0.1 localhost.localdomain localhost internal-name
> 12.34.56.78 publicname.domainname.com
> 12.34.56.79 db-server.domainname.com db-server
----
sounds sleepy - perhaps the mysql connections drop after inactivity and
then it takes that long to connect

You might just want to set up a cron script that does a curl or wget of
a standard page every 2 minutes or so just to keep things active.

Craig

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-28-2008, 09:02 PM
"David Miller"
 
Default Possibly OT: intermittently long response times when connecting to host

--snip--
> ----
> sounds sleepy - perhaps the mysql connections drop after inactivity and
> then it takes that long to connect
>
> You might just want to set up a cron script that does a curl or wget of
> a standard page every 2 minutes or so just to keep things active.
>
> Craig
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

hmmm.... you may be onto something there, Craig. Since it was the same
environment (we set up lots of LAMP servers and use a common MySQL
backend) I gave it the benefit of the doubt when I was initially
troubleshooting (the connection was there and everything looked sane).
I'll take another look at that.

Thanks.
Dave
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-29-2008, 01:13 PM
Johnny Hughes
 
Default Possibly OT: intermittently long response times when connecting to host

David Miller wrote:
> --snip--
>> ----
>> sounds sleepy - perhaps the mysql connections drop after inactivity and
>> then it takes that long to connect
>>
>> You might just want to set up a cron script that does a curl or wget of
>> a standard page every 2 minutes or so just to keep things active.
>>
>> Craig
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
> hmmm.... you may be onto something there, Craig. Since it was the same
> environment (we set up lots of LAMP servers and use a common MySQL
> backend) I gave it the benefit of the doubt when I was initially
> troubleshooting (the connection was there and everything looked sane).
> I'll take another look at that.

Also, take a good look at the logs /var/log/messages and dmesg for hard
drive errors. Lots of times a bad drive will cause controller bus
resets that happen intermittently ... that can cause a 5-10 second delay
and then things seem to work fine till the next time.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-29-2008, 01:18 PM
Johnny Hughes
 
Default Possibly OT: intermittently long response times when connecting to host

David Miller wrote:
> --snip--
>> ----
>> sounds sleepy - perhaps the mysql connections drop after inactivity and
>> then it takes that long to connect
>>
>> You might just want to set up a cron script that does a curl or wget of
>> a standard page every 2 minutes or so just to keep things active.
>>
>> Craig
>>
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
> hmmm.... you may be onto something there, Craig. Since it was the same
> environment (we set up lots of LAMP servers and use a common MySQL
> backend) I gave it the benefit of the doubt when I was initially
> troubleshooting (the connection was there and everything looked sane).
> I'll take another look at that.

Also, make sure that this is not set in apache (the default is Off ...
but things like that can take massive amounts of time):

HostnameLookups On

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-29-2008, 03:29 PM
"David Miller"
 
Default Possibly OT: intermittently long response times when connecting to host

--snip--
>
> Also, take a good look at the logs /var/log/messages and dmesg for hard
> drive errors. Lots of times a bad drive will cause controller bus
> resets that happen intermittently ... that can cause a 5-10 second delay
> and then things seem to work fine till the next time.
>
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

I should have mentioned that.... /var/log/messages, /var/log/secure,
/var/log/dmesg, /var/log/audit/audit.log.... well, any log I could
think of - none show anything out of the ordinary.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-29-2008, 03:30 PM
"David Miller"
 
Default Possibly OT: intermittently long response times when connecting to host

>> --snip--
>
> Also, make sure that this is not set in apache (the default is Off ...
> but things like that can take massive amounts of time):
>
> HostnameLookups On
>
>

Yup - HostnameLookups is off. Other than adding the virtual hosts I
haven't altered the httpd.conf file.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 10-29-2008, 04:56 PM
"David Miller"
 
Default Possibly OT: intermittently long response times when connecting to host

--snip--
> sounds sleepy - perhaps the mysql connections drop after inactivity and
> then it takes that long to connect
>
> You might just want to set up a cron script that does a curl or wget of
> a standard page every 2 minutes or so just to keep things active.
>
> Craig
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

The server hosting the web page looks to be creating the connecting
immediately to the MySQL server (verified by mytop and tcpdump).
Troubleshooting MySQL connections isn't my forte - to now the
connections have always "just worked" so there hasn't been much of a
need. Another thing that has led me away from thinking database
problem is that this existed when the database was hosted locally and
when it was on our production database server.

Sorry for the delay in posting. 2-3 attempts before the problem
"resolves" itself makes troubleshooting slow.

Thanks to all that have replied.

Dave
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 07:15 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org