FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 11-06-2008, 01:33 PM
"Neil Aggarwal"
 
Default iptables starts blocking outbound http traffic

Hello:

I have a machine running CentOS 5 x86_64.

It is running apache httpd and tomcat.

For some reason, after running for a few days,
web requests stop responding. It happened again
this morning. I check the syslog and see a HUGE
number of logs like this:

OUTPUT IN= OUT=eth0 SRC=[MyIP] DST=[OutsideIP] LEN=532 TOS=0x00 PREC=0x00
TTL=64 ID=52669 DF PROTO=TCP SPT=80 DPT=54697 WINDOW=61 RES=0x00 ACK PSH FIN
URGP=0

Here are my iptables commands for http connections (I have the default
policy set to drop):

# Allow http connections from the outside world
/sbin/iptables -A INPUT -i eth0 -d $ETH0_IP -p tcp --sport 1024: --dport
http -m state --state NEW,ESTABLISHED -j ACCEPT
/sbin/iptables -A OUTPUT -o eth0 -s $ETH0_IP -p tcp --sport http --dport
1024: -m state --state ESTABLISHED -j ACCEPT

Here are some strange things:
1. I have the exact same rules running on two other servers which do
not give me any trouble.
2. If I stop and restart httpd and tomcat, the problem goes away. This
suggests the firewall is not a problem.

Any ideas what is going on?

Thanks,
Neil

--
Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com
Eliminate junk email and reclaim your inbox.
Visit http://www.spammilter.com for details.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-06-2008, 02:07 PM
"Filipe Brandenburger"
 
Default iptables starts blocking outbound http traffic

Hi,

On Thu, Nov 6, 2008 at 09:33, Neil Aggarwal <neil@jammconsulting.com> wrote:
> # Allow http connections from the outside world
> /sbin/iptables -A INPUT -i eth0 -d $ETH0_IP -p tcp --sport 1024: --dport
> http -m state --state NEW,ESTABLISHED -j ACCEPT
> /sbin/iptables -A OUTPUT -o eth0 -s $ETH0_IP -p tcp --sport http --dport
> 1024: -m state --state ESTABLISHED -j ACCEPT
>
> Any ideas what is going on?

If you're using ESTABLISHED, it depends on ip_conntrack being able to
track the connections. ip_conntrack keeps a table of all connections,
but this table is limited in size, so it may be overflowing.

You can see how many entries you have in that table at any moment with
this command:
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count

And you can see what the maximum is set with this command:
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max

The default in CentOS 5 is 16k connections.

IIRC, you can increase that dinamically with echo ... >/proc/sys/...
or with sysctl. Also, I believe you can set the default after a reboot
in /etc/sysctl.conf. I think it's also possible to do that in
/etc/modprobe.conf, but I'm not sure what the syntax is anymore, and
modinfo ip_conntrack didn't give me any clues. Google should help with
that.

HTH,
Filipe
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-06-2008, 02:42 PM
"Neil Aggarwal"
 
Default iptables starts blocking outbound http traffic

Filipe:

Thanks for the information.

If I do:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
on each of my servers, they all report 65536 which
seems like a pretty high limit.

If I do:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
on each of my servers, the highest number is just over
1100.

If this is the source of the problem, how would restarting
httpd and tomcat help? I did not restart the machine nor
reset iptables.

I am not asking this to be argumentative, just trying to
understand how the facts I am seeing are related.

Thanks,
Neil

--
Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com
Eliminate junk email and reclaim your inbox.
Visit http://www.spammilter.com for details.

> If you're using ESTABLISHED, it depends on ip_conntrack being able to
> track the connections. ip_conntrack keeps a table of all connections,
> but this table is limited in size, so it may be overflowing.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-06-2008, 03:04 PM
"Filipe Brandenburger"
 
Default iptables starts blocking outbound http traffic

Hi,

On Thu, Nov 6, 2008 at 10:42, Neil Aggarwal <neil@jammconsulting.com> wrote:
> If this is the source of the problem, how would restarting
> httpd and tomcat help? I did not restart the machine nor
> reset iptables.

Because this might potentially close several connections and free
slots in the conntrack table.

You are right that your conntrack table size is high enough and this
should not be happening. It might be an attack, a synflood or
something, that is causing this problem to happen. In that case, the
semi-opened connections will be kept on the table, but as the other
side will not complete the handshake, they will only be removed from
the table after a timeout. I also think that when you stop Apache,
there will be no process listening on port 80 anymore, and then
conntrack may get rid of those semi-opened connections since the other
side is not listening anymore. A lot of especulation here, but it
might be what is affecting you.

In any case, next time you have this same problem, considering looking
at the counters to see if _count is reaching _max, that would confirm
the hypothesis.

If that is indeed the case, you can dump the conntrack information
with this command:
# cat /proc/net/ip_conntrack

You can do that and save it to another file, restart Apache and do the
same, so that you can see what is really happening there. This might
give you a better idea of why it's happening.

If conntrack is really overflowing, you may consider increasing the
table size, but this will mean more memory usage on your server.

Alternatively you might choose to redo your firewall rules to be
stateless, by removing --state NEW and --state ESTABLISHED, and by
adding ! --syn on the ones you want to allow for established
connections only. It's not going to be as perfect as actually tracking
the connections, but for protocols like HTTP is a good enough
compromise.

But your problem is probably being caused by something else, like an
attack, so probably the best way to deal with it is to find out what
is causing it and try to take measures to correct that problem
instead.

> I am not asking this to be argumentative, just trying to
> understand how the facts I am seeing are related.

No problem! Didn't sound argumentative to me in any way.

Let us know how that goes, and if you get more cluse, let us know if
you need more help in fixing the root problem.

HTH,
Filipe
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-06-2008, 04:31 PM
Kai Schaetzl
 
Default iptables starts blocking outbound http traffic

Neil Aggarwal wrote on Thu, 6 Nov 2008 08:33:59 -0600:

> /sbin/iptables -A OUTPUT -o eth0 -s $ETH0_IP -p tcp --sport http --dport
> 1024: -m state --state ESTABLISHED -j ACCEPT

Why do you try to filter outbound connections at all? If "something" makes
it on your machine the first thing they will do is drop your rules.

Kai

--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-06-2008, 05:16 PM
"Joseph L. Casale"
 
Default iptables starts blocking outbound http traffic

>Why do you try to filter outbound connections at all? If "something" makes
>it on your machine the first thing they will do is drop your rules.

You imply the *only* reason for outbound filtering is stop a hacker. In some
environments it serves as an additional layer of protection against other problems
related to configuration/application issues as an example.

jlc

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-11-2008, 04:23 PM
"Neil Aggarwal"
 
Default iptables starts blocking outbound http traffic

Filipe:

I changed the firewall rules on the server that had stopped
responding to not use ESTABLISHED.

Now, one of the servers that was still using ESTABLISHED
stopped responding.

I am seeing logs like this in the syslog:

OUTPUT IN= OUT=eth0 SRC=[myIP] DST=[otherIP] LEN=52 TOS=0x00 PREC=0x00
TTL=64 ID=35076 DF PROTO=TCP SPT=80 DPT=36953 WINDOW=54 RES=0x00 ACK PSH FIN
URGP=0

I did:
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_count
and it gave me: 615

That seems like the conntrack is not overflowing, but the firewall
was blocking the outbound traffic.

I updated all my servers to not use ESTABLISHED, but I am still
baffled on how this could occur.

Any other ideas?

Thanks,
Neil

--
Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com
Eliminate junk email and reclaim your inbox.
Visit http://www.spammilter.com for details.

> You are right that your conntrack table size is high enough and this
> should not be happening. It might be an attack, a synflood or
> something, that is causing this problem to happen. In that case, the
> semi-opened connections will be kept on the table, but as the other
> side will not complete the handshake, they will only be removed from
> the table after a timeout. I also think that when you stop Apache,
> there will be no process listening on port 80 anymore, and then
> conntrack may get rid of those semi-opened connections since the other
> side is not listening anymore. A lot of especulation here, but it
> might be what is affecting you.
>
> In any case, next time you have this same problem, considering looking
> at the counters to see if _count is reaching _max, that would confirm
> the hypothesis.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-12-2008, 04:44 PM
"Neil Aggarwal"
 
Default iptables starts blocking outbound http traffic

Filipe:

One of my servers stopped responding again.
This time, it was one of those which was not using
ESTABLISHED.

I am now convinced the problem is not in the firewall.
It must be somewhere in Apache, Tomcat, or my
application code (Most likely). I think I
was seeing the firewall logs after I restarted
Apache since the responses were rejected since they
no longer were attached to an established connection.

Sorry for the red herring.

Neil

--
Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com
Eliminate junk email and reclaim your inbox.
Visit http://www.spammilter.com for details.

> I am seeing logs like this in the syslog:
>
> OUTPUT IN= OUT=eth0 SRC=[myIP] DST=[otherIP] LEN=52 TOS=0x00 PREC=0x00
> TTL=64 ID=35076 DF PROTO=TCP SPT=80 DPT=36953 WINDOW=54
> RES=0x00 ACK PSH FIN
> URGP=0

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-12-2008, 05:09 PM
"Filipe Brandenburger"
 
Default iptables starts blocking outbound http traffic

Hi,

On Wed, Nov 12, 2008 at 12:44, Neil Aggarwal <neil@jammconsulting.com> wrote:
> Sorry for the red herring.

No problem.

> I am now convinced the problem is not in the firewall.
> It must be somewhere in Apache, Tomcat, or my
> application code (Most likely). I think I
> was seeing the firewall logs after I restarted
> Apache since the responses were rejected since they
> no longer were attached to an established connection.

Look into the number of busy httpd servers, that might be your problem
(and why it's not accepting any new connections). To do that you can
use a configured URL in Apache (I believe it is /server-status) or you
can at least estimate using "ps" and comparing with the settings for
maximum number of servers in your httpd.conf.

HTH,
Filipe
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 11-12-2008, 05:09 PM
"Filipe Brandenburger"
 
Default iptables starts blocking outbound http traffic

Hi,

On Wed, Nov 12, 2008 at 12:44, Neil Aggarwal <neil@jammconsulting.com> wrote:
> Sorry for the red herring.

No problem.

> I am now convinced the problem is not in the firewall.
> It must be somewhere in Apache, Tomcat, or my
> application code (Most likely). I think I
> was seeing the firewall logs after I restarted
> Apache since the responses were rejected since they
> no longer were attached to an established connection.

Look into the number of busy httpd servers, that might be your problem
(and why it's not accepting any new connections). To do that you can
use a configured URL in Apache (I believe it is /server-status) or you
can at least estimate using "ps" and comparing with the settings for
maximum number of servers in your httpd.conf.

HTH,
Filipe
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 10:07 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org