Too many sockets in SYN_RECV
 

Hello Debian-isp,

On almost idle server (12 GB RAM, 2x 6-core xeon, utilized to about
15-20%) I am running nginx serving one small php file over php5-fpm.
It is ad tracking site, so it has many connections per second from all
over the world.

During the peak hours (about 1500 conns/s) the response time goes high
- I want to get the nginx status page and I often get connection
timeout. I guess it is because there are too many connections waiting
in SYN_RECV state (it tops at 512 - is it possible to increase it).
But the server is almost idle, stracing nginx shows that it is
normally processing incoming connections. I tried increasing the
number of nginx worker processes, but it didn't help.
As I have learned, connection is ready for accept() by nginx only
after it is ESTABLISHED, so these SYN_RECV must be waiting in the
kernel for some reason...

Debian squeeze, 2.6.32-5-amd64, php & nginx from dotdeb

bill3:/home:# munin-run netstat_tcpstates
TIME_WAIT.value 38369
CLOSE_WAIT.value 0
SYN_SENT.value 0
FIN_WAIT1.value 364
CLOSE.value 0
ESTABLISHED.value 2263
FIN_WAIT2.value 3895
SYN_RECV.value 463
CLOSING.value 14
LAST_ACK.value 53

bill3:/home:# ss -nlt
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 511 xx.xx.xx.xx:80 *:*

bill3:/home:# netstat -ts
Tcp:
113348 active connections openings
1840559456 passive connection openings
2363047 failed connection attempts
11536549 connection resets received
2234 connections established
2692177651 segments received
2990428444 segments send out
53602607 segments retransmited
151889 bad segments received.
261748080 resets sent
TcpExt:
2360438 resets received for embryonic SYN_RECV sockets
3 packets pruned from receive queue because of socket buffer overrun
643 ICMP packets dropped because they were out-of-window
48 ICMP packets dropped because socket was locked
914787009 TCP sockets finished time wait in fast timer
164848867 TCP sockets finished time wait in slow timer
86861297 passive connections rejected because of time stamp
388715 packets rejects in established connections because of timestamp
1112821217 delayed acks sent
615599 delayed acks further delayed because of locked socket
Quick ack mode was activated 12384186 times
310610 times the listen queue of a socket overflowed
310610 SYNs to LISTEN sockets dropped
387813160 packets directly queued to recvmsg prequeue.
5771 bytes directly in process context from backlog
7711423 bytes directly received in process context from prequeue
977281196 packet headers predicted
311065 packets header predicted and directly queued to user
739893500 acknowledgments not containing data payload received
697503290 predicted acknowledgments
1 times recovered from packet loss due to fast retransmit
19583 times recovered from packet loss by selective acknowledgements
2380 bad SACK blocks received
Detected reordering 9 times using FACK
189 congestion windows fully recovered without slow start
2 congestion windows partially recovered using Hoe heuristic
2688626 congestion windows recovered without slow start by DSACK
5521689 congestion windows recovered without slow start after partial ack
123 TCP data loss events
TCPLostRetransmit: 2
4272 timeouts after reno fast retransmit
2405072 timeouts after SACK recovery
124467 timeouts in loss state
19632 fast retransmits
38 forward retransmits
2137269 retransmits in slow start
33938539 other TCP timeouts
9706 SACK retransmits failed
202 packets collapsed in receive queue due to low socket buffer
15999828 DSACKs sent for old packets
13914 DSACKs sent for out of order packets
11364774 DSACKs received
77 DSACKs for out of order packets received
5 connections reset due to unexpected SYN
32975997 connections reset due to unexpected data
120171 connections reset due to early user close
2272680 connections aborted due to timeout
TCPSACKDiscard: 40431
TCPDSACKIgnoredOld: 2049268
TCPDSACKIgnoredNoUndo: 1429462
TCPSackShifted: 156
TCPSackMerged: 315
TCPSackShiftFallback: 6461253




--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 1315535110.20111206103003@marki-online.net">http://lists.debian.org/1315535110.20111206103003@marki-online.net

Too many sockets in SYN_RECV
 

On 12/06/2011 05:30 PM, Marek Podmaka wrote:
> Hello Debian-isp,
>
> On almost idle server (12 GB RAM, 2x 6-core xeon, utilized to about
> 15-20%) I am running nginx serving one small php file over php5-fpm.
> It is ad tracking site, so it has many connections per second from all
> over the world.
>
> During the peak hours (about 1500 conns/s) the response time goes high
> - I want to get the nginx status page and I often get connection
> timeout. I guess it is because there are too many connections waiting
> in SYN_RECV state (it tops at 512 - is it possible to increase it).
> But the server is almost idle, stracing nginx shows that it is
> normally processing incoming connections. I tried increasing the
> number of nginx worker processes, but it didn't help.
> As I have learned, connection is ready for accept() by nginx only
> after it is ESTABLISHED, so these SYN_RECV must be waiting in the
> kernel for some reason...
>
> Debian squeeze, 2.6.32-5-amd64, php & nginx from dotdeb

Have you tried using syn cookies? That normally helps a lot when you
have a great amount of connections. I'm not sure if the Debian kernel
uses it by default though (you might need to rebuild the kernel???).

Thomas


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4EDDF3D7.1020001@debian.org">http://lists.debian.org/4EDDF3D7.1020001@debian.org

Too many sockets in SYN_RECV
 

Hello,

Tuesday, December 6, 2011, 11:52:07, Thomas Goirand wrote:

> Have you tried using syn cookies? That normally helps a lot when you
> have a great amount of connections. I'm not sure if the Debian kernel
> uses it by default though (you might need to rebuild the kernel???).

I have just enabled them (they are compiled in debian kernel, but
disabled by default) and it seems to help.
Can you explain why syncookies do help also in my case of many
connections (without any DDoS attack)? Or the other way - why the
kernel can't handle the new connections without syncookies?

Now the netstat -ts shows this:
55980975 SYN cookies sent
58244121 SYN cookies received
18497897 invalid SYN cookies received
2518647 resets received for embryonic SYN_RECV sockets
3 packets pruned from receive queue because of socket buffer overrun
766 ICMP packets dropped because they were out-of-window
67 ICMP packets dropped because socket was locked
976666560 TCP sockets finished time wait in fast timer
168877183 TCP sockets finished time wait in slow timer
89850676 passive connections rejected because of time stamp
424051 packets rejects in established connections because of timestamp
1164228384 delayed acks sent
648215 delayed acks further delayed because of locked socket
Quick ack mode was activated 13416055 times
4049190 times the listen queue of a socket overflowed
4049190 SYNs to LISTEN sockets dropped
1042838538 packets directly queued to recvmsg prequeue.
5771 bytes directly in process context from backlog
9404314 bytes directly received in process context from prequeue
2120925297 packet headers predicted
322307 packets header predicted and directly queued to user
1124450655 acknowledgments not containing data payload received
1805039253 predicted acknowledgments
1 times recovered from packet loss due to fast retransmit
20223 times recovered from packet loss by selective acknowledgements
2547 bad SACK blocks received
Detected reordering 9 times using FACK
199 congestion windows fully recovered without slow start
2 congestion windows partially recovered using Hoe heuristic
2752149 congestion windows recovered without slow start by DSACK
5702588 congestion windows recovered without slow start after partial ack
130 TCP data loss events
TCPLostRetransmit: 2
6085 timeouts after reno fast retransmit
2529709 timeouts after SACK recovery
134114 timeouts in loss state
20274 fast retransmits
38 forward retransmits
2307669 retransmits in slow start
36174127 other TCP timeouts
10001 SACK retransmits failed
202 packets collapsed in receive queue due to low socket buffer
16775872 DSACKs sent for old packets
14552 DSACKs sent for out of order packets
11816825 DSACKs received
80 DSACKs for out of order packets received
5 connections reset due to unexpected SYN
38344636 connections reset due to unexpected data
245406 connections reset due to early user close
2455128 connections aborted due to timeout
TCPSACKDiscard: 43862
TCPDSACKIgnoredOld: 2118496
TCPDSACKIgnoredNoUndo: 1514802
TCPSackShifted: 156
TCPSackMerged: 328
TCPSackShiftFallback: 6766189



--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 908830125.20111207113035@marki-online.net">http://lists.debian.org/908830125.20111207113035@marki-online.net

Too many sockets in SYN_RECV
 

On 12/07/2011 06:30 PM, Marek Podmaka wrote:
> Hello,
>
> Tuesday, December 6, 2011, 11:52:07, Thomas Goirand wrote:
>
>> Have you tried using syn cookies? That normally helps a lot when you
>> have a great amount of connections. I'm not sure if the Debian kernel
>> uses it by default though (you might need to rebuild the kernel???).
>
> I have just enabled them (they are compiled in debian kernel, but
> disabled by default) and it seems to help.
> Can you explain why syncookies do help also in my case of many
> connections (without any DDoS attack)? Or the other way - why the
> kernel can't handle the new connections without syncookies?

Ok, I'll explain.

Normally, when you receive a TCP connection, you'd receive a SYN
request. Your server then would reply with a SYN_ACK (which, obviously,
is both a SYN and an ACK). The TCP connection is really established when
you receive a last ACK form the originator of the first SYN packet
you've received at the beginning of the negotiation.

All these exchanges are there to make sure that data can be sent both
ways before declaring connectivity. The only issue is that between the
first SYN and the last ACK, some resources are being allocated (eg:
memory segments, or storage space in a table, or any kernel internals
which I don't really know about: I didn't have a look into the code).

So, to avoid resource exhaustion when you receive so many SYN, instead
of allocating some resources, in your SYN_ACK you send a cookie which is
a kind of cryptography magic number that your server will be able to
recognize when it will receive the ACK. This way, your Linux kernel
doesn't have to allocate any resources at all when it receives a SYN
packet. Your kernel just basically sends SYN_ACK when it receives a SYN
packet, and doesn't do anything else. It only starts allocating
resources when it receives a *valid* ACK packet.

I hope you understood my (very bad) explanations about syncookies! :)
If you don't get some of what I explained, please ask.

To me, this should be enabled by default, as I think it's a very nice
feature.

Thomas


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4EDF7ECF.7090602@goirand.fr">http://lists.debian.org/4EDF7ECF.7090602@goirand.fr

Too many sockets in SYN_RECV
 

On 12/07/2011 06:30 PM, Marek Podmaka wrote:
> Or the other way - why the
> kernel can't handle the new connections without syncookies?

Let me try again, with that way.

Without syncookies, you got resources exhaustion. Not necessarily
memory, but just internal kernel resources.

Without having a look into the kernel code, I'd picture in my mind an
internal table with tcp connections that have received a SYN, but not
yet an ACK, or something similar. And there's a limit that you could
reach. Passed that limit, the kernel would start ignoring SYN packets,
and you can't connect anymore.

I hope this (a way shorter) version of the explanations are more clear.

It'd be cool to have someone with more internal Linux kernel know-how to
correct me with something more accurate facts: I only know the TCP
protocol theory, not the implementation in Linux itself, and I never
wrote a TCP/IP stack myself...

Thomas


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4EDF84E5.9080809@goirand.fr">http://lists.debian.org/4EDF84E5.9080809@goirand.fr

Too many sockets in SYN_RECV
 

Hello,

Wednesday, December 7, 2011, 16:23:17, Thomas Goirand wrote:

> On 12/07/2011 06:30 PM, Marek Podmaka wrote:
>> Or the other way - why the
>> kernel can't handle the new connections without syncookies?

> Without syncookies, you got resources exhaustion. Not necessarily
> memory, but just internal kernel resources.

> Without having a look into the kernel code, I'd picture in my mind an
> internal table with tcp connections that have received a SYN, but not
> yet an ACK, or something similar. And there's a limit that you could
> reach. Passed that limit, the kernel would start ignoring SYN packets,
> and you can't connect anymore.

Thanks for the explanation and also for the explanation of the
syncookies (which I was little familiar with).

Then I suppose the long connection times were because of the first
SYN being dropped and only retransmitted SYN packets (after few seconds
timeout) got accepted and replied by the server.

The question is why kernel just stopped at 512 SYN_RECV connections
when all limits were at least 1024 (for example
/proc/sys/net/ipv4/tcp_max_syn_backlog) and I was not able to raise
that limit of 512.


--
bYE, Marki


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 117560463.20111207163206@marki-online.net">http://lists.debian.org/117560463.20111207163206@marki-online.net