Why You Get TCP Connection Reset Storms and How to Stop Them

That Moment When Everything Breaks

You're running a web app or a database connection pool. Everything works fine for a few minutes. Then suddenly, your logs fill up with 'Connection reset by peer' or WSAECONNRESET. The app stalls. Users report timeouts. You restart the service and it's fine for another five minutes, then it happens again.

I've seen this exact pattern in production. One client had a Node.js app that fetched data from a MySQL database. Every 60 seconds, the connection would reset. The app would reconnect, get data, then reset again. It turned out the load balancer had a 60-second idle timeout. Any connection idle longer than 60 seconds got an RST packet. The app wasn't sending keepalive probes.

This isn't a code bug. It's a network timeout mismatch. Your app thinks the connection is still alive. The firewall, load balancer, or server OS thinks it's dead and sends a reset.

What Actually Causes a TCP Reset Storm

When you connect to a remote server (say port 443 for HTTPS), your system sends a SYN. The server replies with SYN-ACK. You send ACK. Now you have a TCP connection. If either side sends a packet with the RST flag set, the connection ends immediately. No graceful FIN. Just a hard disconnect.

A reset storm happens when multiple connections get RST packets in a short time. This usually happens because:

• A middlebox (firewall, load balancer, proxy) has an idle timeout shorter than your keepalive interval.
• The server closes the connection while your app still thinks it's open (typical with HTTP/1.0 or slow clients).
• Kernel network stack parameters are set too aggressively (Linux tcp_retries2 too low, for example).
• Your app doesn't send TCP keepalive probes, so the firewall drops the connection thinking it's dead.

The most common trigger: a load balancer (like AWS ALB, HAProxy, Nginx) with a 60-second idle timeout. Your app makes a request, gets a response, then sits idle for 65 seconds. At 60 seconds, the load balancer sends an RST. Your app tries to reuse that connection and gets hit with the reset.

How to Fix It (Step by Step)

1. Find the timeout culprit.
   Check your load balancer, firewall, or reverse proxy config. Look for settings like keepalive_timeout, idle_timeout, client_body_timeout.
   Example for Nginx: proxy_read_timeout 300s;
   For AWS ALB: the default idle timeout is 60 seconds. Change it to 300 seconds (or more).
2. Enable TCP keepalive on your app server.
   Keepalive sends a tiny probe every few seconds to keep the connection alive. Without it, the firewall thinks your connection is dead.
   On Linux, set these sysctl values in /etc/sysctl.conf:

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6

Then run sysctl -p to apply. This sends a keepalive probe every 60 seconds, then every 10 seconds after that, up to 6 retries.

On Windows, change the registry key:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
KeepAliveTime (DWORD) = 60000 (60 seconds in milliseconds)
KeepAliveInterval (DWORD) = 10000 (10 seconds)
TcpMaxDataRetransmissions (DWORD) = 6

Reboot or restart the network service.

3. Increase the retransmission timeout.
   If the server is slow to respond, tcp_retries2 on Linux controls how many times the kernel retries before resetting. Default is 15. I set it to 5 or 6 for production. Change it:

net.ipv4.tcp_retries2 = 5

4. Make your app set a socket timeout.
   This isn't a kernel fix, but it helps. Set a read timeout on your sockets to match the keepalive interval. For example, in Python:

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(30)  # seconds
s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

Don't set the timeout too low. A TCP RST can happen even with keepalive if the server restarts.

5. Test with a raw TCP connection.
   Use nc (netcat) or telnet to see if the reset happens. On Linux:

nc -vz your-server.com 443

If you get a reset immediately, the port isn't open. If it hangs for 60 seconds then resets, you've found the timeout.

What to Check If It Still Fails

• Check the server's keepalive settings too. Both sides need to agree. If the server has keepalive off, your client keepalive won't help.
• Look at the firewall logs. Most firewalls log RST packets. Check if they're coming from your load balancer or the remote server.
• Check the application logs. Sometimes the app itself sends a RST by closing the socket without a graceful shutdown. This is common in badly written HTTP clients that don't read the full response.
• Try a packet capture. Use tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0' on Linux. This shows only RST packets. You can see the exact IPs and ports involved.
• If you're behind a NAT gateway (like AWS NAT Gateway), these also have idle timeouts. Default is 350 seconds. If your app stays idle longer, you'll get resets. You can't change NAT Gateway timeouts, so you must use keepalive.

Once you align the timeouts between your app, your load balancer, and your OS, those reset storms should stop. It's a boring fix—no fancy code—but it works every time.