Fix IPsec tunnel fragmentation: DF bit, MTU, and MSS clamping

You're on a Linux box or Cisco router, and your IPsec tunnel works for small stuff like SSH, but bigger transfers stall. A 1472-byte ping to the remote side works, but 1473 bytes fails. Or you get 'frag needed and DF set' in logs. What's happening here is the tunnel adds overhead (50-70 bytes for IPsec+GRE), so a standard 1500-byte Ethernet frame becomes 1550+ bytes and doesn't fit the physical link. The DF (Don't Fragment) bit is set on the inner packets, so the router can't fragment them — it just drops them and sends an ICMP 'frag needed' back. But firewalls often block that ICMP, so your machine never knows and keeps sending big packets into a black hole.

Why this happens

Every tunnel eats bytes. For a typical IPsec tunnel using ESP in transport mode, you lose about 50-54 bytes. With GRE inside IPsec, it's 60-70. The outer packet gets bigger than the link MTU (usually 1500). The router can't fragment because the inner packet's DF bit is set (default on TCP). It should send ICMP type 3 code 4 back, but most networks block that. So the connection just times out.

The real fix: MSS clamping on the tunnel interface

Instead of fighting MTU on every device, clamp the TCP MSS (Maximum Segment Size) on the tunnel interface. TCP negotiates MSS during handshake. If you force it lower, the TCP stack will never send packets that need fragmentation. That's the cleanest fix.

1. Find the tunnel interface name. On Linux, run ip link show and look for something like tun0, ipsec0, or vti0. On Cisco, it's TunnelX (e.g., Tunnel1).
2. Calculate the right MSS value. Standard Ethernet MTU is 1500. Subtract IPsec overhead: 50 bytes for ESP-AES+SHA. 1500 - 50 = 1450. Then subtract 40 for IP+TCP headers: 1450 - 40 = 1410. That's your MSS clamp value. For GRE+IPsec, use 1360.
3. Apply the clamp. On Linux (using iptables or nftables):
   iptables -t mangle -A FORWARD -o tun0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1410
   Or with nftables:
   nft add rule ip mangle forward oifname "tun0" tcp flags syn tcp option maxseg size set 1410
   On Cisco: ip tcp adjust-mss 1410 under the tunnel interface.
4. Test with a large transfer. Try scp a big file or iperf3 -c remote_ip -t 10. Should work now.
5. Make the change permanent. For iptables, save rules with iptables-save > /etc/iptables/rules.v4. On Cisco, the ip tcp adjust-mss already sticks in config.

If it still fails

• Check ICMP is not blocked. From the remote side, run ping -M do -s 1472 and see if you get a reply. If not, something is filtering ICMP type 3 code 4. Add a firewall rule to allow it between tunnel endpoints. Without ICMP, PMTUD can't work even with MSS clamped.
• Verify the clamp is hitting the right traffic. Use conntrack -E on Linux to see SYN packets and check if MSS is being rewritten. Or tcpdump -i any port 22 and look for MSS option in SYN packets — it should be 1410.
• Try a lower MSS. Some carriers use MPLS with overhead. Try 1360 for IPsec alone, or 1300 with GRE. It's fast to test: iptables -R ... --set-mss 1360 and retry.
• On Cisco, check the 'ip mtu' command. If you set ip mtu 1400 on the tunnel, it forces the router to fragment before encryption — but that adds CPU load and can break some apps. Better to use only MSS clamp and keep MTU at default.

One gotcha: if your app uses UDP (like DNS, VoIP, or QUIC), MSS clamping does nothing. For UDP, you need to set the DF bit off on the inner packets, or fragment at the application layer. But for 90% of tunnel problems, it's TCP and MSS clamp wins.