MTU Black Hole and Fragmentation PCAP Analysis: Finding PMTUD Failures, MSS Problems, and Stalled TCP

MTU problems create some of the most confusing network failures. Small requests work. Pings may work. DNS may work. TCP connects. TLS may start. Then large responses stall, uploads hang, websites partially load, VPN traffic breaks, or RTSP/HTTP streams fail at higher bitrate. Users search for "MTU black hole pcap", "path MTU discovery failure", "TCP MSS problem", "fragmentation packet capture", and "large packets stall VPN" because the connection is not simply up or down.

A packet capture can show whether large packets are sent, fragmented, dropped, retransmitted, or blocked by missing ICMP feedback. But the trace must be interpreted carefully because capture point, offload, and tunnel overhead can distort the visible packet sizes.

PCAP Surgery is useful here because MTU investigations often need focused traces: one TCP conversation, original timing, ICMP messages, retransmissions, MSS negotiation, and enough packets before and after the stall.

What MTU means

MTU is the maximum transmission unit on a link. Ethernet commonly uses 1500 bytes. VPNs, PPPoE, tunnels, overlays, and cloud networking can reduce the effective path MTU because they add headers.

If a sender transmits packets larger than a path can carry, one of two things must happen:

• Packets are fragmented where allowed.
• Packets are dropped and the sender must learn a smaller path MTU.

Path MTU Discovery relies on ICMP messages to report that a packet was too large. If those ICMP messages are blocked, the sender may keep sending packets that disappear. That is the classic MTU black hole.

Symptoms of MTU black hole

Common symptoms include:

• TCP connection opens but large transfer stalls.
• HTTPS handshake stalls after ClientHello or ServerHello.
• Login page loads but file upload fails.
• VPN connects but some sites do not load.
• SSH works until large output appears.
• API calls with small payloads work, large payloads fail.
• RTSP control works but media or large interleaved packets behave poorly.
• Retransmissions repeat with similar packet sizes.

The key pattern is size sensitivity. Small packets pass. Larger packets fail.

MSS negotiation

TCP Maximum Segment Size is negotiated during the SYN exchange:

Client -> Server: SYN, MSS 1460
Server -> Client: SYN-ACK, MSS 1460

MSS tells the peer the largest TCP payload segment it should send. For normal Ethernet with 1500 MTU, MSS is often 1460 for IPv4 TCP without options. Through a VPN, a lower MSS may be needed.

If MSS is too high for the real path, large segments may be dropped later.

In a capture, inspect the SYN and SYN-ACK. If the path includes a VPN or tunnel but MSS remains 1460, MSS clamping may be missing.

ICMP fragmentation needed

For IPv4 with Don't Fragment set, a router that cannot forward a packet should send an ICMP "fragmentation needed" message. For IPv6, packet-too-big messages are essential because routers do not fragment packets in the same way.

If the capture shows:

• Large packet sent.
• ICMP packet-too-big received.
• Sender reduces packet size.
• Transfer continues.

then Path MTU Discovery is working.

If the capture shows:

• Large packet sent repeatedly.
• No ICMP feedback.
• Retransmissions repeat.
• Transfer stalls.

then an MTU black hole is plausible.

Fragmentation evidence

Fragmentation itself is not always a failure. But fragmented traffic can be dropped by firewalls, NAT devices, or overloaded paths. Large UDP payloads are especially vulnerable because there is no TCP retransmission at the transport layer.

In a pcap, look for:

• IP fragments.
• Missing fragment sequences.
• Reassembly failure.
• Large UDP packets.
• DNS or RTP payloads that exceed path limits.
• ICMP packet-too-big messages.

If only one fragment is missing, the whole packet cannot be reassembled.

TLS handshake stalls and MTU

TLS handshakes can expose MTU issues because certificate messages may be larger than small TCP packets. A client may send ClientHello, the server may respond with certificate data, and then the handshake stalls because large packets disappear.

Users often search for "TLS handshake timeout MTU" or "HTTPS works on some sites not others" in this situation. The packet evidence may show repeated retransmission of the same large TLS-carrying TCP segment.

The problem is not the certificate itself; it is that larger packets cannot traverse the path.

VPN and tunnel overhead

VPNs reduce effective MTU because they wrap packets with additional headers. WireGuard, IPsec, OpenVPN, GRE, VXLAN, PPPoE, and cloud overlays all add overhead.

If the endpoint sends 1500-byte packets into a tunnel that cannot carry them, the tunnel must fragment, drop, or signal a smaller path MTU. If ICMP is blocked, the sender may never adapt.

MSS clamping at the VPN boundary is a common mitigation for TCP. UDP protocols may need application-level packet sizing or tunnel MTU adjustment.

Capture point and offload

Host captures may show packets larger than the actual wire packets because of segmentation offload. This can confuse MTU analysis. A 64 KB packet in a host capture may not have gone on the wire as one packet.

To avoid mistakes:

• Inspect SYN MSS values.
• Capture at a network tap, router, or tunnel boundary when possible.
• Be aware of TSO/GSO/GRO/LRO offload.
• Compare sender-side and receiver-side captures.
• Look for retransmission behavior and ICMP feedback.

Do not diagnose MTU only from one apparent packet size in a host capture.

Checklist for MTU black hole PCAP analysis

Use this process:

1. Identify the affected conversation.
2. Inspect SYN and SYN-ACK MSS values.
3. Note whether a VPN, tunnel, PPPoE, or overlay is in the path.
4. Find the first stall or retransmission burst.
5. Check packet sizes around the stall.
6. Look for ICMP fragmentation-needed or packet-too-big messages.
7. Check whether the sender reduces segment size after ICMP.
8. Look for repeated retransmission of similar large segments.
9. Compare small and large requests.
10. Account for offload and capture point.

Final diagnosis

MTU black holes are size-dependent network failures. They often look like random application timeouts because small packets work and large packets fail. The strongest packet evidence is MSS negotiation, large segment retransmissions, missing ICMP packet-too-big feedback, fragmentation behavior, and path context such as VPN or tunnel overhead.

PCAP Surgery helps preserve the exact packet sequence needed to prove whether a stalled connection is really an MTU problem instead of generic packet loss, server slowness, or application failure.