2026-06-02

TLS Certificate and Handshake Failure PCAP Analysis: Expired Certs, Alerts, SNI, and Connection Resets

How to analyze TLS handshake failures in packet captures, including expired certificates, unknown CA, SNI mismatch, TLS alerts, ClientHello, ServerHello, and TCP resets.

tls handshake failure, expired certificate, tls alert, sni mismatch, pcap analysis, https troubleshooting

TLS failures often appear as application errors: "SSL handshake failed", "certificate expired", "unknown ca", "connection reset", "remote host closed connection", "wrong version number", or "site cannot provide a secure connection." Users search for "TLS handshake failure pcap", "expired certificate packet capture", "TLS alert Wireshark", "SNI mismatch pcap", and "SSL connection reset analysis" because the application message may not reveal where the handshake failed.

A packet capture cannot decrypt modern TLS without keys, but it can still show a lot. It can show TCP connection setup, ClientHello, SNI, ALPN, TLS version offer, ServerHello, certificate message in older visible handshakes or metadata where available, TLS alerts, connection resets, retransmissions, and timing.

PCAP Surgery is useful because TLS failure evidence often needs to be isolated carefully. You may need to trim a large trace to one connection while preserving DNS lookup, TCP handshake, TLS handshake, alerts, resets, and timing.

TLS handshake stages

A simplified TLS handshake includes:

TCP SYN
TCP SYN-ACK
TCP ACK
TLS ClientHello
TLS ServerHello
Certificate and key exchange messages
TLS Finished
Application data

Depending on TLS version, some details are encrypted earlier than others. TLS 1.3 hides more handshake content than older TLS versions, but the first steps still reveal useful metadata.

The first question is: did the connection fail before TLS, during TLS, or after TLS?

Failure before TLS

If the TCP connection does not establish, the problem is not a certificate problem. Look for:

  • SYN retransmissions.
  • TCP reset after SYN.
  • No route or firewall drop.
  • Proxy refusal.
  • Server port closed.

An application may still report a generic SSL error if it expected TLS but never established a TCP path. The pcap separates network reachability from TLS negotiation.

ClientHello evidence

The ClientHello can reveal:

  • Server Name Indication, or SNI.
  • ALPN protocols such as h2 or http/1.1.
  • TLS versions offered.
  • Cipher suites offered.
  • Supported groups and extensions.

SNI is especially important. A server hosting multiple certificates needs SNI to choose the right certificate. If the client omits SNI or sends the wrong hostname, the server may present a default certificate that does not match the requested site.

This creates errors like:

  • Certificate name mismatch.
  • Wrong certificate served.
  • Handshake failure.
  • Server closes connection after ClientHello.

Expired certificate vs unknown CA

From the application perspective, certificate failures may look similar. But they mean different things:

  • Expired certificate: the certificate validity period is over.
  • Unknown CA: the certificate chain is not trusted.
  • Name mismatch: the certificate does not match the hostname.
  • Incomplete chain: intermediate certificate missing.
  • Revoked certificate: certificate is no longer valid.

In packet captures, certificate details may be visible in TLS 1.2 handshakes if not encrypted by the negotiated mode. In many modern cases, application logs are needed for exact certificate validation reason. The pcap still shows whether the server presented handshake data and whether the client responded with an alert or reset.

TLS alerts

TLS alerts are extremely useful. Common alerts include:

  • unknown_ca
  • bad_certificate
  • certificate_expired
  • handshake_failure
  • protocol_version
  • unrecognized_name
  • decrypt_error
  • close_notify

If the client sends an alert after receiving certificate data, the server may not be the side that "failed." The client rejected something.

If the server sends handshake_failure immediately after ClientHello, the server may not support the offered TLS version, cipher suite, SNI, or client certificate behavior.

Connection reset after ClientHello

A TCP reset after ClientHello can mean:

  • Middlebox blocked the handshake.
  • Server rejected TLS version or SNI.
  • Proxy expected plain HTTP, not TLS.
  • Client connected to the wrong port.
  • Server process crashed or closed aggressively.
  • Security appliance terminated the flow.

Do not assume every reset is a certificate problem. Check who sent the reset and when.

SNI mismatch

SNI mismatch is common in environments with load balancers, reverse proxies, Kubernetes ingress, CDNs, and multi-tenant servers. The TCP destination IP may host many domains. The server chooses certificate and backend based on SNI.

If the client connects by IP address, uses an internal hostname, or sends the wrong SNI, the server may present the wrong certificate.

Packet evidence:

  • ClientHello SNI value.
  • Destination IP and port.
  • DNS answer used by the client.
  • Server certificate or alert behavior.

If DNS resolves correctly but SNI is wrong, the application or proxy configuration is the place to inspect.

Protocol version and cipher mismatch

Modern clients may reject old TLS versions. Old servers may not support modern versions or cipher suites. The pcap can show what the client offered and how the server responded.

Look for:

  • Client offers only TLS 1.2/1.3, server requires older behavior.
  • Server sends protocol version alert.
  • Server closes without ServerHello.
  • Client and server share no cipher suite.
  • Middlebox interferes with TLS 1.3.

This is common in legacy embedded devices, old cameras, outdated appliances, and internal services that have not been updated.

DNS, TCP, and TLS timing together

For user-visible latency, preserve the whole path:

  1. DNS query and response.
  2. TCP connect.
  3. TLS handshake.
  4. Application data or failure.

A "TLS timeout" may actually be DNS delay, TCP retransmission, MTU black hole during certificate transfer, or server delay before ServerHello.

PCAP Surgery helps keep this timeline intact when reducing a trace.

Checklist for TLS handshake PCAP analysis

Use this process:

  1. Identify the target host, IP, and port.
  2. Preserve DNS lookup before the connection.
  3. Confirm TCP handshake succeeds.
  4. Inspect ClientHello for SNI, ALPN, versions, and ciphers.
  5. Check whether ServerHello appears.
  6. Look for certificate messages where visible.
  7. Look for TLS alerts and identify sender.
  8. Look for TCP resets and identify sender.
  9. Measure timing between handshake steps.
  10. Correlate with application certificate validation logs.

Final diagnosis

TLS certificate and handshake failures are not one category. The packet trace can distinguish TCP reachability failure, ClientHello rejection, SNI mismatch, protocol version mismatch, certificate validation failure, TLS alert, TCP reset, MTU-related stall, and server delay.

PCAP Surgery helps by preserving the DNS, TCP, TLS, alert, reset, and timing evidence needed to explain why a secure connection failed without reducing the issue to a generic "SSL error."