“Connection reset by peer” is the networking equivalent of a shrug. Something, somewhere, slammed the door mid-conversation. Your app reports an error. The business reports an outage. And three teams immediately start a spirited game of “not it.”
This case file is about ending that game. On Ubuntu 24.04, you can usually prove who sent the reset (client, proxy/load balancer, or server) with a handful of captures, counters, and logs—if you collect the right evidence in the right order.
What “connection reset by peer” actually means (and what it doesn’t)
At the wire level, a “reset” is TCP saying: “Stop. We’re not doing this.” That’s a RST segment. It’s not a timeout, not packet loss (though packet loss can lead to it), and not necessarily “the server crashed.” It’s an explicit teardown that aborts a connection immediately.
Applications see this as:
- ECONNRESET (common in Node.js, Go, Python, Java)
- recv() failed (104: Connection reset by peer) (common in Nginx logs)
- Connection reset by peer from curl/OpenSSL when the peer aborts during HTTP/TLS
Important nuance: “peer” means “the other side from the app’s point of view.” If your client talks to a reverse proxy, the “peer” is the proxy. If your proxy talks to an upstream, the “peer” is the upstream. If a firewall injects a reset, the “peer” is effectively “whoever forged the reset,” which is why we capture packets to identify the sender.
TCP resets show up for a few legitimate reasons:
- The process closed a socket abruptly (or the OS did it for the process).
- The other side received data for a connection it doesn’t recognize (no state), so it RSTs.
- A middlebox (proxy, firewall, NAT) decides your flow is unwelcome and sends RST.
- A timeout or resource exhaustion makes a component drop state; later packets trigger RST.
And here’s the practical takeaway: the fastest path to truth is to find the device that sent the RST. That’s not philosophy. That’s tcpdump.
Fast diagnosis playbook (first/second/third)
When you’re on-call, you don’t want a networking seminar. You want a sequence that narrows the blast radius in minutes.
First: classify the reset (client-facing or upstream-facing)
- If users hit a proxy/LB, check whether the proxy logs show client reset or upstream reset.
- If the app server itself is directly exposed, check server logs and socket stats for resets.
Second: capture the RST at the nearest controlled hop
- Capture on the proxy/LB interface if it exists; it sits between worlds.
- If no proxy, capture on the server NIC.
- If you control the client (synthetic monitor or a bastion), capture there too.
Third: correlate with time, tuple, and direction
- Match on the 5-tuple (src IP, src port, dst IP, dst port, protocol).
- Identify who sends the RST by looking at the packet’s source IP/MAC and the capture vantage point.
- Confirm the component’s reason using logs/counters (app logs, proxy logs, kernel counters, conntrack, TLS errors).
If you only remember one thing: don’t argue about blame until you can point at an RST frame and say “this box sent it.”
Facts and context that change how you debug
These are not trivia. Each one changes what evidence you trust and what hypotheses you prioritize.
- TCP RST predates most of your tooling. It’s part of TCP’s core behavior from the early RFCs; your shiny service mesh is just the newest actor sending old-school packets.
- Linux will send RST if there’s no listening socket. If nothing is bound to the destination port, the kernel responds with RST to a SYN, which clients often interpret as “reset.”
- RST is not “always the server.” Firewalls, load balancers, NAT gateways, and IDS/IPS can generate resets. Middleboxes do it to fail closed—or to fail in creative ways.
- Keep-alive misalignment is a classic reset generator. One side reuses an idle connection after the other side already timed it out and dropped state. The first new request gets a reset.
- NAT state expiration causes “ghost resets.” If a NAT device forgets a mapping while endpoints keep talking, the next packet can trigger a reset or be dropped—either way it looks like random flakiness.
- Path MTU problems can masquerade as resets. Technically that’s more often “hangs” than resets, but some stacks and middleboxes react badly and abort connections during TLS or HTTP/2.
- Ubuntu 24.04 ships with nftables by default tooling. iptables may still exist as a compatibility layer, but you need to know whether rules are in nft, not in your memory.
- QUIC/HTTP/3 moved many failure modes to UDP. Your browser might “fix” a broken TCP path by switching protocols, making the reset look intermittent and user-agent dependent.
- Observability changed the social dynamic. Ten years ago, teams argued based on “it feels like.” Today you can—and should—argue based on packet captures and structured logs.
One paraphrased idea, because it’s still the most operationally useful framing: paraphrased idea
— “Hope is not a strategy.” (attributed in ops culture to Edsger W. Dijkstra)
Prove who did it: client vs proxy vs server
Think in layers of custody. The reset originates somewhere, traverses some hops, then becomes “peer reset” to whichever app receives it. Your job is to identify the originator, not the victim.
Case A: the client sent the reset (it happens more than people admit)
Client-originated resets usually show up as:
- Browser navigated away or closed a tab mid-request.
- Mobile app backgrounded; OS killed sockets to save power.
- Client-side timeout shorter than server/proxy timeout; client aborts early.
- A retry library that cancels the in-flight request aggressively.
How to prove it:
- On the server or proxy capture, you see an RST coming from the client IP.
- Server logs show “client prematurely closed connection.”
- Proxy shows 499 (Nginx) or termination state consistent with client abort.
Case B: the proxy/load balancer sent the reset
Proxies reset connections for reasons that sound reasonable in a design doc and hurt in production:
- Idle timeout hit; proxy reclaims resources.
- Max requests per connection reached; proxy closes without graceful draining.
- Health-check or circuit-breaker logic decides upstream is bad and fails fast.
- Buffer limits, header limits, or request body policies trigger an abort.
- TLS inspection boxes decide your cert looks “weird.”
How to prove it:
- On client-side capture, the RST source IP is the proxy/LB VIP or node IP.
- On proxy capture, you see no upstream RST—only the proxy generating the client RST.
- Proxy logs show a local error (timeout, limit exceeded, upstream unavailable), not an upstream abort.
Case C: the server sent the reset
Servers reset connections for three big categories of reasons:
- Application behavior: process crashes, panics, exits, closes sockets abruptly, or rejects connections due to overload logic.
- Kernel behavior: no listener, SYN backlog overflow, memory pressure, or firewall rules that reject with tcp-reset.
- Infrastructure behavior: conntrack exhaustion, ephemeral port exhaustion on the server acting as a client, or routing changes.
How to prove it:
- Packet capture near the server shows RST sourced from the server IP.
- Server logs coincide with restarts, OOM kills, or application-level abort messages.
- Kernel counters show listen drops or aborts climbing during the incident window.
Joke #1: A TCP reset is like being hung up on mid-sentence—suddenly you’re not debating architecture, you’re debating who slammed the phone.
Field tasks: commands, outputs, and decisions (12+)
These are the tasks I actually run on Ubuntu 24.04 systems. Each one includes: command, a realistic snippet of output, what it means, and what decision you make next. Run them from the point closest to the symptom: client, proxy, or server.
Task 1: Reproduce with curl and force a single connection
cr0x@server:~$ curl -vk --http1.1 --no-keepalive https://api.internal.example/v1/ping
* Trying 10.20.30.40:443...
* Connected to api.internal.example (10.20.30.40) port 443 (#0)
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) Recv failure: Connection reset by peer
Meaning: The reset happened during TLS handshake. That narrows suspects: TLS-terminating proxy, IDS, misconfigured server TLS stack, or path issues causing middleboxes to abort.
Decision: Capture packets on the TLS terminator (proxy/LB) first. If there is no proxy, capture on the server NIC.
Task 2: Identify whether you’re talking to a proxy/LB VIP
cr0x@server:~$ getent ahostsv4 api.internal.example
10.20.30.40 STREAM api.internal.example
10.20.30.40 DGRAM api.internal.example
10.20.30.40 RAW api.internal.example
Meaning: Single IP. Still could be a VIP.
Decision: Check ARP/neigh to see if it looks like a VIP (MAC vendor patterns help) and ask “where does TLS terminate?”
Task 3: Confirm routing path and next hop (server side)
cr0x@server:~$ ip route get 10.20.30.40
10.20.30.40 via 10.10.0.1 dev ens5 src 10.10.2.15 uid 1000
cache
Meaning: Traffic exits via gateway 10.10.0.1.
Decision: If you suspect a firewall/NAT device, that gateway is a candidate. Capture on both sides if you can.
Task 4: Baseline socket state and reset counters with ss
cr0x@server:~$ ss -s
Total: 1234 (kernel 0)
TCP: 642 (estab 120, closed 410, orphaned 0, timewait 350)
Transport Total IP IPv6
RAW 0 0 0
UDP 18 12 6
TCP 232 180 52
INET 250 192 58
FRAG 0 0 0
Meaning: Lots of TIME-WAIT and closed sockets isn’t automatically bad, but it’s a smell if it spikes during resets.
Decision: If TIME-WAIT explodes under load, check for aggressive client disconnects, short keep-alives, or poor connection reuse.
Task 5: Inspect kernel TCP statistics for aborts and listen drops
cr0x@server:~$ nstat -az | egrep 'TcpExtListen|TcpAbort|TcpTimeout|TcpRetrans'
TcpExtListenDrops 18 0.0
TcpExtListenOverflows 5 0.0
TcpAbortOnData 22 0.0
TcpAbortOnTimeout 9 0.0
TcpTimeouts 133 0.0
TcpRetransSegs 420 0.0
Meaning: Listen overflows/drops suggest the server couldn’t accept connections fast enough (SYN backlog / accept queue pressure). Abort counters imply local stack aborted connections.
Decision: If these climb with incidents, stop blaming the client. Fix accept backlog, app accept loop, CPU saturation, or proxy connection bursts.
Task 6: Validate the server is actually listening (and on what)
cr0x@server:~$ sudo ss -ltnp '( sport = :443 )'
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 4096 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=1421,fd=6))
Meaning: Something is listening on 443, queue depth 4096 configured.
Decision: If nothing is listening, the kernel will RST SYNs. That’s your answer. If it is listening, move on.
Task 7: Packet capture to identify who sends the RST (server vantage)
cr0x@server:~$ sudo tcpdump -ni ens5 'tcp port 443 and (tcp[tcpflags] & (tcp-rst) != 0)' -vv
tcpdump: listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:22:10.118322 IP 10.20.30.40.443 > 10.10.2.15.51544: Flags [R.], seq 20394822, ack 129388, win 0, length 0
Meaning: At the server, we see a packet sourced from 10.20.30.40:443 sending RST to the client (10.10.2.15). If 10.20.30.40 is a proxy VIP, this implicates the proxy. If it’s the server IP, it implicates the server.
Decision: Capture on the proxy node itself. If the proxy didn’t generate it, you’ll see it inbound from upstream. If it did, you won’t.
Task 8: Capture on the proxy to separate “upstream reset” vs “proxy reset”
cr0x@server:~$ sudo tcpdump -ni ens5 'host 10.10.2.15 and tcp port 443' -vv
tcpdump: listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:22:10.118319 IP 10.10.2.15.51544 > 10.20.30.40.443: Flags [S], seq 129387, win 64240, options [mss 1460,sackOK,TS val 332191 ecr 0,nop,wscale 7], length 0
14:22:10.118322 IP 10.20.30.40.443 > 10.10.2.15.51544: Flags [R.], seq 0, ack 129388, win 0, length 0
Meaning: Immediate RST to SYN (seq 0, ack SYN+1) from 10.20.30.40 suggests “nothing is listening” on that VIP/port from the proxy’s perspective or a local firewall rule rejecting.
Decision: Check listener on the proxy host/container and firewall rules. If the proxy should be terminating TLS, verify the service is up and bound.
Task 9: Check Nginx error/access logs for client abort vs upstream abort
cr0x@server:~$ sudo tail -n 5 /var/log/nginx/error.log
2025/12/30 14:22:10 [error] 1421#1421: *441 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.10.2.15, server: api.internal.example, request: "GET /v1/ping HTTP/1.1", upstream: "https://10.30.40.50:8443/v1/ping", host: "api.internal.example"
Meaning: Nginx (proxy) is saying the upstream reset the connection while Nginx was reading the upstream response.
Decision: Stop accusing the client. Move investigation to upstream server 10.30.40.50:8443, and capture there.
Task 10: Check HAProxy termination state (if you use it)
cr0x@server:~$ sudo tail -n 3 /var/log/haproxy.log
Dec 30 14:22:10 lb1 haproxy[2011]: 10.10.2.15:51544 [30/Dec/2025:14:22:10.118] fe_https be_api/srv2 0/0/1/2/3 200 512 - - ---- 12/12/0/0/0 0/0 "GET /v1/ping HTTP/1.1"
Dec 30 14:22:10 lb1 haproxy[2011]: 10.10.2.15:51545 [30/Dec/2025:14:22:10.221] fe_https be_api/srv2 0/0/0/0/0 0 0 - - SD-- 3/3/0/0/0 0/0 "GET /v1/ping HTTP/1.1"
Meaning: The termination flags (here “SD–”) suggest the server side got shut down. Exact codes vary by config, but the log tells you which side ended the session.
Decision: If server-side termination dominates during the incident window, focus on upstream stability, timeouts, and resource pressure.
Task 11: Check for conntrack exhaustion (middlebox-ish symptom on Linux NATs)
cr0x@server:~$ sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262041
net.netfilter.nf_conntrack_max = 262144
Meaning: You’re basically out of conntrack entries. At that point, new connections get dropped or mishandled. Some environments will show resets; others show timeouts.
Decision: Raise nf_conntrack_max (with memory awareness), shorten conntrack timeouts where safe, and reduce connection churn (keep-alive, pooling).
Task 12: Check nftables rules for “reject with tcp reset”
cr0x@server:~$ sudo nft list ruleset | sed -n '1,120p'
table inet filter {
chain input {
type filter hook input priority filter; policy drop;
ct state established,related accept
iif "lo" accept
tcp dport 22 accept
tcp dport 443 reject with tcp reset
}
}
Meaning: The host firewall is explicitly sending TCP resets on port 443. That will manifest as “connection reset by peer” for clients.
Decision: Fix the rule. Replace reject with accept for allowed sources, or use a controlled drop if you prefer timeouts (but be ready for slow failures).
Task 13: Identify MTU and offload mismatches (quietly nasty)
cr0x@server:~$ ip link show dev ens5
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:ab:cd:ef:12:34 brd ff:ff:ff:ff:ff:ff
Meaning: MTU is 1500 on this interface. If other segments use jumbo frames or tunnels reduce MTU, fragmentation/PMTUD issues can appear during TLS.
Decision: If you suspect MTU, run a targeted ping with DF set between the same endpoints and adjust MTU/MSS clamping on the right device.
Task 14: Prove PMTUD works (or doesn’t) with DF pings
cr0x@server:~$ ping -M do -s 1472 10.20.30.40 -c 3
PING 10.20.30.40 (10.20.30.40) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1450
ping: local error: message too long, mtu=1450
ping: local error: message too long, mtu=1450
--- 10.20.30.40 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2040ms
Meaning: Your effective path MTU is 1450 (common with VXLAN/GRE/cloud encapsulation). If your stack/proxy doesn’t clamp MSS properly, you can see bizarre handshake failures and sometimes resets.
Decision: Clamp MSS on the edge (or set appropriate MTUs on interfaces), then retest TLS.
Task 15: Check systemd journal around the exact failure time
cr0x@server:~$ sudo journalctl -u nginx --since "2025-12-30 14:20:00" --until "2025-12-30 14:25:00" | tail -n 8
Dec 30 14:22:09 api1 nginx[1421]: 2025/12/30 14:22:09 [warn] 1421#1421: *439 upstream timed out (110: Connection timed out) while reading response header from upstream
Dec 30 14:22:10 api1 nginx[1421]: 2025/12/30 14:22:10 [error] 1421#1421: *441 recv() failed (104: Connection reset by peer) while reading response header from upstream
Dec 30 14:22:11 api1 systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
Dec 30 14:22:11 api1 systemd[1]: nginx.service: Failed with result 'signal'.
Dec 30 14:22:12 api1 systemd[1]: nginx.service: Scheduled restart job, restart counter is at 1.
Meaning: Now we have a narrative: upstream timeouts, then resets, then Nginx gets killed and restarts. This is not a “random network blip.” Something is killing Nginx (OOM killer? deployment? watchdog?).
Decision: Check dmesg for OOM and check deployment automation. If Nginx restarts, it can reset in-flight connections.
Task 16: Check for OOM kills and kernel complaints
cr0x@server:~$ dmesg -T | tail -n 8
[Mon Dec 30 14:22:11 2025] Out of memory: Killed process 1421 (nginx) total-vm:512000kB, anon-rss:210000kB, file-rss:12000kB, shmem-rss:0kB, UID:0 pgtables:900kB oom_score_adj:0
[Mon Dec 30 14:22:11 2025] oom_reaper: reaped process 1421 (nginx), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Meaning: The kernel killed Nginx. That will absolutely produce resets and half-open weirdness.
Decision: Fix memory pressure (limits, leaks, log buffering, too-large caches). Until then, chasing “the network” is a hobby, not an incident response.
Joke #2: If you guess the cause without a packet capture, you’re basically doing incident response by astrology.
Three corporate mini-stories (anonymized, plausible, technically accurate)
1) The incident caused by a wrong assumption: “The server reset us”
The symptom was clean: mobile clients in a specific region got “connection reset by peer” during login. The API team saw nothing obvious. The load balancer dashboards looked calm. Someone said the dangerous sentence: “It must be the app; it’s the only thing that changed last week.”
So they rolled back the app. No change. They rolled back a database migration. No change. They started planning a regional failover. Meanwhile, customer support learned new and creative ways to say “please try again.”
Finally, an SRE did the unglamorous thing: tcpdump on the load balancer nodes, filtered for RST. The RST packets weren’t coming from the API servers at all. They were coming from a security appliance upstream of the load balancer, sourced with the appliance’s IP, and only for specific client ASN ranges.
The root cause was a threat-intel update that mislabeled a shared mobile carrier NAT block as hostile. The appliance started injecting resets for TLS handshakes that matched a heuristic. The “peer” in the client logs was the load balancer VIP, but the sender was neither LB nor server—it was the box no one wanted to admit existed.
Fix was simple and political: whitelist, then change the change-control process so security updates were treated like production deployments with rollback and monitoring. The big lesson: assumptions are expensive; packets are cheap.
2) The optimization that backfired: aggressive idle timeouts
A platform team wanted to reduce connection count on an internal API gateway. They tuned “idle timeout” down hard. The graphs looked great: fewer open connections, less memory, slightly better CPU. People congratulated each other in a meeting. That was the whole postmortem, in their heads.
A week later, resets spiked—but only for a handful of services. Those services used HTTP keep-alive with long-lived connections and infrequent bursts. The clients were well-behaved, reusing connections. The gateway was now timing them out in the quiet moments.
So what happened? The client would send the next request on a connection it believed was alive. The gateway had already dropped state and reclaimed it. The gateway responded with RST (or sometimes the kernel did, depending on how the socket got closed). From the client’s perspective, “peer reset.” From the gateway’s perspective, “I told you to stop being idle.”
The worst part: retries made it noisier. Some clients retried immediately, multiplying load. The change intended to reduce resource usage ended up increasing churn and latency and adding unpredictable failure spikes.
The fix was to align timeouts end-to-end: client keep-alive idle, gateway idle, upstream idle, plus jitter. They also added explicit connection draining for deployments. Optimization is fine. Optimization without failure-mode thinking is performance art.
3) The boring but correct practice that saved the day: tuple-based correlation
A financial services company had a mixed environment: Ubuntu servers, a managed load balancer, and a service mesh in Kubernetes. Resets showed up as intermittent 502s, and the teams were already drafting their favorite narratives.
The on-call lead did a boring thing: they standardized incident evidence collection. For any reset report, you had to produce: timestamp (UTC), client IP, server IP, destination port, and if possible the ephemeral source port. That’s the 5-tuple, minus protocol because it was always TCP.
With that, they matched a single failing request across: client logs, mesh sidecar logs, load balancer logs, and a tcpdump on one node. The capture proved the RST originated from the sidecar, not the app container and not the upstream. The sidecar was enforcing a policy update that had briefly invalidated a SAN entry in a certificate chain during rotation.
The fix was to stage certificate rotation with overlap and to validate with synthetic traffic before flipping policy. The bigger win was cultural: evidence format became muscle memory. No heroics. Just repeatable proof.
Common mistakes: symptoms → root cause → fix
This is the section you skim while someone is asking “Is it our fault?” on a call.
1) Resets happen immediately on connect
- Symptom: curl fails right after “Connected”, or SYN gets RST immediately.
- Root cause: Nothing listening on that IP:port; VIP misrouted; firewall rejects with tcp-reset.
- Fix: Verify listener with
ss -ltnp; verify VIP config; check nftables for reject rules; confirm the service is bound to the correct address.
2) Resets spike during deployments
- Symptom: short bursts of ECONNRESET during rollout, then normal.
- Root cause: In-flight connections killed by process restart; no graceful shutdown/draining; LB still sending to terminating pods.
- Fix: Add connection draining; increase terminationGracePeriod; send readiness to false before SIGTERM; align LB deregistration delay with app shutdown.
3) Only long requests fail with resets
- Symptom: small endpoints fine; large downloads/uploads or slow upstream calls reset mid-stream.
- Root cause: proxy read timeout; upstream timeouts; buffer limits; L7 policy device aborts large payloads.
- Fix: Check proxy timeouts; tune upstream timeouts; validate max body size; confirm whether a DLP/WAF is injecting RST.
4) Resets appear as “random” across many services
- Symptom: lots of unrelated apps see resets at the same time.
- Root cause: shared dependency: conntrack exhaustion on a NAT, overloaded LB, firewall policy push, kernel OOM on a shared proxy node.
- Fix: Check conntrack count/max, LB health, firewall change logs, and OOM events; stabilize shared layer first.
5) Resets mostly affect one client type (mobile, one region, one ISP)
- Symptom: desktop fine, mobile flaky; or one region fails.
- Root cause: MTU/path differences; carrier NAT behavior; regional security appliance; Geo-based routing to a broken POP.
- Fix: Compare packet captures from different networks; run DF pings; check routing policies; verify POP-specific proxy configs.
6) Upstream resets blamed on “network” but kernel counters show listen drops
- Symptom: Nginx reports upstream reset; upstream team says “not us.”
- Root cause: upstream accept queue overflow or SYN backlog pressure; upstream process is too slow to accept.
- Fix: Increase backlog; tune
net.core.somaxconn, service-specific backlog; fix CPU starvation; add capacity or rate limiting.
Checklists / step-by-step plan
Checklist A: “I need an answer in 15 minutes”
- Reproduce with curl from a controlled host; record exact UTC time and destination IP/port.
- Identify whether a proxy/LB is in the path; determine where TLS terminates.
- Capture RST packets on the proxy/LB (best) or server (next best) for 60–120 seconds during reproduction.
- From the capture, identify the RST sender IP. If it’s the client IP, it’s client-side abort. If it’s the proxy VIP, it’s proxy-side. If it’s the upstream IP, it’s upstream.
- Correlate with logs on the sender: proxy error log, app log, kernel journal, firewall logs.
- Make a call: mitigate (rollback, disable feature, bypass WAF, raise timeouts) while you investigate root cause.
Checklist B: “Prove it rigorously so the right team fixes it”
- Collect 5-tuple evidence from at least two vantage points (proxy + upstream, or client + proxy).
- Capture both directions; don’t just filter “tcp-rst” without context. Keep a full flow for a failing connection.
- Confirm time synchronization (NTP) on all involved nodes; clock skew ruins correlation.
- Check kernel counters on the suspected sender (listen drops, aborts, retransmits).
- Check config timeouts alignment: client timeout, proxy idle timeout, upstream keep-alive, server timeouts.
- Check for resource pressure (CPU, memory, file descriptors) and forced restarts.
- Check firewall/nftables rules that can generate RSTs and recent changes.
- Document the minimal proof: one pcap excerpt + one log line + one counter change that all point to the same sender.
Checklist C: “Prevent the next round”
- Standardize keep-alive and idle timeout budgets across layers; add jitter.
- Implement graceful shutdown and connection draining for proxies and apps.
- Monitor conntrack utilization on NAT/proxy nodes.
- Expose reset-related counters: TcpExtListenDrops, aborts, nginx 499/502, haproxy termination states.
- Run synthetic probes that track handshake failures separately from application failures.
FAQ
1) Does “Connection reset by peer” always mean the server crashed?
No. It means a TCP RST was received. The sender could be the server process, the server kernel, a proxy, a firewall, or even the client itself.
2) How do I tell client reset vs server reset in Nginx?
Nginx often logs client aborts as “client prematurely closed connection” and upstream aborts as “recv() failed (104) while reading response header from upstream.” Combine that with packet capture to be sure.
3) Why do I see resets only during TLS handshake?
Common causes: TLS-terminating proxy issues, certificate/policy enforcement by middleboxes, MTU/PMTUD problems, or a service not actually listening on the expected port behind a VIP.
4) Can a firewall send a reset instead of dropping?
Yes. Many rulesets use “reject with tcp reset” to fail fast. That’s great for user experience when intentional, and terrible when it’s accidental.
5) I captured packets and see RST from the proxy IP. Does that prove the proxy is at fault?
It proves the proxy sent the RST. The proxy may be reacting to upstream failure (timeouts, upstream resets) or enforcing a policy (idle timeout, limits). Next step: check whether upstream traffic shows a prior error that triggered the proxy decision.
6) What’s the difference between a FIN and an RST?
FIN is a polite close: “I’m done sending.” RST is an abort: “Stop immediately; this connection is invalid.” FIN tends to produce clean EOF in apps; RST tends to produce ECONNRESET.
7) Why do resets increase when we reduce keep-alive timeouts?
Because you increase connection churn and the chance that one side reuses a connection the other side has already dropped. Misaligned idle timeouts are reset factories.
8) Can Linux itself generate resets even if the application is fine?
Yes. No listener, backlog overflow, local firewall reject rules, and some abrupt socket closures can all result in kernel-sent RSTs.
9) We run Kubernetes. Who is “the peer” now?
Potentially: the Service VIP, kube-proxy/iptables/nft rules, an Ingress controller, a sidecar proxy, or the pod itself. That’s why you capture at the node and at the ingress tier to find the actual RST sender.
10) If I can’t run tcpdump in production, what’s the next best thing?
Proxy logs plus kernel counters plus strict tuple/time correlation. It’s weaker evidence, but it can still prove directionality. Push for controlled, time-boxed captures as a standard incident tool.
Conclusion: next steps that actually reduce resets
“Connection reset by peer” is not a diagnosis. It’s a symptom with a very specific physical artifact: a TCP RST. If you can capture it, you can stop guessing. If you can identify the sender, you can stop debating.
Practical next steps:
- Adopt the fast playbook: reproduce, capture RST, identify sender, correlate logs.
- Standardize evidence: UTC timestamp + 5-tuple, every time. Make it boring.
- Align timeouts end-to-end, and don’t “optimize” idle timeouts without testing long-idle reuse.
- Harden shared layers: conntrack sizing, proxy memory limits, graceful shutdown, and firewall rules reviews.
- Turn this into an on-call runbook. Your future self deserves fewer surprises.