Asymmetric Routing: The Invisible Cause of “Random Drops”

February 6, 2026 • February 6, 2026 • Read: 21 min • Views: 0

Was this helpful?

Every SRE eventually meets the outage that behaves like a ghost. Metrics spike. TCP resets appear and disappear. One AZ looks cursed. Users swear it’s “random,” and your incident channel turns into a group therapy session.

Half the time, the network is doing exactly what you told it to do—just not what you thought you told it to do. Asymmetric routing is the classic culprit: traffic goes out one way and comes back another, and anything stateful in the middle starts making “helpful” decisions that look like sabotage.

What asymmetric routing actually is (and why it hurts)

Asymmetric routing means the forward path and the return path of a flow are different. Not “different by a hop.” Different by a device boundary that matters: a firewall cluster, a NAT gateway, a load balancer, a DDoS appliance, an IDS/IPS, or even just a Linux host doing connection tracking.

The internet is inherently asymmetric. BGP doesn’t promise symmetry. It promises reachability, and even that comes with a shrug. In many environments, asymmetry is normal and harmless because routers are stateless forwarding machines and don’t care which direction packets arrive from.

Production networks, however, are filled with devices and kernels that keep state:

Stateful firewalls track sessions. If SYN goes one way and SYN-ACK comes back another, you get drops or “invalid state” logs.
NAT is state by definition. Return packets need the same translator state.
Load balancers often depend on a consistent 5-tuple path, or they inject SNAT/DSR behavior that changes where return traffic should go.
Conntrack in Linux (iptables/nftables) can drop packets that don’t match expected state.
Reverse-path filtering (rp_filter) can drop replies if the kernel thinks the source shouldn’t be reachable via that interface.

Asymmetric routing isn’t always “bad.” It’s bad when your stateful middleboxes disagree about reality, or when your security posture assumes “if it didn’t come through me, it’s suspicious.” Most networks are designed by optimists and then operated by pessimists. That’s the tension.

Why it looks random: state, hashing, and selective pain

Asymmetry doesn’t usually break everything. It breaks some flows, in patterns that feel like superstition. Here’s why:

ECMP makes per-flow behavior look like dice rolls

Equal-Cost Multi-Path routing (ECMP) spreads flows across multiple next-hops using a hash (often of the 5-tuple). That means:

One TCP connection can be perfectly fine while another, to the same IP, fails.
Retries might work because the tuple changes (new source port, new hash, new next-hop).

To the human eye, that’s “random.” To the router, it’s deterministic math.

Stateful devices fail closed, selectively

Firewalls, conntrack, and NAT devices typically drop packets they can’t classify. But classification depends on seeing the beginning of the conversation (SYN) and keeping track of translation/state. If only return traffic is asymmetric, your client sees stalls and retransmits; if only forward traffic is asymmetric, your server sees half-open sessions. Either way, the application team gets paged first. Naturally.

Retries hide the crime scene

Modern stacks retry aggressively. HTTP clients open new connections. QUIC changes the game again. Load balancers retry upstreams. What you see is latency spikes, not total failure. That’s worse because nobody believes it’s “the network” until it’s too late.

Joke #1: Asymmetric routing is like losing one sock in the laundry—everything still works, but it’s quietly ruining your morning.

Interesting facts and historical context

Some context helps because asymmetric routing is not a “cloud problem” or a “modern problem.” It’s a property of how we built the internet and then bolted security and observability onto it.

BGP never guaranteed symmetry. It’s a path-vector protocol focused on policy and reachability, not round-trip elegance.
Early networks assumed stateless forwarding. The original IP design expected routers to be dumb and endpoints to handle reliability.
Stateful firewalls popularized “return path must match.” Once enterprises standardized on stateful inspection, asymmetry became a practical outage trigger.
NAT made asymmetry operationally dangerous. NAT requires translation state; breaking the return path breaks the session, not just the trace.
ECMP became common as bandwidth needs grew. Hash-based load spreading is great until you combine it with stateful devices that aren’t sharing state properly.
“Anti-spoofing” checks evolved into rp_filter and uRPF. These mitigations improved security but made asymmetry a drop condition rather than a curiosity.
Firewall clustering drove session synchronization tech. Vendors added state sync/HA features largely because asymmetric traffic and failovers were ruining sessions.
Anycast made asymmetry normal at global scale. Anycast routes you to the nearest site—until the return path follows a different policy and you learn humility.
Cloud networking made paths less visible. When you don’t own the underlay, you debug symptoms—and asymmetry thrives in the shadows.

Failure modes: what breaks and how it shows up

Asymmetric routing itself is not a failure. It’s the precondition. The failure is what happens when something expects symmetry. Here are the common breakages, with the “smell” you notice first.

1) Stateful firewall drops “out of state” packets

Smell: some connections hang at SYN-SENT or ESTABLISHED with retransmissions. Firewall logs show “invalid state” or “no session.”

Mechanism: forward SYN goes through Firewall A, return SYN-ACK comes through Firewall B. Firewall B never saw the SYN; it drops the SYN-ACK.

2) NAT return traffic misses the translator

Smell: outbound connections work intermittently; inbound responses never arrive; packet captures show replies hitting a different edge.

Mechanism: source NAT created on one device, return arrives at another without that translation table entry.

3) Linux rp_filter drops replies on the “wrong” interface

Smell: single host problem; pings from one side work, from another don’t; dmesg shows martian source or rp_filter drops depending on settings/logging.

Mechanism: reverse path check fails: the kernel believes the source would be reached via a different interface than the one the packet arrived on.

4) Conntrack sees “INVALID” and your rules drop it

Smell: nftables/iptables counters increment on INVALID drop rules; some flows die after reroutes/failovers.

Mechanism: packets arrive without matching conntrack entry because the initiating packet took a different path, or because a mid-flow reroute occurred.

5) Load balancer return-path mismatch (SNAT/DSR confusion)

Smell: health checks are green, user traffic is flaky; only certain client networks fail; server sees traffic, client doesn’t receive replies.

Mechanism: in Direct Server Return (DSR) or partial-NAT designs, return traffic bypasses the LB or chooses a different egress; upstream security devices drop it or it gets routed incorrectly.

6) PMTUD/MTU issues get misdiagnosed as asymmetry (and vice versa)

Smell: small packets work, large ones stall; TCP MSS clamping “fixes” it; traceroute is inconsistent.

Mechanism: MTU blackholing is its own beast, but asymmetric paths can have different MTUs and different ICMP behavior, making it look like random loss.

Joke #2: If your packet takes a different way home, don’t call it “adventurous”—call it “about to get dropped by a firewall.”

Fast diagnosis playbook (first/second/third)

This is the part you follow at 02:13 when everyone wants an answer and you want your bed back.

First: Prove it’s path-related, not host-local

Pick one failing 5-tuple (src IP:port → dst IP:port). Don’t debug “the service.” Debug a specific flow.
Check for retransmits and resets on the client and server. If you see SYN retransmits or mid-stream stalls, you’re in network/state territory.
Run forward and reverse traceroutes (or at least compare hop patterns from both ends). If paths differ meaningfully across security boundaries, suspect asymmetry.

Second: Identify the stateful chokepoint

Find NAT/firewall/LB boundaries in the path. Most “random drops” are stateful devices enforcing a story about sessions.
Look for “invalid state” counters/logs on those devices (or on Linux conntrack drop rules).
Check ECMP / multiple next-hops on the relevant routes. If you have two ways out, you can have two ways back.

Third: Confirm with packet capture and routing tables

Capture on both ends. If the server sends replies and the client never sees them (or vice versa), you’ve boxed the problem into the network middle.
Check rp_filter and policy routing on the hosts if the issue is one-sided.
Force symmetry temporarily (pin route, disable ECMP for the prefix, set session persistence correctly) to validate the hypothesis before making a permanent change.

Rule of thumb: if a flow works from one source subnet but not another, you’re likely looking at a routing policy difference, not a flaky NIC.

Hands-on tasks: commands, outputs, decisions (12+)

Below are tasks you can actually run. Each includes: command, sample output, what it means, and the decision you make.

Task 1: Confirm route selection to the destination (Linux)

cr0x@server:~$ ip route get 203.0.113.10
203.0.113.10 via 192.0.2.1 dev eth0 src 192.0.2.20 uid 1000
    cache

What it means: The kernel will send traffic to 203.0.113.10 via gateway 192.0.2.1 on eth0, using source IP 192.0.2.20.

Decision: If the expected egress is different (wrong interface, wrong source IP), you have policy routing or route priority issues before you even reach the network.

Task 2: Check policy routing rules (Linux)

cr0x@server:~$ ip rule show
0:	from all lookup local
100:	from 192.0.2.0/24 lookup 100
32766:	from all lookup main
32767:	from all lookup default

What it means: Traffic sourced from 192.0.2.0/24 uses routing table 100. That can easily create asymmetric return paths if replies pick a different table.

Decision: If asymmetry is host-specific, validate table 100’s default route and ensure reply traffic uses the same policy (or deliberately SNAT it).

Task 3: Inspect the alternate routing table used by policy routing

cr0x@server:~$ ip route show table 100
default via 198.51.100.1 dev eth1
198.51.100.0/24 dev eth1 proto kernel scope link src 198.51.100.20

What it means: Replies sourced from 192.0.2.0/24 may be routed out eth1 via a different gateway. Classic asymmetry trigger.

Decision: Align the intended source subnet with the intended egress, or SNAT/mark traffic so forward and return match your security boundaries.

Task 4: Check reverse-path filtering (rp_filter)

cr0x@server:~$ sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth0.rp_filter
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1

What it means: Strict reverse-path filtering is enabled. If packets arrive on an interface that the kernel wouldn’t use to reach the source, they can be dropped.

Decision: If you have legitimate asymmetric paths (multi-homing, VRFs, tunnels), set rp_filter to loose mode (2) or disable per-interface after risk review.

Task 5: Observe conntrack state drops via nftables counters

cr0x@server:~$ sudo nft list chain inet filter input
table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;
		ct state established,related accept
		ct state invalid counter packets 184 bytes 11040 drop
		iif "lo" accept
		tcp dport 22 accept
	}
}

What it means: 184 packets hit the INVALID conntrack rule and were dropped. That’s a smoking gun when “random” connections fail.

Decision: Investigate why packets are INVALID: asymmetric routing, mid-flow reroutes, NAT issues, or missing conntrack helpers. Don’t just accept INVALID blindly without understanding.

Task 6: Watch conntrack events live (identify resets/asymmetry symptoms)

cr0x@server:~$ sudo conntrack -E -o timestamp,extended
[1707052741.122334] [NEW] tcp      6 120 SYN_SENT src=192.0.2.20 dst=203.0.113.10 sport=43122 dport=443 [UNREPLIED] src=203.0.113.10 dst=192.0.2.20 sport=443 dport=43122
[1707052744.127991] [DESTROY] tcp  6 120 src=192.0.2.20 dst=203.0.113.10 sport=43122 dport=443 [UNREPLIED] src=203.0.113.10 dst=192.0.2.20 sport=443 dport=43122

What it means: SYN went out, but no reply was seen (UNREPLIED). That points to return path problems or filtering of SYN-ACK.

Decision: Capture on the server-side interface and upstream firewall/LB to locate where the SYN-ACK dies.

Task 7: Capture the failing flow on the client (tcpdump)

cr0x@server:~$ sudo tcpdump -ni eth0 host 203.0.113.10 and tcp port 443
14:13:01.000001 IP 192.0.2.20.43122 > 203.0.113.10.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 1 ecr 0], length 0
14:13:02.000314 IP 192.0.2.20.43122 > 203.0.113.10.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 2 ecr 0], length 0

What it means: SYN retransmits. No SYN-ACK coming back to this interface.

Decision: Either the SYN never arrived, or the SYN-ACK is returning via a different path. Now capture on the server side too.

Task 8: Capture on the server: did it receive SYN and send SYN-ACK?

cr0x@server:~$ sudo tcpdump -ni eth0 host 192.0.2.20 and tcp port 443
14:13:01.010010 IP 192.0.2.20.43122 > 203.0.113.10.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 1 ecr 0], length 0
14:13:01.010200 IP 203.0.113.10.443 > 192.0.2.20.43122: Flags [S.], seq 2000, ack 1001, win 65160, options [mss 1460,sackOK,TS val 9 ecr 1], length 0

What it means: Server is replying. The reply is leaving the server interface.

Decision: The drop is between server and client on the return path. Look at routing, firewalls, NAT, and ECMP between those two points.

Task 9: Validate return route from server to client IP

cr0x@server:~$ ip route get 192.0.2.20
192.0.2.20 via 203.0.113.1 dev eth1 src 203.0.113.10 uid 0
    cache

What it means: Server will route replies to the client via eth1, not eth0. That’s asymmetry by design or accident.

Decision: If your firewall/NAT/LB expects both directions via eth0, fix server routing (policy route, default route, or source-based routing) or fix the middle to tolerate asymmetry (state sync, DSR design, etc.).

Task 10: Check ECMP next-hops on a router (example via FRR/vtysh)

cr0x@server:~$ sudo vtysh -c "show ip route 203.0.113.0/24"
Routing entry for 203.0.113.0/24
  Known via "bgp", distance 20, metric 0
  * 198.51.100.2, via swp1, weight 1
  * 198.51.100.3, via swp2, weight 1

What it means: Two equal-cost next-hops exist. Different flows can take different upstream devices, which may not share session/NAT state.

Decision: If the upstream devices are stateful and not clustered with state sync, either make them state-sharing or remove ECMP for the affected prefix/traffic class.

Task 11: Confirm firewall “no session” drops (Linux host acting as firewall)

cr0x@server:~$ sudo journalctl -k -g "IN=.*OUT=.*STATE=INVALID" -n 5
Feb 04 14:12:59 fw1 kernel: [UFW BLOCK] IN=eth2 OUT= MAC=... SRC=203.0.113.10 DST=192.0.2.20 LEN=60 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=443 DPT=43122 WINDOW=65160 RES=0x00 ACK SYN URGP=0 STATE=INVALID

What it means: A SYN-ACK (ACK SYN) arrived but was considered INVALID by conntrack, and was blocked.

Decision: Treat as asymmetry until proven otherwise. Find where the corresponding SYN went (different interface or different firewall node), then enforce symmetric routing or enable proper state sync.

Task 12: Compare forward vs reverse path with traceroute from both ends

cr0x@server:~$ traceroute -n 203.0.113.10
 1  192.0.2.1  0.312 ms
 2  198.51.100.9  1.102 ms
 3  203.0.113.10  2.004 ms

cr0x@server:~$ traceroute -n 192.0.2.20
 1  203.0.113.1  0.401 ms
 2  198.51.100.10  1.220 ms
 3  192.0.2.20  2.331 ms

What it means: Paths are different even at hop 2. If hop 2 includes stateful devices (firewall pair, NAT gateways), you have a return-path mismatch.

Decision: Identify which hop is the state boundary and correct routing policy so both directions traverse the same stateful cluster member (or make the state shared).

Task 13: Check ARP/neighbor and L2 adjacency (asymmetry can be L2 too)

cr0x@server:~$ ip neigh show dev eth0 | head
192.0.2.1 lladdr 00:11:22:33:44:55 REACHABLE
192.0.2.254 lladdr 00:aa:bb:cc:dd:ee STALE

What it means: You might have multiple gateways, VRRP, or HSRP. If gateway MAC flips or you have multiple active routers unexpectedly, flows may return via a different L3 hop.

Decision: Validate FHRP configuration and ensure only the intended gateway answers for the virtual IP, or ensure consistent hashing/pinning across the redundant pair.

Task 14: Validate MSS/MTU to rule out “not asymmetry”

cr0x@server:~$ ping -M do -s 1472 203.0.113.10 -c 2
PING 203.0.113.10 (203.0.113.10) 1472(1500) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500

--- 203.0.113.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss

What it means: This host can’t send 1500-byte packets with DF set. That’s local MTU, not necessarily path MTU, but it’s a quick signal to verify MTU end-to-end.

Decision: If MTU differs across asymmetric paths, you can see “random” stalls on larger payloads. Fix MTU consistency, clamp MSS where appropriate, and don’t blame asymmetry alone.

Three corporate mini-stories from the trenches

Mini-story 1: The incident caused by a wrong assumption

They had a “simple” design: two firewalls in HA, two core routers, and ECMP everywhere because bandwidth is expensive and diagrams are cheap. The application team complained about sporadic login failures. Not total failure. Just enough to make customers question whether the company was solvent.

The network team’s assumption was reasonable on paper: “Both firewalls are in the same cluster, so state is shared.” In reality, state sync was enabled only for some traffic classes, and certain zones weren’t included due to an old performance concern. Nobody remembered because the engineer who set it up had moved on, as engineers do.

The failure mode was brutal: SYN went out through Firewall A, SYN-ACK came back through Firewall B, and B didn’t have state for that zone. It dropped the SYN-ACK. TCP retransmitted. Sometimes the hash landed on the “correct” member and it worked, which made it look like an application bug.

The fix wasn’t heroic. They narrowed ECMP to keep both directions pinned through the same firewall member for that prefix, and then corrected state-sync coverage. The postmortem’s real lesson was not “enable state sync.” It was “stop assuming HA means shared state for every path.” HA without shared state is just two devices taking turns disappointing you.

Mini-story 2: The optimization that backfired

A different company wanted to reduce latency for an internal API consumed by thousands of services. Someone proposed a “smart routing” change: send east-west traffic through the closest exit router and let return traffic take the shortest path back. The proposal sounded modern: fewer hops, better utilization, more “resilience.”

The problem: between those routers lived an IDS appliance configured in inline mode for only one direction of the path, because the original design assumed symmetry and placed it accordingly. After the change, half the flows were inspected one way, while return traffic bypassed inspection and hit a stateful ACL on a different device. That ACL dropped packets it didn’t expect because it never saw the initial session setup.

Symptoms were classic “randomness”: only some microservices failed, only some pods, only some times. The team spent days chasing CPU throttling and garbage collection. The network change was “too small to matter,” which is exactly how you get outages.

They rolled back the optimization and reintroduced it later with a proper design: either ensure inspection devices see both directions, or make inspection stateless/observational rather than inline. Latency improved eventually. The backfire lesson was sharper: do not optimize paths in a stateful network without mapping every stateful dependency. Your packets are not just data; they’re a narrative, and middleboxes are nosy editors.

Mini-story 3: The boring but correct practice that saved the day

A third org ran a multi-region setup with anycast front doors and regional firewalls. They had a policy: every change that could influence routing required two artifacts—an annotated path diagram and a “state boundary list” naming every NAT/firewall/LB/conntrack point.

It sounded bureaucratic. Engineers complained. But during a provider incident, traffic to one region shifted and return paths started doing odd things. The on-call pulled up the state boundary list and immediately asked the right question: “Which device expects to see both SYN and SYN-ACK?”

They already had counters exported from the firewalls for “no session” drops, plus conntrack INVALID drops on certain Linux gateways. Those dashboards lit up within minutes. Instead of debating theories, they pinned routes for the affected prefixes so both directions traversed the same firewall cluster member and disabled a strict uRPF policy temporarily for a known asymmetric segment.

No heroics. No magic. Just a team that had pre-decided what mattered and instrumented it. The incident still happened—networks are networks—but it ended fast, with minimal collateral damage. Boring is good when production is on fire.

Common mistakes: symptoms → root cause → fix

This section is intentionally specific. If you recognize the symptom, act like it’s guilty until proven innocent.

1) “Some clients can’t connect, but others are fine”

Symptom: Same service IP/port; certain source subnets fail more.

Root cause: different ingress/egress routing policy by source; return traffic crosses a different firewall/NAT path.

Fix: enforce symmetric routing for that service (PBR, VRF alignment, or route pinning); if using ECMP, ensure stateful devices share state or avoid ECMP across them.

2) “Retries work” (especially after a few seconds)

Symptom: first connection attempt fails, second succeeds; user perceives slowness.

Root cause: ECMP hash selects different next-hop; one path hits a device that drops out-of-state packets.

Fix: remove the broken path from ECMP set; fix state sync; add consistent hashing on the correct tuple; verify return path symmetry.

3) “Firewall shows invalid state drops”

Symptom: logs like “no session,” “invalid state,” “SYN-ACK out of state.”

Root cause: asymmetry across firewall cluster members; or asymmetric bypass of firewall in one direction.

Fix: ensure both directions pass through the same firewall member (session affinity / symmetric routing) or use a properly configured state-sharing cluster. Stop splitting flows across independent stateful nodes.

4) “Only one host is broken; the rest are fine”

Symptom: a single VM/node has intermittent connectivity; moving workload fixes it.

Root cause: rp_filter strict mode with multi-homing/tunnels; local policy routing mismatch; incorrect source IP selection.

Fix: set rp_filter=2 (loose) on relevant interfaces; correct ip rules/routes; pin source IP selection with routing policy; validate that replies exit the intended interface.

5) “Health checks are green, users are not”

Symptom: load balancer says backends are fine; real traffic fails intermittently.

Root cause: health checks originate from a different subnet/path (symmetric) than user traffic (asymmetric); or DSR return path bypasses critical devices.

Fix: make health checks representative (source IP, path, MTU); align LB mode (SNAT vs DSR) with routing/security expectations; ensure return path correctness.

6) “It smells like MTU, but only sometimes”

Symptom: large transfers stall; small requests okay; intermittent based on client or time.

Root cause: asymmetric paths with different MTUs or different ICMP filtering policies; PMTUD fails on one path but not another.

Fix: normalize MTU across paths; permit needed ICMP; clamp MSS at edges when you must; validate with DF pings and captures.

7) “We enabled strict anti-spoofing and now things are flaky”

Symptom: new drops after enabling uRPF or kernel rp_filter; asymmetric segments break.

Root cause: strict reverse-path checks assume symmetric routing.

Fix: use loose mode where asymmetry is legitimate; redesign routing to be symmetric for stateful/security boundaries; document exceptions rather than pretending they don’t exist.

Checklists / step-by-step plan

Step-by-step: diagnose a suspected asymmetric routing incident

Pick a single failing flow. Record src/dst IP, ports, protocol, time window.
Check client-side tcpdump. Do you see SYN leaving? Do you see SYN-ACK returning?
Check server-side tcpdump. Did SYN arrive? Did server reply?
Compare ip route get on both ends. Confirm expected egress interface and gateway for each direction.
Check policy routing (ip rule) on both ends. Especially if multi-homed, tunneled, or using multiple VRFs.
Inspect rp_filter settings. If strict, consider it guilty until proven innocent.
Inspect conntrack INVALID counters/logs. On firewalls, NAT gateways, and Linux middleboxes.
Identify ECMP sets for affected prefixes. If multiple next-hops exist, test by pinning temporarily.
Validate state sync in firewall/NAT clusters. Verify it covers the right zones/VRFs and isn’t “mostly enabled.”
Re-test with forced symmetry. Pin route, disable one path, or adjust policy for a controlled experiment.
Fix permanently. Decide: enforce symmetry, or ensure state sharing / stateless design.
Instrument it. Export counters/logs so next time you see it in minutes, not hours.

Prevention checklist: keep asymmetry from becoming an outage

Inventory stateful devices in every critical path (firewall, NAT, LB, IDS/IPS, conntrack hosts).
Decide where symmetry is required and enforce it with routing policy and design, not hope.
Don’t ECMP across independent stateful nodes unless you have proven state sharing at scale.
Standardize rp_filter policy and document exceptions for multi-homed/tunnel nodes.
Make health checks realistic (same source network, same path characteristics).
Export the right signals: INVALID drops, “no session” counters, conntrack table usage, and path change events.

FAQ

1) Is asymmetric routing always a problem?

No. It’s normal in many networks. It becomes a problem when a stateful device expects to see both directions of a flow (firewall, NAT, conntrack, some LBs/IDS).

2) Why does it fail only for some connections?

ECMP and load distribution are per-flow. One 5-tuple hashes to a “good” path; another hashes to a “bad” one. Retries change source ports, changing the hash, which looks like randomness.

3) How can I prove return traffic is taking a different path?

Use packet capture on both ends and compare traceroutes from both sides. If the server sends SYN-ACK and the client never sees it, the return path is the crime scene.

4) Does TCP automatically handle asymmetric routing?

TCP doesn’t care about symmetry. TCP cares about packets arriving. Middleboxes care about symmetry because they’ve inserted themselves into the session story.

5) What’s the difference between asymmetric routing and MTU blackholing?

MTU blackholing usually correlates with packet size (small works, large fails). Asymmetry correlates with path/state boundaries and often shows “invalid state” or missing SYN-ACK.

6) How does rp_filter relate to asymmetry?

rp_filter performs a reverse-route lookup and drops packets if they arrive on an interface the kernel wouldn’t use to reach the source. Strict mode breaks legitimate multi-homing and asymmetric return paths.

7) We have an HA firewall pair. Why does asymmetry still hurt?

HA doesn’t guarantee shared session state for every traffic class, VRF, or zone. Also, ECMP can spray flows across members in ways the cluster design didn’t anticipate.

8) What’s the cleanest fix: enforce symmetry or make devices tolerate asymmetry?

Enforcing symmetry is usually simpler and more predictable. Tolerating asymmetry requires robust state sharing or removing stateful inline dependencies (harder, but sometimes the right long-term move).

9) Can Kubernetes cause asymmetric routing?

Yes. Node-level SNAT, overlay networks, multiple CNI interfaces, and externalTrafficPolicy/local can all influence return paths. The diagnosis approach is the same: pick a flow, capture both ends, check policy routing and SNAT points.

10) What should I alert on to catch this early?

Alert on firewall “no session/invalid state” counters, conntrack INVALID drops, sudden increases in TCP retransmits, and route/next-hop churn for critical prefixes.

Conclusion: practical next steps

Asymmetric routing isn’t spooky. It’s just invisible until you force it to show itself. The trick is to stop treating “random drops” as a property of the universe and start treating them as a property of state.

If you want a reliable network:

Map your state boundaries and assume they are fragile until proven robust.
Make routing policy explicit where symmetry is required. Don’t let ECMP and “smart” defaults decide for you.
Instrument the boring signals (INVALID drops, no-session drops, conntrack behavior) so you can diagnose in minutes.
Test changes with realistic flows, not just pings and health checks.

One guiding idea, widely attributed in operations culture, is this: “Hope is not a strategy” — (paraphrased idea, often cited in engineering/ops contexts). Asymmetric routing is where hope goes to die. Be explicit. Be measurable. And when the outage hits, debug a single flow like it personally offended you.