Docker “No route to host” from containers: routing and iptables fixes that stick

Fast diagnosis playbook

When containers throw No route to host, don’t start by rebooting Docker. Don’t start by reinstalling Docker. Start by figuring out which layer is lying.
Here’s the order that finds the bottleneck quickly.

1) Confirm the failure mode from inside the container

• If ping says “Network is unreachable,” you have a routing problem in the container namespace.
• If ping says “No route to host,” the kernel believes a route exists but can’t deliver (often ARP/neighbor, firewall reject, or routing + rp_filter).
• If TCP connects time out, suspect firewall drop, missing NAT/masquerade, or return-path issues.

2) Compare container namespace routes to the host routes

Containers usually have a default route via the Docker bridge gateway (often 172.17.0.1). If the host has special routes (policy routing, static routes
to RFC1918 segments, VPN interfaces), the container might not inherit what you think it does.

3) Check forwarding and filter policy on the host

The container’s packets leave its namespace and hit the host’s FORWARD chain. If FORWARD is DROP (common with hardened baselines), your container can have perfect
routes and still go nowhere.

4) Check NAT expectations

If the destination is outside the Docker bridge subnet, Docker typically relies on NAT (MASQUERADE) unless you’re using routed networking. Missing NAT is a classic
“works on host, fails in container” scenario.

5) Check rp_filter if you have asymmetric paths

If traffic leaves via one interface (say, a VPN) and returns via another (say, the default gateway), strict reverse path filtering will drop it. This is painfully
common in corporate networks with split tunnels and internal routing.

What “No route to host” actually means (and what it does not)

The phrase looks like “routing table missing route,” but Linux uses it for several situations. The kernel may return EHOSTUNREACH (“No route to host”)
when:

• A route exists but the next-hop is unreachable (neighbor/ARP resolution failure on L2).
• ICMP host unreachable comes back from a router and is reflected to the socket.
• A firewall rule explicitly rejects with icmp-host-prohibited or similar.
• Policy routing sends the packet into a black hole or unreachable route type.
• rp_filter drops return traffic so aggressively that the stack behaves like the path is broken.

It does not reliably mean “you forgot to add a route.” Sometimes you have a route and the network still refuses to cooperate. Linux is honest, but it’s not
always chatty.

One practical quote that holds up in incident rooms: Hope is not a strategy. — General Gordon R. Sullivan.
When the error is “No route,” hope-based debugging is particularly expensive.

How container traffic really leaves the box (bridge mode)

Most Docker hosts still run the default bridge network (docker0) or a user-defined bridge. Inside a container you get a veth pair: one end in the
container namespace (often eth0), one end on the host (named like vethabc123). The host bridges that into docker0.

From there packets hit normal Linux routing on the host, plus iptables/nftables. That’s the critical point: container networking is not “Docker magic,” it’s Linux
networking with extra rules.

In bridge mode, outbound internet/internal access usually depends on NAT:

• Container source IP: 172.17.x.y
• Host MASQUERADE rewrites it to the host’s egress IP
• Return traffic comes back to host and is de-NATed to the container

If you expect routed networking instead (no NAT), you need to provide real routes in your upstream network to the container subnets, and you must allow forwarding.
Many teams accidentally build a “half-routed, half-NAT” setup that works until it doesn’t.

Interesting facts and historical context (why this problem keeps coming back)

1. Linux network namespaces landed in the mainline kernel in 2008–2009. Containers are largely namespaces plus cgroups; Docker didn’t invent the packet path.
2. Docker originally leaned heavily on iptables to stitch networking together because it was ubiquitous and scriptable; nftables arrived later and complicated the story.
3. The default Docker bridge subnet (172.17.0.0/16) collides with real corporate networks far more often than anyone wants to admit.
4. The FORWARD chain default policy being DROP became more common as security baselines hardened. That’s great for hosts, until you forget containers are “forwarded” traffic.
5. firewalld and ufw became popular as “friendlier” firewall managers; both can override or reorder Docker’s rules if you don’t integrate them explicitly.
6. Reverse path filtering (rp_filter) was designed to reduce IP spoofing. In modern multi-homed servers (VPNs, multiple uplinks), it can punish legitimate asymmetric routing.
7. conntrack (connection tracking) made stateful firewalling practical at scale, but it also introduced failure modes: table exhaustion looks like random networking flakiness.
8. As cloud networking grew, policy routing (multiple routing tables, rules by source) became common. Containers rarely match the assumptions baked into those rules.

Joke #1: Docker networking is like office Wi‑Fi—when it breaks, everyone suddenly becomes a network engineer for 20 minutes.

Practical tasks: commands, outputs, and what decision to make

Below are real tasks I’d run during an incident. Each one includes a command, representative output, what it means, and the next decision. Run them in order until you
find the lie. You’ll fix the problem faster and you’ll know what you fixed.

Task 1: Reproduce from inside the container (don’t guess)

cr0x@server:~$ docker exec -it app1 sh -lc 'ip route; echo ---; ping -c1 -W1 10.20.30.40; echo ---; nc -vz -w2 10.20.30.40 5432'
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 scope link  src 172.18.0.12
---
PING 10.20.30.40 (10.20.30.40): 56 data bytes
ping: sendto: No route to host
---
nc: connect to 10.20.30.40 port 5432 (tcp) failed: No route to host

Meaning: The container believes it has a default route, but the kernel can’t deliver packets to that destination. This is not “no default gateway.”

Decision: Shift attention to the host: forwarding, bridge, firewall reject, neighbor resolution, or host routing policy.

Task 2: Confirm the Docker network and gateway mapping

cr0x@server:~$ docker inspect -f '{{.Name}} {{range .NetworkSettings.Networks}}{{.NetworkID}} {{.IPAddress}} gw={{.Gateway}}{{end}}' app1
/app1 9c2d3a7f6c8a 172.18.0.12 gw=172.18.0.1

Meaning: You now know the container subnet (172.18.0.0/16) and gateway IP.

Decision: Look at the corresponding bridge and veth on the host.

Task 3: Identify the bridge interface and its state

cr0x@server:~$ ip -br link show | egrep 'docker0|br-'
docker0           UP             5a:3f:2b:aa:bb:cc <BROADCAST,MULTICAST,UP,LOWER_UP>
br-9c2d3a7f6c8a    UP             02:42:7c:8e:11:22 <BROADCAST,MULTICAST,UP,LOWER_UP>

Meaning: User-defined network created a bridge br-9c2d3a7f6c8a, and it’s up.

Decision: Verify the bridge has the expected IP and routes.

Task 4: Verify bridge IP addresses (gateway must exist on host)

cr0x@server:~$ ip -4 addr show br-9c2d3a7f6c8a
8: br-9c2d3a7f6c8a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-9c2d3a7f6c8a
       valid_lft forever preferred_lft forever

Meaning: Gateway exists and is correct.

Decision: If the gateway were missing, you’d restart the Docker network or fix daemon startup. Since it’s present, move on.

Task 5: Check host forwarding sysctls

cr0x@server:~$ sysctl net.ipv4.ip_forward net.ipv4.conf.all.forwarding
net.ipv4.ip_forward = 0
net.ipv4.conf.all.forwarding = 0

Meaning: Host is not forwarding IPv4 packets. Containers can talk to the host, but not through it.

Decision: Enable forwarding persistently (not with a one-off sysctl that vanishes on reboot).

Task 6: Turn on forwarding (persistent) and verify

cr0x@server:~$ sudo install -d -m 0755 /etc/sysctl.d
cr0x@server:~$ printf '%s\n' 'net.ipv4.ip_forward=1' 'net.ipv4.conf.all.forwarding=1' | sudo tee /etc/sysctl.d/99-docker-forwarding.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
cr0x@server:~$ sudo sysctl --system | tail -n 5
* Applying /etc/sysctl.d/99-docker-forwarding.conf ...
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1

Meaning: Forwarding is now enabled and will survive reboots.

Decision: Retest from the container. If still broken, you’re into firewall/NAT/rp_filter territory.

Task 7: Inspect FORWARD chain policy and Docker’s chains (iptables)

cr0x@server:~$ sudo iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-9c2d3a7f6c8a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-9c2d3a7f6c8a -j DOCKER
-A FORWARD -i br-9c2d3a7f6c8a ! -o br-9c2d3a7f6c8a -j ACCEPT
-A FORWARD -i br-9c2d3a7f6c8a -o br-9c2d3a7f6c8a -j ACCEPT

Meaning: Default DROP is fine only if accept rules cover your traffic. The -i br-... ! -o br-... ACCEPT should allow container egress.

Decision: If those ACCEPT rules are missing, Docker’s rule insertion is being blocked/overridden. Investigate firewalld/ufw/nftables integration.

Task 8: Check DOCKER-USER chain (your org’s “security rules” often live here)

cr0x@server:~$ sudo iptables -S DOCKER-USER
-N DOCKER-USER
-A DOCKER-USER -i br-9c2d3a7f6c8a -d 10.0.0.0/8 -j REJECT --reject-with icmp-host-prohibited
-A DOCKER-USER -j RETURN

Meaning: Someone is explicitly rejecting container traffic to 10/8 with an ICMP host-prohibited. Many apps report that as “No route to host.”

Decision: Remove/adjust the reject rule, or add an allow exception for the actual destination subnet/ports.

Task 9: Confirm NAT/MASQUERADE exists for the container subnet

cr0x@server:~$ sudo iptables -t nat -S POSTROUTING | egrep 'MASQUERADE|172\.18\.0\.0/16'
-A POSTROUTING -s 172.18.0.0/16 ! -o br-9c2d3a7f6c8a -j MASQUERADE

Meaning: NAT is configured for egress (anything leaving not via the bridge gets masqueraded).

Decision: If missing, either Docker isn’t managing iptables, or another firewall manager flushed nat rules. Decide whether you want NAT or routed networking.

Task 10: Verify Docker’s iptables setting (the “why didn’t it program rules?” check)

cr0x@server:~$ docker info --format '{{json .SecurityOptions}} {{.Name}}' | head -n 1
["name=seccomp,profile=default","name=cgroupns"] server

cr0x@server:~$ sudo cat /etc/docker/daemon.json 2>/dev/null || echo "no /etc/docker/daemon.json"
{
  "iptables": false
}

Meaning: Docker is configured not to touch iptables. That can be intentional, but you now own all rules for NAT and forwarding.

Decision: Either set "iptables": true (and manage integration properly), or implement full equivalent rules yourself and make them persistent.

Task 11: Inspect route selection to the destination from the host

cr0x@server:~$ ip route get 10.20.30.40
10.20.30.40 via 10.20.0.1 dev tun0 src 10.20.10.5 uid 0
    cache

Meaning: Host routes that destination via tun0 (VPN). Container traffic will also likely egress via tun0 after forwarding/NAT rules.

Decision: If VPN is involved, immediately suspect rp_filter and policy routing mismatches. Continue with rp_filter and rules checks.

Task 12: Check policy routing rules (containers can fall into the wrong table)

cr0x@server:~$ ip rule show
0:      from all lookup local
100:    from 10.20.10.5 lookup vpn
32766:  from all lookup main
32767:  from all lookup default

Meaning: There’s a source-based rule: only traffic sourced from 10.20.10.5 uses the vpn table. NAT may rewrite source to something else.

Decision: Decide if you want NAT (container traffic becomes host IP, matches rule) or routing without NAT (container source is 172.18/16, won’t match).
If the latter, you must add ip rule for container subnets or route them properly.

Task 13: Check rp_filter settings (global and interface)

cr0x@server:~$ sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.default.rp_filter net.ipv4.conf.tun0.rp_filter
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.tun0.rp_filter = 1

Meaning: Strict rp_filter is enabled. If the return path for forwarded/NATed traffic doesn’t match the kernel’s idea of “best route,” packets get dropped.

Decision: For multi-homed/VPN hosts, set rp_filter to loose mode (2) at least on involved interfaces, persistently.

Task 14: Watch counters while you reproduce (iptables rule hit counts)

cr0x@server:~$ sudo iptables -L DOCKER-USER -v -n --line-numbers
Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in              out  source        destination
1      120  7200 REJECT     all  --  br-9c2d3a7f6c8a  *   0.0.0.0/0     10.0.0.0/8 reject-with icmp-host-prohibited
2      900 54000 RETURN     all  --  *               *   0.0.0.0/0     0.0.0.0/0

Meaning: The reject rule is actively matching traffic (pkts/bytes increasing). This is not theoretical.

Decision: Change that rule. Don’t touch Docker. Don’t touch routes. Fix the policy causing the reject.

Task 15: Confirm neighbor/ARP behavior on the host bridge

cr0x@server:~$ ip neigh show dev br-9c2d3a7f6c8a | head
172.18.0.12 lladdr 02:42:ac:12:00:0c REACHABLE

Meaning: Host sees the container’s MAC and neighbor entry is healthy.

Decision: If you see FAILED/INCOMPLETE repeatedly, suspect L2 issues: bridge misconfig, veth flapping, or weird MTU problems.

Task 16: Capture one packet path (tcpdump that answers one question)

cr0x@server:~$ sudo tcpdump -ni br-9c2d3a7f6c8a host 10.20.30.40
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br-9c2d3a7f6c8a, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:21:18.019393 IP 172.18.0.12.48522 > 10.20.30.40.5432: Flags [S], seq 1234567890, win 64240, options [mss 1460,sackOK,TS val 123 ecr 0,nop,wscale 7], length 0

Meaning: Packet leaves container and reaches the bridge. Next, check egress interface (e.g., eth0 or tun0) to see if it’s forwarded/NATed.

Decision: If you see it on the bridge but not on egress, host firewall/forwarding is blocking. If you see it egress but no reply, it’s routing/return path upstream.

Joke #2: The fastest way to find a broken firewall rule is to say “it can’t be the firewall” out loud in the incident channel.

iptables vs nftables vs firewalld: who’s actually in charge

The most durable “no route to host” outages are political: multiple systems think they own the firewall. Docker writes rules. firewalld writes rules. Your security agent
writes rules. Someone adds nftables directly because they read a blog post. Then an OS upgrade changes the backend from iptables-legacy to iptables-nft and everyone
pretends nothing changed.

Know your backend: iptables-legacy or iptables-nft

On many modern distros, iptables is a compatibility wrapper over nftables. Docker has improved here, but “improved” is not the same as “immune to local
policy.”

cr0x@server:~$ sudo update-alternatives --display iptables 2>/dev/null | sed -n '1,12p'
iptables - auto mode
  link best version is /usr/sbin/iptables-nft
  link currently points to /usr/sbin/iptables-nft

Meaning: You’re using the nft backend. Rules may be visible via nft list ruleset.

Decision: If you debug with iptables but production enforcement is nftables in a different table, you’ll chase ghosts. Verify with nft too.

cr0x@server:~$ sudo nft list ruleset | sed -n '1,40p'
table inet filter {
  chain forward {
    type filter hook forward priority filter; policy drop;
    jump DOCKER-USER
    ct state related,established accept
  }
}

Meaning: nftables is enforcing a forward policy drop. Docker might still be inserting rules, but your base policy matters.

Decision: Ensure Docker’s accepts are present and ordered, or explicitly allow the bridge subnets in your managed firewall layer.

firewalld integration: don’t fight it, configure it

firewalld is not evil. It’s just confident. If firewalld is running and you also expect Docker to manage iptables freely, you need an explicit plan. The plan is usually:
let Docker handle its chains, and ensure your zones and policies allow forwarding from Docker bridges.

cr0x@server:~$ sudo systemctl is-active firewalld
active

cr0x@server:~$ sudo firewall-cmd --get-active-zones
public
  interfaces: eth0

Meaning: firewalld is active and managing at least eth0. Docker bridges may not be in any zone, or may default to a restrictive zone.

Decision: Put Docker bridge interfaces into a trusted/allow-forward zone (or create a dedicated zone) and make it permanent.

cr0x@server:~$ sudo firewall-cmd --permanent --zone=trusted --add-interface=br-9c2d3a7f6c8a
success
cr0x@server:~$ sudo firewall-cmd --reload
success

Meaning: The bridge is now in the trusted zone across reboots/reloads.

Decision: Re-test container connectivity. If you have compliance constraints, don’t use trusted; use a custom zone with explicit rules.

Routing fixes that stick (not just “ip route add”)

The most tempting fix is the least durable: add a static route on the host, see it work, and declare victory. Then a reboot happens. Or NetworkManager re-applies
profiles. Or the VPN client reconfigures routes. Your incident returns, but now it’s at 3 a.m.

Fix category A: Your container subnet collides with the real network

If your corporate network uses 172.16/12 heavily, Docker’s defaults are a landmine. Routing gets ambiguous. Packets that should go to a real remote subnet may be
interpreted as local on the bridge, or vice versa.

Best practice: choose a site-specific container CIDR that you know won’t collide, and bake it into daemon configuration early.

cr0x@server:~$ sudo cat /etc/docker/daemon.json
{
  "bip": "10.203.0.1/16",
  "default-address-pools": [
    { "base": "10.203.0.0/16", "size": 24 }
  ]
}

Meaning: Docker bridge IP and user-defined networks will draw from 10.203.0.0/16 with /24 networks.

Decision: This is a disruptive change. Do it on fresh hosts or during a migration window; existing networks/containers may need recreation.

Fix category B: You need routed containers (no NAT)

Some environments hate NAT. Auditors want real source IPs. Network teams want to route container CIDRs like any other subnet. You can do that, but you must commit.

• Upstream routers must have routes to container subnets via the Docker host(s).
• The Docker host must allow forwarding and must not masquerade those subnets.
• Return traffic must be symmetric enough to avoid rp_filter drops.

If you try to “mostly route” and still have NAT rules around “just in case,” you’ll create intermittent behavior that makes you question your career choices.

Fix category C: Policy routing must include container sources

If your host uses multiple routing tables, containers aren’t special. They’re just additional source networks. Add rules intentionally.

cr0x@server:~$ sudo ip rule add from 172.18.0.0/16 lookup vpn priority 110
cr0x@server:~$ sudo ip rule show | sed -n '1,6p'
0:      from all lookup local
100:    from 10.20.10.5 lookup vpn
110:    from 172.18.0.0/16 lookup vpn
32766:  from all lookup main
32767:  from all lookup default

Meaning: Container-sourced traffic will now use the vpn table, matching the host’s intent.

Decision: Make it persistent with your distro’s network tooling (NetworkManager dispatcher scripts, systemd-networkd, or static scripts). One-off ip rules die on reboot.

Persistence patterns that don’t rot

• systemd-networkd: declare routes and routing policy in .network files.
• NetworkManager: use nmcli connection modify to add routes and rules to the profile.
• VPN clients: if they push routes, configure “route-up” hooks to also handle container sources, or disable pushed routes and manage centrally.

If you don’t know which tooling manages your routes, you don’t have a routing configuration. You have a routing vibe.

iptables/nftables fixes that stick

Persistent firewall fixes are about ownership. Pick one system to own the rules and integrate the others. The worst option is “Docker owns some, security agent owns
some, and we manually add a few during incidents.”

Option 1 (common): Let Docker manage its rules, but constrain with DOCKER-USER

Docker inserts its chains and jumps. The supported place for your policy is the DOCKER-USER chain. It’s evaluated before Docker’s own accept rules.
That’s where you add allowlists/denylists without getting into a fight with Docker’s rule generator.

If you use DOCKER-USER, be careful with REJECT vs DROP. REJECT produces immediate failures like “No route to host,” which is good for user experience but terrible for
misdiagnosis. DROP produces timeouts, which is worse for user experience but sometimes clearer for security posture. Pick intentionally and document it.

Option 2: Docker iptables disabled; you own everything

This is valid in locked-down environments, but then you must implement:

• Forwarding acceptance rules between bridge and egress interfaces
• NAT masquerade for container subnets (if you want NAT)
• Established/related acceptance for return traffic
• Isolation rules if you care about network segmentation between bridges

The upside: predictable policy. The downside: you are now Docker’s firewall engineer, whether you wanted the job or not.

Persist iptables rules properly

On Debian/Ubuntu, people often use iptables-persistent. On RHEL-like systems, firewalld/nftables is the usual path. The goal is: rules reappear after
reboot in the right order.

cr0x@server:~$ sudo iptables-save | sed -n '1,35p'
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -j RETURN
COMMIT

Meaning: This is what will be restored if you persist it. Also notice: order matters.

Decision: If your persistence mechanism restores rules before Docker starts, you may need a systemd unit ordering fix so Docker can re-create chains before policy applies (or your policy must tolerate absent chains).

NAT that survives firewall reloads

firewalld reloads can flush and reapply rules. If Docker’s NAT disappears after a reload, you’ll see “worked yesterday, broken after a change window.” If you’re using
firewalld, prefer configuring masquerade and forward policy via firewalld itself for the relevant zone(s), or ensure Docker’s integration is supported on your distro.

cr0x@server:~$ sudo firewall-cmd --zone=public --query-masquerade
no
cr0x@server:~$ sudo firewall-cmd --permanent --zone=public --add-masquerade
success
cr0x@server:~$ sudo firewall-cmd --reload
success
cr0x@server:~$ sudo firewall-cmd --zone=public --query-masquerade
yes

Meaning: Masquerading is enabled at the firewall manager level, so a reload won’t quietly remove egress NAT behavior.

Decision: Only do this if your environment expects NAT for that zone; don’t blindly enable masquerade on hosts that should be strictly routed.

rp_filter and asymmetric routing: the silent container killer

If you run a simple single-NIC host with a default gateway, you can mostly ignore rp_filter. Production systems rarely stay that simple. Add a VPN interface, add a
second uplink, add a VRF-like routing policy with ip rules, and strict rp_filter becomes a drop machine.

How it fails: container traffic enters the host on br-*, gets forwarded out tun0, and replies come back via eth0
(or vice versa). Strict rp_filter checks whether the source address of an incoming packet would be routed back out the same interface; if not, the packet can be dropped.
You see “No route to host” or timeouts. Your logs are quiet. Everyone blames Docker.

Loose mode is usually the correct compromise

rp_filter=2 (loose) verifies the source is reachable via some interface, not necessarily the one it arrived on. That’s generally acceptable for
servers in complex routing environments.

cr0x@server:~$ printf '%s\n' \
'net.ipv4.conf.all.rp_filter=2' \
'net.ipv4.conf.default.rp_filter=2' \
'net.ipv4.conf.tun0.rp_filter=2' \
| sudo tee /etc/sysctl.d/99-rpfilter-loose.conf
net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.default.rp_filter=2
net.ipv4.conf.tun0.rp_filter=2
cr0x@server:~$ sudo sysctl --system | tail -n 6
* Applying /etc/sysctl.d/99-rpfilter-loose.conf ...
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.tun0.rp_filter = 2

Meaning: Loose rp_filter is now persistent.

Decision: If you’re in a hostile network environment where spoofing is a major concern, consider alternatives: more explicit routing symmetry, or only relaxing rp_filter on specific interfaces involved in forwarding.

Three corporate mini-stories from the trenches

1) The incident caused by a wrong assumption: “The host can reach it, so containers can too.”

A platform team rolled out a new internal metrics pipeline. The collector ran in Docker, scraped services in several RFC1918 ranges, and shipped data to a central
cluster. On day one it worked in staging and a small production slice.

Then it went wide. A chunk of hosts started logging No route to host when the collector tried to reach a set of services behind a VPN interface. The host
itself could curl those services just fine. The team’s assumption hardened into a diagnosis: “Docker routing is broken.”

The reality: the host had policy routing rules that only applied to the host’s VPN source IP. Host-originated traffic used that source, matched the vpn table, and
went out the tunnel. Container traffic was being NATed sometimes, routed other times (depending on which node had which firewall baseline), and in the failure case it
didn’t match the rule. It went out the default gateway instead, hit a router that rejected it, and the collector reported “No route to host.”

The fix wasn’t “restart Docker.” It was adding a policy routing rule for the container subnets on the affected class of nodes, and making it persistent in the same
system that managed the VPN routes. After that, containers behaved like the host because the network policy actually applied to them.

Postmortem action item that mattered: whenever a host uses source-based routing, container CIDRs are treated as first-class sources in the routing design. Not special,
not an afterthought, not “someone else’s problem.”

2) The optimization that backfired: “Let’s turn off Docker’s iptables for performance.”

A security team and a performance-minded engineer agreed on a change: set Docker’s "iptables": false so Docker wouldn’t mutate the firewall, and replace
it with a centrally managed nftables ruleset. The pitch was clean: fewer moving parts, faster rule evaluation, better compliance.

The rollout looked fine on a handful of nodes running simple HTTP services. Weeks later, a different service class—containers that needed to reach internal databases
on non-default routes—started failing during a routine firewall reload. The apps logged timeouts and occasional No route to host. Engineers toggled
container restarts, host reboots, even replaced nodes. Symptoms moved around.

The backfire was subtle: the centrally managed ruleset had forward accepts, but NAT was incomplete for one of the user-defined bridge pools. Under certain source/dest
combinations, packets left with unroutable 172.x source addresses. Some upstream routers rejected quickly (surface as “no route”), others dropped (timeouts). After a
reload, the order of chains also changed and briefly placed a REJECT rule ahead of the accept.

The fix was not “turn iptables back on” (though that would have worked). They instead codified NAT and forward behavior for each container address pool in the
nftables configuration, added regression tests that validated rule presence after reload, and standardized bridge CIDRs across fleets to reduce drift.

Lesson: you can absolutely own the firewall yourself. But once you take the wheel from Docker, you don’t get to be surprised when you crash into NAT.

3) The boring but correct practice that saved the day: “We wrote a runbook and enforced invariants.”

A payments-adjacent team ran a fleet of Docker hosts with a mix of legacy services and new containers. Their environment included firewalld, host-based intrusion
prevention, and a corporate VPN client that sometimes updated routes. In other words: a perfect storm generator.

After one unpleasant outage, they implemented a dull practice: every node boot ran a health check that validated a handful of invariants—ip_forward=1,
FORWARD chain contains accepts for Docker bridges, NAT masquerade exists for each configured pool, rp_filter is loose on VPN interfaces, and Docker bridge CIDRs do not
overlap site routes. The check emitted a single line of status and failed the node out of rotation if any invariant was broken.

Months later, a base image update flipped a sysctl profile that disabled forwarding. Half the fleet would have become dead weight. Instead, the health check flagged
nodes immediately after reboot, before they took production traffic. The team adjusted the sysctl drop-in, rolled forward, and users never noticed.

Nobody got applause for “we verified sysctls.” That’s fine. Boring correctness is how you buy weekends.

Common mistakes: symptoms → root cause → fix

1) Symptom: “No route to host” only from containers; host works

Root cause: net.ipv4.ip_forward=0 or forwarding disabled via sysctl baseline.

Fix: Enable forwarding persistently with /etc/sysctl.d/*.conf and apply with sysctl --system.

2) Symptom: Immediate failure, not timeout; iptables counters rise on a REJECT

Root cause: DOCKER-USER chain contains a REJECT (often “block private networks from containers”).

Fix: Replace with an allowlist-based policy, or scope it tightly by subnet/port; verify hit counts.

3) Symptom: Works until firewalld reload; then containers lose outbound

Root cause: NAT/forward rules are being flushed; Docker’s rules not reinserted or overridden.

Fix: Integrate Docker bridges into firewalld zones and configure masquerade/forwarding in firewalld, or ensure Docker is allowed to manage rules.

4) Symptom: Only certain destinations fail (especially via VPN); intermittent “No route”

Root cause: Policy routing rules don’t match container source CIDR; traffic goes out wrong interface and gets rejected.

Fix: Add ip rule for container CIDR(s) to the correct routing table, persist via network manager.

5) Symptom: SYN leaves host, no replies; host curl works

Root cause: Missing MASQUERADE; upstream doesn’t know how to route 172/10 container subnet back.

Fix: Add NAT for container pool, or implement routed container subnets with upstream routes.

6) Symptom: Replies arrive on a different interface; conntrack shows drops; VPN involved

Root cause: Strict rp_filter drops asymmetric return traffic.

Fix: Set rp_filter to loose (2) for all/default and key interfaces, or make routing symmetric.

7) Symptom: Containers can reach some private subnets but not others; “Network is unreachable”

Root cause: Route collision (Docker bridge overlaps real network), causing wrong route selection.

Fix: Change Docker address pools (bip, default-address-pools) to non-overlapping ranges and recreate networks.

8) Symptom: After OS upgrade, rules appear in iptables but not enforced (or vice versa)

Root cause: Backend mismatch (iptables-legacy vs iptables-nft); debugging the wrong plane.

Fix: Confirm backend with alternatives; use nft list ruleset when nft is active; standardize across fleet.

Checklists / step-by-step plan

Step-by-step: Fix container “No route to host” in a durable way

1. Reproduce inside the container: run ip route, then a minimal TCP connect to the exact IP:port. Record whether it’s “No route” or timeout.
2. Identify container subnet and bridge: via docker inspect and ip link. Write down the bridge name and CIDR.
3. Verify host forwarding: check net.ipv4.ip_forward. If off, enable persistently with sysctl drop-in.
4. Check FORWARD policy and DOCKER-USER: look for DROP defaults without accept rules, and for explicit REJECTs. Use counters to prove hits.
5. Confirm NAT (if you expect NAT): verify POSTROUTING MASQUERADE for the container subnet. Decide: NAT or routed? Pick one.
6. Check routing to destination from host: ip route get <dest>. If it uses VPN or non-default path, check policy routing rules.
7. Check rp_filter: if multi-homed or VPN, set to loose where appropriate, persistently.
8. Integrate firewall manager: if firewalld/ufw is present, assign Docker bridges to a zone and configure masquerade/forwarding in that system.
9. Retest and packet-capture once: tcpdump on bridge and egress to confirm where packets stop.
10. Make it stick: encode changes in config management (sysctl.d, NetworkManager profiles, firewalld permanent config, docker daemon.json).

Operational checklist: invariants worth monitoring

• net.ipv4.ip_forward=1 on container hosts
• FORWARD chain accepts bridge-to-egress traffic and established return
• DOCKER-USER chain does not contain broad REJECTs without explicit allow exceptions
• NAT masquerade exists for each container pool when using bridge networking with NAT
• Container CIDRs do not overlap corporate/private subnets or VPN routes
• rp_filter set appropriately for multi-homed routing designs
• Single source of truth for firewall (Docker + DOCKER-USER, or central nftables/firewalld—but not a brawl)

FAQ

Why do I get “No route to host” instead of a timeout?

Because something is actively rejecting or declaring the host unreachable (iptables REJECT, ICMP unreachable from a router, neighbor failure). Timeouts usually mean DROP.

Containers can reach the internet but not a private subnet. What’s special about private ranges?

Private ranges often involve policy routing (VPN), explicit firewall policy, or overlapping CIDRs. Internet is usually just default route + NAT; private networks are where
your organization’s “special” starts.

Is disabling Docker’s iptables management a good idea?

Only if you’re ready to fully own NAT/forwarding/isolation rules and test them after reloads and upgrades. It’s fine in disciplined environments; it’s chaos in ad hoc ones.

What’s the quickest way to prove it’s the firewall?

Check DOCKER-USER and FORWARD counters while reproducing the failure. If counters tick on a REJECT/DROP rule, you have proof without philosophical debate.

Does changing the Docker bridge CIDR require recreating containers?

Usually yes. Existing networks and containers were created with the old pools. Plan a migration: drain workloads, recreate networks, redeploy.

How does rp_filter show up as a container problem?

Containers are forwarded traffic. If replies come back on an unexpected interface, strict rp_filter may drop them. The app experiences it as unreachable or timing out.

Is this different for macvlan or host networking?

Yes. macvlan bypasses the Docker bridge and NAT model and relies on L2 adjacency and upstream switching behavior; host networking bypasses namespace routing but still faces
host firewall/routing policy. “No route” causes change, but the debugging approach (routes, forwarding, firewall, rp_filter) stays relevant.

I’m on nftables. Should I stop using iptables commands entirely?

Use whichever matches enforcement. If iptables is the nft wrapper, iptables commands often work but can hide nft-specific structure. When in doubt, inspect both and
standardize fleet-wide to reduce confusion.

Why does it work right after Docker restart and then break later?

Because something else is reapplying firewall or sysctl policy after Docker starts (firewalld reload, security agent, cloud-init, config management). Fix ownership and ordering.

Conclusion: next steps that reduce future pain

“No route to host” from containers is rarely a Docker mystery and almost always a Linux networking truth you haven’t written down yet. The durable fix is not a magic
command; it’s making your routing and firewall intent explicit and persistent.

• Pick non-overlapping container CIDRs and standardize them early.
• Decide NAT vs routed containers and implement it consistently.
• Make forwarding, rp_filter posture, and policy routing rules explicit in persistent config.
• Choose a firewall owner (Docker+DOCKER-USER, or firewalld/nftables) and integrate instead of competing.
• Automate invariants checks so “works until reboot” stops being a recurring genre.

If you do only one thing: add a health check that validates forwarding, NAT presence, and DOCKER-USER policy after every boot and firewall reload. It’s boring. It’s correct.
It will save you later.