ZFS Replication Over WAN: Making Sends Fast on Slow Links

January 19, 2026 • February 3, 2026 • Read: 21 min • Views: 12

Was this helpful?

WAN replication is where ZFS stops being a neat filesystem feature and becomes an operational personality test. Locally, zfs send | zfs recv feels like a conveyor belt. Over a 50–200 ms link with packet loss and “helpful” middleboxes, it becomes a pinball machine: throughput collapses, CPU pegs, SSH stalls, and your RPO starts negotiating with reality.

This is a field guide for making ZFS sends fast on slow links without turning your storage platform into a science fair. It’s written for people who ship bytes for a living: SREs, storage engineers, and the unlucky on-call who inherited “simple replication” that only works when nobody is looking.

1. The WAN replication mental model

ZFS replication performance isn’t a single dial. It’s a pipeline with competing constraints: disk I/O on the sender, CPU for compression/encryption, memory and buffering, TCP congestion behavior across latency, SSH overhead, and write behavior on the receiver. If you only tune one component, you often just move the bottleneck somewhere else—and sometimes you make the new bottleneck harder to see.

The easiest way to stay sane is to treat replication like a production service with SLOs:

RPO: how much data you can afford to lose (drives snapshot frequency and send size).
RTO: how quickly you can restore (drives retention, testing, and whether you need zfs receive properties to be correct).
Budget: bandwidth, CPU headroom, and how much operational complexity you can tolerate at 3 AM.

Then define the pipeline in your head:

Snapshot selection: what changed since last time?
Send: ZFS reads blocks and emits a stream.
Transform: optional compression, optional encryption (either ZFS native or SSH).
Transport: TCP across a high-latency path.
Receive: ZFS writes blocks, updates metadata, commits TXGs.

In a LAN, disk and CPU dominate. In a WAN, latency and packet loss amplify every small inefficiency. A link with 100 ms RTT isn’t just “a bit slower than local”—it changes how TCP ramps up, how quickly buffers drain, and how punishing a single retransmit can be.

First joke (required, and earned): WAN replication is like moving apartments using a scooter—technically possible, but every missing strap becomes a life choice.

2. Interesting facts & historical context

Six to ten concrete context points that matter in practice:

ZFS send/receive is older than many “cloud-native” backup products. The original OpenSolaris ZFS design baked in snapshot replication as a first-class operation, not an add-on.
Incremental sends are block-based, not file-based. That’s why they can be spectacularly efficient—but also why certain workloads (VM images with churn) can generate surprisingly large deltas.
TCP throughput is bounded by bandwidth-delay product (BDP). If your socket buffers are smaller than BDP, you’re leaving capacity on the table even when everything else is perfect.
SSH’s defaults are conservative for interactive sessions. They’re not designed for sustained multi-hour bulk transfer, especially across high latency.
OpenZFS added resumable send support so an interrupted replication can continue without restarting from the beginning—one of the best “ops quality of life” features in the stack.
Compression has swung in and out of fashion. With modern CPUs, lightweight compression (like LZ4) is often “free enough” and can be the difference between saturating a link or crawling.
Recordsize and volblocksize decisions echo into replication. Big sequential blocks replicate efficiently; small random blocks can increase metadata and overhead—especially visible over WAN.
Replication is sensitive to receiver write behavior. A receiver pool with a strained SLOG, tiny ashift mismatch expectations, or heavy sync workloads can throttle the entire pipeline.

3. Baseline: what “slow” really means

Before tuning ZFS, figure out what your link can actually do. People routinely claim “we have a 1 Gbps circuit” and then replicate at 30–80 Mbps and blame ZFS. Half the time the circuit is real; the usable throughput isn’t—because QoS, VPN overhead, packet loss, or BDP misconfiguration quietly eats your lunch.

What you need to know:

RTT: 20 ms is a different planet from 120 ms.
Loss: 0.1% loss can crater TCP throughput on some paths.
MTU/MSS: fragmentation and blackholing are silent killers.
BDP: bandwidth × RTT; compare to socket buffer sizes.

Only after you understand the network do you interpret replication numbers. Otherwise you’re tuning a filesystem to compensate for a firewall that thinks large packets are a moral failing.

4. The replication pipeline: where speed goes to die

4.1 Snapshot strategy: fewer, smarter deltas

On WAN links, replication is a game of consistency and delta size. Many small snapshots reduce worst-case delta size and reduce risk of “one huge send” clogging the pipe—but too many snapshots increase metadata churn and management overhead.

Operationally, the winning pattern is:

Frequent snapshots for short RPO (e.g., 5–15 minutes) on critical datasets.
Less frequent snapshots for bulky/low-value datasets.
Retention that keeps at least one known-good baseline for every replication target.

4.2 Sender: read pattern, CPU, and ZFS send flags

zfs send can be fast enough to saturate NVMe locally and still be the wrong shape for WAN. The big levers:

Incremental sends: -i or -I, plus sane snapshot naming.
Raw sends: -w keeps blocks encrypted/compressed as stored (when dataset encryption is enabled) and avoids re-processing; often a big win for CPU.
Large blocks: replication benefits from bigger sequential reads; if your workload is inherently random, buffering matters more.
Include properties: -p or -R decisions affect correctness, not just speed—mistakes here create “fast” replications that don’t restore correctly.

4.3 Transform: compression and encryption choices

On slow links, compression often buys more than it costs—until it doesn’t. The constraint is CPU headroom at both ends. LZ4 is the default “probably safe” choice; heavy compression can bottleneck on CPU, and double-compressing can waste cycles for zero gain.

Encryption: you will encrypt over WAN. The only question is where. If you already use ZFS native encryption, zfs send -w (raw) lets you avoid decrypt/re-encrypt cycles. If you rely on SSH for encryption, tune SSH for bulk transfer and consider ciphers that are fast on your hardware.

4.4 Transport: latency, buffers, and why your throughput flatlines

TCP needs in-flight data to keep the pipe full. On a 200 Mbps link with 100 ms RTT, BDP is about 2.5 MB (200 megabits/sec ≈ 25 MB/sec; × 0.1 sec ≈ 2.5 MB). If your sender/receiver socket buffers are smaller than that, you physically cannot hit line rate.

Now add loss. TCP interprets loss as congestion, shrinks the window, and ramps back up. Across long RTTs, ramp-up is slow. This is why a link that “feels fine” for web traffic can be miserable for sustained replication.

4.5 Receiver: writes, TXGs, and the “my receive is slower than send” mystery

Receiver-side performance is often the hidden bottleneck. Common culprits:

Pool contention: other workloads causing random writes and sync pressure.
Sync behavior: datasets with sync=always or a stressed SLOG device can throttle commits.
Small recordsize: receiver has to do more metadata work for the same amount of user data.
ashift mismatch expectations: not a replication setting, but a pool design factor that affects write amplification.

5. Practical tasks (commands + interpretation)

These are real tasks you can run on typical OpenZFS-on-Linux systems. Adjust dataset names and interfaces. Each task includes what to look for, because “run command” isn’t operations.

Task 1: Confirm dataset and snapshot inventory

cr0x@sender:~$ zfs list -t filesystem,volume -o name,used,avail,refer,mountpoint
cr0x@sender:~$ zfs list -t snapshot -o name,creation,used -s creation | tail -n 20

Interpretation: You’re verifying what you’re about to replicate and whether snapshots exist in a consistent chain. If snapshots are missing or irregular, incremental sends will fail or force expensive full sends.

Task 2: Estimate delta size between snapshots

cr0x@sender:~$ zfs list -t snapshot -o name,used,refer -s creation tank/app
cr0x@sender:~$ zfs send -nPv -i tank/app@snaphourly-001 tank/app@snaphourly-002
total estimated size is 38.2G
send from @snaphourly-001 to tank/app@snaphourly-002 estimated size is 38.2G

Interpretation: If your “hourly” delta is 38 GB, you don’t have a replication problem—you have a workload or snapshot cadence mismatch. This output is your RPO reality check.

Task 3: Measure raw send throughput locally (remove the WAN from the equation)

cr0x@sender:~$ zfs send -Pv tank/app@snaphourly-002 | pv > /dev/null
  12.5GiB 0:00:18 [ 702MiB/s] [         <=>  ]

Interpretation: If local send is fast, sender I/O and ZFS send itself are not your bottleneck. If this is slow, fix the sender pool, ARC pressure, or snapshot structure before blaming the network.

Task 4: Measure raw receive throughput locally

cr0x@receiver:~$ zfs create -o mountpoint=none tank/replica-test
cr0x@sender:~$ zfs send -Pv tank/app@snaphourly-002 | ssh receiver 'pv | sudo zfs recv -u tank/replica-test/app'

Interpretation: This tests end-to-end but still not your WAN. If receive is slow here, your receiver pool is the culprit (or CPU for decompression/encryption).

Task 5: Confirm TCP RTT, loss signals, and route sanity

cr0x@sender:~$ ping -c 20 receiver
cr0x@sender:~$ ip route get receiver
cr0x@sender:~$ ss -ti dst receiver | head -n 40

Interpretation: Ping gives a crude RTT. ss -ti shows TCP state including retransmits, congestion window, and pacing. If you see retransmits climbing during replication, fix the network path or MTU before you tune ZFS.

Task 6: Check MTU and detect common PMTUD pain

cr0x@sender:~$ ip link show dev eth0
cr0x@sender:~$ tracepath receiver | head -n 20

Interpretation: If you’re on a VPN, your effective MTU is often smaller than you think. tracepath will hint at PMTU changes. Path MTU weirdness often presents as “random stalls” or inconsistent throughput.

Task 7: Inspect sender and receiver CPU during replication

cr0x@sender:~$ mpstat -P ALL 1 10
cr0x@sender:~$ pidstat -t 1 -p $(pgrep -n "zfs|ssh|mbuffer|pv" | tr '\n' ',')

Interpretation: If a single core is pegged (often SSH crypto or compression), you’re CPU-bound. WAN tuning won’t help if your pipeline can’t generate or consume bytes fast enough.

Task 8: Use mbuffer to smooth WAN jitter and keep the pipe full

cr0x@sender:~$ zfs send -Pv -i tank/app@snapA tank/app@snapB | mbuffer -q -m 1G -s 128k | ssh receiver 'mbuffer -q -m 1G -s 128k | sudo zfs recv -u tank/replica/app'

Interpretation: mbuffer decouples sender/receiver pacing and absorbs jitter. If this materially improves throughput, your bottleneck is likely transport variability or receiver write stalls, not ZFS send itself.

Task 9: Enable resumable sends and capture the resume token

cr0x@sender:~$ zfs send -v -s -i tank/app@snapA tank/app@snapB | ssh receiver 'sudo zfs recv -s -u tank/replica/app'
cr0x@receiver:~$ zfs get -H -o value receive_resume_token tank/replica/app
1-ETcK...truncated...GQ

Interpretation: If the link drops, you can resume instead of restarting. This is mandatory on flaky WANs. A non-empty token means a resumable receive is pending.

Task 10: Resume an interrupted replication

cr0x@receiver:~$ token=$(zfs get -H -o value receive_resume_token tank/replica/app)
cr0x@sender:~$ zfs send -v -t "$token" | ssh receiver 'sudo zfs recv -s -u tank/replica/app'

Interpretation: If resume works and throughput returns, you’ve proven the interruption was transport or remote-side transient, not snapshot chain corruption.

Task 11: Verify replication integrity by comparing snapshots

cr0x@receiver:~$ zfs list -t snapshot -o name,creation -s creation tank/replica/app | tail -n 5
cr0x@sender:~$ zfs list -t snapshot -o name,creation -s creation tank/app | tail -n 5

Interpretation: Snapshot names and order should match your intended replication set. If they don’t, you may be silently not replicating what you think you are.

Task 12: Confirm which properties are being received and whether they should be local

cr0x@receiver:~$ zfs get -o name,property,value,source compression,encryption,keylocation,atime,recordsize tank/replica/app

Interpretation: Properties can impact performance and restore behavior. For example, forcing atime=on on a replica used for DR testing can create background write noise at the worst time.

Task 13: Find out if receiver is blocked on sync or TXG pressure

cr0x@receiver:~$ zpool iostat -v 1 10
cr0x@receiver:~$ cat /proc/spl/kstat/zfs/arcstats | head -n 5
cr0x@receiver:~$ cat /proc/spl/kstat/zfs/txgs | head -n 50

Interpretation: zpool iostat shows if the pool is saturated and whether a particular vdev is lagging. TXG stats can hint at commit delays. If the pool is the choke point, you can optimize the network forever and still crawl.

Task 14: Tune SSH for bulk transfer (carefully) and observe CPU impact

cr0x@sender:~$ zfs send -Pv -i tank/app@snapA tank/app@snapB | \
ssh -T -o Compression=no -o IPQoS=throughput receiver 'pv | sudo zfs recv -u tank/replica/app'

Interpretation: Disabling SSH’s own compression avoids double-compressing already-compressed data. IPQoS=throughput can prevent some networks from treating the stream like interactive traffic. Always measure CPU and throughput before and after; “tuning” can be an expensive way to feel productive.

Task 15: Validate that the receive side isn’t accidentally mounting and indexing

cr0x@receiver:~$ zfs get -o name,property,value mounted,mountpoint canmount tank/replica/app
cr0x@receiver:~$ zfs recv -u tank/replica/app < /dev/null

Interpretation: For DR replicas, you often want canmount=off or mountpoint=none and zfs recv -u to avoid auto-mount behavior that triggers scanners, indexers, or “helpful” monitoring agents that create writes during replication.

6. Fast diagnosis playbook

When replication is slow, don’t start by swapping flags randomly. Use a short playbook that isolates the bottleneck in minutes.

Step 1: Is it network-limited or host-limited?

Check RTT and retransmits while a replication runs.
Check CPU saturation on sender and receiver.
Check receiver pool iostat for sustained high utilization and latency.

cr0x@sender:~$ ss -ti dst receiver | sed -n '1,25p'
cr0x@sender:~$ mpstat -P ALL 1 5
cr0x@receiver:~$ zpool iostat -v 1 5

Decision: If retransmits and cwnd collapse show up, treat it as a network problem first. If CPU is pegged, treat it as a CPU/crypto/compression problem. If receiver pool is at 90–100% with high wait, treat it as a storage problem.

Step 2: Is the send size unexpectedly huge?

cr0x@sender:~$ zfs send -nPv -i tank/app@last tank/app@now

Decision: If delta size is bigger than expected, fix snapshot cadence, workload churn, or your choice of incremental base snapshot (-i vs -I). WAN tuning can’t shrink the delta.

Step 3: Does buffering stabilize throughput?

cr0x@sender:~$ zfs send -Pv -i tank/app@last tank/app@now | mbuffer -m 1G -s 128k | \
ssh receiver 'mbuffer -m 1G -s 128k | sudo zfs recv -u tank/replica/app'

Decision: If throughput becomes smoother and higher, you were seeing pipeline stalls (receiver commits, network jitter, or SSH pacing). Keep mbuffer and then investigate receiver write behavior.

Step 4: Can you resume reliably?

cr0x@receiver:~$ zfs get -H -o value receive_resume_token tank/replica/app

Decision: If you can’t resume on flaky links, you’re one transient outage away from a full resend and an unpleasant weekend. Fix resumability before chasing marginal speedups.

7. Three corporate-world mini-stories

Story 1: The incident caused by a wrong assumption

The setup looked reasonable on a whiteboard: replicate production datasets to a DR site nightly over a private circuit. The team assumed “private circuit” implied “stable throughput,” and “nightly” implied “enough.” No one wrote down an RPO; it lived as a vibe.

Then a product launch happened. The workload shifted from steady database writes to lots of object-like blobs and VM image churn. Incremental snapshot deltas grew quietly, because ZFS did exactly what it was told: capture changes, then ship them later. The replication job started taking longer, creeping past morning. People noticed but shrugged—after all, it was still “running.”

During an unrelated maintenance event, the primary site had an outage that required DR promotion. The DR dataset was behind by far more than anyone expected. They had replicated “nightly,” but the night’s send hadn’t finished. Their RPO wasn’t 24 hours; it was “whenever the send happens to finish,” which is not a number you can put in a risk register with a straight face.

The fix wasn’t clever tuning. It was boring engineering: increase snapshot frequency to reduce deltas, add monitoring for “replication lag in snapshots/time,” and establish a policy that replication must complete within a defined window—or trigger paging. They also started running zfs send -nPv estimates as part of change reviews for big workload shifts.

Story 2: The optimization that backfired

A different company had a slow transcontinental link and a mandate: “make replication faster.” Someone decided to crank up compression everywhere—on the datasets, in the send pipeline, and in SSH. It looked great in a quick test: the first few minutes showed higher throughput and smaller bytes on the wire.

Then reality arrived. CPU on the sender climbed, but worse, CPU on the receiver became spiky. Replication throughput oscillated: bursts of progress followed by long flatlines. The team interpreted flatlines as “network issues” and asked for a bigger circuit. The network team, predictably, asked for proof.

The proof showed up in mpstat and pidstat: a few cores were pinned doing compression and crypto, and the receiver’s write pipeline wasn’t keeping up. Double-compression was wasting cycles because much of the data (compressed application blobs) didn’t compress further. SSH compression, in particular, added latency and CPU cost for no gain.

The fix was counterintuitive: turn off SSH compression, keep lightweight dataset compression (LZ4), and use buffering to smooth receiver stalls. Throughput improved, CPU cooled down, and—best of all—the system became predictable. The bigger lesson: if you can’t explain where the CPU cycles go, you’re not optimizing; you’re gambling with better spreadsheets.

Story 3: The boring but correct practice that saved the day

The third organization had a habit that looked painfully conservative: they insisted all WAN replication jobs be resumable, logged, and tested monthly with a simulated link interruption. They also kept a small runbook snippet taped into their incident channel: “resume token first, panic second.” It wasn’t glamorous, and nobody got promoted for it.

One quarter, a provider had intermittent packet loss for days. Not a total outage—just enough to knock long-lived TCP flows off balance and occasionally reset sessions. Replication jobs started failing mid-stream. But because the jobs used resumable sends, they resumed automatically or with a single operator command. No one was forced into “restart from scratch” transfers that would have saturated the link for weeks.

During the same period, a security team pushed stricter firewall timeouts that killed “idle” SSH sessions. The replication pipeline would pause during receiver TXG pressure, appear idle to the firewall, and get cut. Again: resumable replication turned this from a crisis into a nuisance.

When leadership asked why DR stayed healthy during “network instability,” the honest answer was not “we tuned TCP perfectly.” It was “we assumed the WAN is unreliable and engineered for that.” In enterprise operations, this is what competence looks like: it’s quiet.

8. Common mistakes (symptoms + fixes)

Mistake 1: Treating throughput as the only metric

Symptom: You celebrate a fast replication run, then discover snapshots are missing, properties aren’t replicated, or restores fail.

Fix: Define correctness: which datasets, which snapshots, recursive vs not, properties included, and whether it’s raw encrypted replication. Use zfs list -t snapshot comparisons and periodic restore tests.

Mistake 2: Full sends over WAN because “incrementals are hard”

Symptom: Replication never finishes after the first seed; the link is saturated for days; RPO drifts endlessly.

Fix: Use snapshot naming conventions and incremental sends. If you must seed, do it once (ideally by shipping disks or using a temporary high-bandwidth path), then run incrementals.

Mistake 3: Double compression (dataset + pipeline + SSH)

Symptom: CPU spikes, throughput oscillates, and “faster” settings make it slower.

Fix: Pick one place to compress. Often: LZ4 on datasets, SSH compression off. Measure with CPU tools.

Mistake 4: No buffering across jittery WAN

Symptom: Throughput looks like a sawtooth: bursts then stalls; SSH sessions sometimes drop during quiet phases.

Fix: Add buffering (mbuffer) on both ends with sensible memory allocation. This smooths short receiver stalls and network jitter.

Mistake 5: Ignoring receiver pool health

Symptom: Sender appears fast locally, but over WAN end-to-end speed is poor even when the link is underutilized.

Fix: Check zpool iostat -v, latency, and contention. Replication is a write-heavy workload on the receiver; treat it like one.

Mistake 6: Not using resumable send on unreliable links

Symptom: A 6-hour transfer dies at hour 5:45 and restarts from the beginning. Morale drops faster than throughput.

Fix: Use zfs send -s and zfs recv -s. Capture and monitor resume tokens.

Mistake 7: Overlapping replication jobs fighting each other

Symptom: Multiple datasets replicate “in parallel,” but total throughput is worse and latency-sensitive apps complain.

Fix: Serialize big streams or use a controlled concurrency. Your WAN link and receiver pool don’t get faster because you added more pipes; you just added contention.

Mistake 8: Assuming “the WAN is fine” because speed tests look good

Symptom: Browser-based speed tests pass, but long-lived replication flows crawl.

Fix: Observe TCP behavior (ss -ti), retransmits, and MTU/PMTUD. Long flows reveal path issues that short tests hide.

Second joke (required): If you think WAN packet loss is “basically zero,” I have a bridge to sell you—over VPN, with an MTU of 9000.

9. Checklists / step-by-step plan

9.1 Step-by-step plan for a new WAN replication setup

Define intent: RPO/RTO targets per dataset class (critical, important, best-effort).
Normalize snapshot naming: predictable names, consistent cadence, and retention policy.
Seed the baseline: do a one-time full send during a maintenance window or via an alternate path; confirm receive-side layout.
Make it resumable by default: zfs send -s and zfs recv -s.
Add buffering: deploy mbuffer with conservative memory sizing (e.g., 512MB–2GB) depending on host RAM.
Pick compression strategy: usually dataset LZ4, no SSH compression. If using native encryption, prefer raw sends when appropriate.
Throttle if necessary: avoid crushing business traffic; rate limiting is sometimes a feature, not a concession.
Verify correctness: ensure snapshot chains, properties, and expected datasets arrive. Keep receives unmounted unless needed.
Instrument replication lag: measure “latest replicated snapshot age” and alert when it exceeds thresholds.
Test restores: mount a replica clone or promote a test environment on a schedule. Replication you haven’t restored from is a theory.

9.2 Operational checklist for “replication feels slow today”

Check receiver pool contention: zpool iostat -v 1 5.
Check TCP retransmits/cwnd: ss -ti dst receiver.
Check CPU pinning: mpstat and pidstat.
Estimate delta size: zfs send -nPv -i.
Try buffering if not already used: add mbuffer.
Check resumable token status; resume instead of restarting.

10. FAQ

Q1: Should I use raw send (`zfs send -w`) over WAN?

If you use ZFS native encryption and you want to replicate encrypted datasets without decrypting, raw send is usually the right move. It can also reduce CPU overhead because ZFS can transmit stored encrypted/compressed blocks. Operationally, it preserves encryption boundaries, which is exactly what you want for offsite copies.

Q2: Is `mbuffer` mandatory?

Not mandatory, but it’s one of the highest ROI tools for WAN replication because it smooths bursts and stalls. If your throughput is spiky or your SSH sessions die during “quiet” phases, buffering is a practical fix.

Q3: Why does replication stall at the same point every time?

Often it’s receiver-side pool pressure or a network middlebox timeout. Check receiver zpool iostat for a vdev that hits 100% utilization or shows long waits, and check if your firewall or NAT has aggressive idle timeouts that kill long-lived sessions.

Q4: Should I run multiple sends in parallel to use the link better?

Sometimes, but be cautious. Parallel streams can help if a single stream can’t fill the pipe due to per-flow limitations, but they also increase contention on disks and CPU and can make tail latency worse for applications. Try one well-tuned stream first; if it can’t saturate and the hosts are idle, then test controlled parallelism.

Q5: What’s the best snapshot frequency for WAN replication?

One that keeps your incremental deltas comfortably within your replication window with margin for retries. If your link can reliably move X GB/hour and your workload changes Y GB/hour, your snapshot cadence needs to keep each delta below what you can ship plus overhead.

Q6: Should I replicate recursively with `-R`?

-R is correct when you need a full hierarchy (children, snapshots, properties) replicated consistently. It’s also how people accidentally replicate more than they intended. Use it when you mean it, and verify the dataset tree on the receiver.

Q7: How do I know if I’m CPU-bound by SSH encryption?

Run a replication and watch mpstat/pidstat. If one or more cores are pegged in the ssh process and your link is underutilized, SSH crypto is a prime suspect. Consider raw sends with native encryption (if applicable) or adjust SSH options and host CPU capacity.

Q8: Is it safe to set `sync=disabled` on the receiver to speed up `zfs recv`?

It’s fast, and it’s a risk. For a DR replica where you can tolerate losing the last few seconds of in-flight received data during a crash, teams sometimes choose it knowingly. But don’t do it casually: you’re changing durability semantics on the receiver. If you do, document it, scope it, and test failure behavior.

Q9: Why does `zfs send -nPv` estimate not match what I actually send?

It’s an estimate and can vary based on block sharing, metadata, and stream characteristics. It’s still useful for trend detection and “is this delta insane?” decisions, not for billing-grade accounting.

Q10: What’s the single best way to avoid catastrophic re-sends?

Resumable replication plus monitoring for resume tokens and lag. WAN links will fail; your plan should assume that and keep progress.

Conclusion

Fast ZFS replication over a slow WAN isn’t about one magic flag. It’s about making the pipeline boring: predictable snapshot cadence, incremental streams, buffering for jitter, correct encryption/compression choices, and a receiver that can actually write what you’re sending. The best setups don’t just go fast on good days—they keep moving on bad days, resume after interruptions, and tell you early when reality is drifting away from your RPO.

If you take only one operational lesson: measure each stage, isolate the bottleneck quickly, and optimize the constraint you actually have—not the one that feels most familiar.