There are two types of data migrations: the ones you plan, and the ones you do while everyone is watching. ZFS zfs send/zfs receive is one of the rare tools that can handle both—moving terabytes with integrity, reproducibility, and a clear model of what changed. But it’s also one of the easiest ways to produce a “successful” transfer that’s subtly wrong, painfully slow, or impossible to resume.
This is a field guide from the sharp end of production: how to replicate fast, how to know what’s actually happening, what breaks under pressure, and which boring practices prevent you from becoming a cautionary tale in next quarter’s postmortem.
Why zfs send is different (and why that matters)
Most “copy tools” move files. ZFS moves truth: a consistent point-in-time view of a dataset as represented by a snapshot. A zfs send stream is not “whatever was in the directory while rsync ran.” It’s “exactly the blocks referenced by snapshot X,” optionally plus “exactly the changed blocks from X to Y.” That difference is why ZFS replication can be both faster and more correct than file-level migration.
The trade is that you must respect ZFS’s model. If you don’t, you’ll end up trying to replicate a moving target, replicate from the wrong base, or replicate encrypted data in a way that looks fine until restore day. In ZFS land, a migration is a chain of snapshots and assumptions. Your job is to keep the chain intact and the assumptions explicit.
One joke, to set the tone: ZFS replication is like moving houses by shipping the foundation—efficient and consistent, but you should probably measure the driveway first.
Interesting facts and historical context
- ZFS was designed for end-to-end integrity: checksums are validated on read, meaning replication plus scrub gives you a corruption detection story most file tools can’t match.
- Snapshots are cheap on purpose: ZFS snapshots are copy-on-write references, not full copies, making incremental replication practical at scale.
- Send streams are deterministic: given the same dataset/snapshot, the stream is stable enough to support resumable replication and reliable receives.
- Incremental send is a first-class feature: it’s not “diffing files,” it’s sending only changed blocks between snapshots.
- Encrypted datasets changed the replication game: “raw” sends allow you to replicate encrypted data without exposing plaintext on the wire or at the receiver.
- Receive-side property handling is deliberate: ZFS gives you knobs to override mountpoints, prevent auto-mount surprises, and manage properties separately from data.
- Resume tokens were added because networks lie: long streams across flaky links needed an official, safe resume mechanism rather than “start over and hope.”
- ZFS replication is used as poor man’s DR (and often good man’s DR): because it can be scheduled, incremental, and verifiable, many organizations build DR on it before buying anything fancy.
The mental model: snapshots, streams, and trust
1) You are not replicating “a dataset,” you are replicating “a snapshot graph”
ZFS datasets have state that changes constantly. Snapshots freeze that state. A send stream references snapshots. Incremental sends reference pairs of snapshots: a base and a target.
The practical implication: if the base snapshot is missing on the receiver, an incremental receive can’t apply. If someone deletes “old, useless snapshots” on the destination, you just broke tomorrow’s replication. This is not a theoretical edge case; it’s a recurring incident pattern.
2) “Full” vs “incremental” isn’t about size, it’s about lineage
A full send (zfs send pool/ds@snap) provides the entire dataset state at that snapshot. An incremental send (-i or -I) relies on the receiver already having a related snapshot.
-isends changes from exactly one snapshot to another (“from A to B”).-Isends an incremental chain including intermediate snapshots (“from A up through B, including snapshots in between”).
3) Your real enemy is not speed—it’s silent divergence
Speed is easy to obsess over because it’s visible. Divergence is what ruins your weekend: a receive that “worked” but mounted into the wrong path; a dataset received with properties that changed application semantics; an encrypted dataset received in a way that makes keys and clones painful later.
4) Streams include more than file contents
ZFS streams can include properties, snapshots, clones (with the right flags), and—in raw mode—encryption metadata. That’s power. It’s also a foot-gun if you don’t decide intentionally what to preserve.
Practical tasks (commands + interpretation)
The commands below assume a source host (src) and destination host (dst) with ZFS pools already created. Replace pool and dataset names accordingly. Outputs are representative; your system will vary.
Task 1: Confirm pool health before you replicate
cr0x@src:~$ zpool status
pool: tank
state: ONLINE
status: Some supported features are not enabled on the pool.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support the features.
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
Interpretation: If you see DEGRADED, resilvering, or checksum errors, fix that first. Replication faithfully moves corruption too; ZFS checksums detect it, but they don’t magically heal the source.
Task 2: Estimate dataset size and snapshot footprint
cr0x@src:~$ zfs list -o name,used,avail,refer,mountpoint -r tank/prod
NAME USED AVAIL REFER MOUNTPOINT
tank/prod 8.2T 12.1T 128K /tank/prod
tank/prod/db 6.9T 12.1T 6.9T /tank/prod/db
tank/prod/log 1.3T 12.1T 1.3T /tank/prod/log
Interpretation: USED includes snapshots and descendants. REFER is the data referenced by the dataset itself. If USED is much larger than REFER, you have significant snapshot retention; incremental replication will be your friend.
Task 3: Create a replication snapshot with a naming convention
cr0x@src:~$ zfs snapshot -r tank/prod@replica-2025-12-25-0001
Interpretation: Use a predictable snapshot prefix (replica-) so you can automate selection and retention without touching human-created snapshots.
Task 4: Dry-run send size estimation (so you can plan)
cr0x@src:~$ zfs send -nP -R tank/prod@replica-2025-12-25-0001
size 9.11T
Interpretation: -nP estimates size without sending. -R replicates dataset plus descendants and properties. The estimate helps you decide whether you need incremental, compression, or a maintenance window.
Task 5: Do the first full replication (safe defaults)
cr0x@src:~$ zfs send -R tank/prod@replica-2025-12-25-0001 | ssh dst sudo zfs receive -u -o mountpoint=/tank/restore tank/recv
Interpretation: -u prevents immediate mounting on the destination, avoiding surprise mounts on busy systems. Overriding mountpoint ensures you don’t accidentally mount into production paths. Receiving into tank/recv creates a clean namespace; you can later promote or rename.
Task 6: Verify received snapshots exist and match names
cr0x@dst:~$ zfs list -t snapshot -r tank/recv | head
NAME USED AVAIL REFER MOUNTPOINT
tank/recv/prod@replica-2025-12-25-0001 0B - 6.9T -
tank/recv/prod/db@replica-2025-12-25-0001 0B - 6.9T -
tank/recv/prod/log@replica-2025-12-25-0001 0B - 1.3T -
Interpretation: Snapshot presence is your proof of lineage. If you don’t see the snapshot you expect, do not proceed to incrementals until you understand why.
Task 7: Run an incremental send (single-step)
cr0x@src:~$ zfs snapshot -r tank/prod@replica-2025-12-25-0600
cr0x@src:~$ zfs send -R -i tank/prod@replica-2025-12-25-0001 tank/prod@replica-2025-12-25-0600 | ssh dst sudo zfs receive -u tank/recv
Interpretation: This sends only changed blocks between those two snapshots, but still uses -R to cover descendants. It will fail if the destination lacks the base snapshot.
Task 8: Incremental with intermediates (-I) for missed runs
cr0x@src:~$ zfs send -R -I tank/prod@replica-2025-12-25-0001 tank/prod@replica-2025-12-25-1800 | ssh dst sudo zfs receive -u tank/recv
Interpretation: If your replication job missed several snapshots, -I can send a chain including intermediate snapshots—often making retention and rollback easier on the destination.
Task 9: Send with compression on the wire (pragmatic)
cr0x@src:~$ zfs send -R tank/prod@replica-2025-12-25-0001 | gzip -1 | ssh dst "gunzip | sudo zfs receive -u tank/recv"
Interpretation: This is simple and frequently effective on log-heavy or text-heavy datasets. On already-compressed data (backups, media), it can waste CPU for little gain. Measure, don’t guess.
Task 10: Replicate an encrypted dataset safely (raw send)
cr0x@src:~$ zfs get -o name,property,value -s local encryptionroot,keystatus -r tank/secure
NAME PROPERTY VALUE
tank/secure encryptionroot tank/secure
tank/secure keystatus available
cr0x@src:~$ zfs snapshot -r tank/secure@replica-0001
cr0x@src:~$ zfs send -w -R tank/secure@replica-0001 | ssh dst sudo zfs receive -u tank/recv-secure
Interpretation: -w sends the raw encrypted stream. The destination receives encrypted blocks; you’re not exposing plaintext during transfer. This is often what security teams actually want, provided you also have a plan for key management on the receiver.
Task 11: Handle interruption with resume tokens (don’t start over)
cr0x@dst:~$ zfs get -H -o value receive_resume_token tank/recv/prod
1-ED8f3a9c0c-...
cr0x@src:~$ zfs send -t 1-ED8f3a9c0c-... | ssh dst sudo zfs receive -u tank/recv
Interpretation: If a receive was interrupted, ZFS can store a resume token on the destination dataset. Sending with -t continues from where it left off, assuming the token is valid and the stream matches. This feature is the difference between “we lost a link for 30 seconds” and “we re-send 40 TB.”
Task 12: Avoid accidental mountpoint chaos on receive
cr0x@dst:~$ sudo zfs receive -u -o mountpoint=/mnt/quarantine -o canmount=off tank/recv < /tmp/stream.zfs
Interpretation: If you’re receiving from a file or staging area, forcing canmount=off and a quarantine mountpoint prevents unexpected mounts, especially when -R brings multiple descendants with inherited properties.
Task 13: Confirm the destination is actually reading healthy data
cr0x@dst:~$ zpool scrub tank
cr0x@dst:~$ zpool status tank
pool: tank
state: ONLINE
status: Scrub in progress since Thu Dec 25 09:10:52 2025
1.23T scanned at 6.10G/s, 180G issued at 900M/s, 9.11T total
0B repaired, 1.98% done, 0:28:10 to go
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
errors: No known data errors
Interpretation: A scrub after a major receive is not paranoia; it’s validation. If your DR copy is corrupt, you want to learn that now, not during an outage.
Task 14: Measure send/receive performance without guesswork
cr0x@src:~$ zfs send -nP tank/prod@replica-2025-12-25-0001
size 9.11T
cr0x@src:~$ time zfs send -R tank/prod@replica-2025-12-25-0001 | ssh dst sudo zfs receive -u tank/recv
real 154m12.331s
user 2m10.912s
sys 12m48.776s
Interpretation: The time output plus estimated size lets you compute effective throughput. Don’t optimize blindly: if CPU time is low but wall time is huge, you’re waiting on network or disks.
Performance: how to make it actually fast
“Fastest way to move terabytes” is true, but with conditions. ZFS can push data at line rate when the stars align: sequential-ish reads, minimal CPU overhead, a receiver pool that can ingest writes, and a network path that doesn’t melt when you sneeze. In real life, one of those is always the bottleneck.
Start with the boring constraint: replication is a pipeline
A send/receive is a pipeline: source pool read → send stream creation → transport (often SSH) → receive stream parsing → destination pool write → optional mount and post-processing. The pipeline runs at the speed of the slowest stage. Your job is to identify that stage quickly and either remove it or accept it.
SSH: convenient, secure, and sometimes your bottleneck
SSH is the default transport because it’s everywhere and secure by default. But encryption can become the ceiling on fast links. If you’re moving tens of terabytes over 10/25/40/100GbE, test CPU headroom for SSH encryption. If CPUs are pegged, you have options:
- Choose ciphers that are hardware-accelerated on your CPUs (often AES-GCM on modern x86).
- Parallelize by splitting datasets (multiple independent sends) if your storage can handle it.
- Use raw sends for encrypted datasets (still encrypted end-to-end at the ZFS layer), but you still have SSH encryption unless you change transport.
Second joke, and then back to work: The only thing faster than a saturated 100Gb link is an engineer saying “it’s probably DNS” when it’s clearly CPU.
Compression: a lever, not a religion
Compression in transit can be magic on write-heavy databases with lots of zeros or repetitive patterns, and a total waste on JPEGs and Parquet. Use zfs send -nP for sizing, then do a short timed sample (smaller dataset or recent incremental) with and without compression. Decide based on measured throughput and CPU burn, not vibes.
Recordsize, volblocksize, and why databases behave differently
ZFS dataset recordsize impacts how data is laid out and can influence send behavior indirectly. Databases that use many small random writes often benefit from smaller blocks (or separate datasets), but replication is still block-based. For zvols (block devices), volblocksize is fixed at creation; if you replicate large zvols with a tiny volblocksize, you’re signing up for more metadata and potentially worse throughput.
Destination pool ingest: the silent killer
You can have a screaming-fast source and network, and still crawl because the destination pool can’t commit writes. Common reasons:
- Small or misconfigured SLOG (for sync writes on certain workloads; receive itself is not purely sync but can still be affected by pool behavior).
- Spinning disks with insufficient vdev width for sustained ingest.
- Ashift mismatch, failing disks, or a pool already busy with other workloads.
- RAM pressure causing poor ARC behavior and extra IO.
Use staging receives when you need to protect production
A common pattern: receive into pool/recv with canmount=off and a safe mountpoint, then do a controlled cutover: rename datasets, set mountpoints, load keys, mount, and start services. This doesn’t make the transfer faster, but it makes it survivable.
Three corporate-world mini-stories
Mini-story #1: The incident caused by a wrong assumption
The migration plan looked clean: full replication on Friday night, incrementals every hour, cutover Sunday. The team assumed “incremental” meant “destination only needs the dataset name.” It did not. It needed the exact base snapshot.
On Saturday morning, someone tidied up the destination because it “had too many snapshots.” They deleted a week-old snapshot that didn’t match the “replica-” naming scheme—because it didn’t. It was an auto snapshot from an earlier test run, and it happened to be the incremental base the job had been using since Friday.
At 10:00, the hourly incremental started failing with a message that looked like a transient receive issue. The runbook said “retry,” so it retried. Then it retried again. Meanwhile, new writes kept happening on the source, widening the gap.
By the time someone noticed the error pattern was consistent, the team was staring at an unpleasant choice: re-seed a multi-terabyte full send during business hours or accept a longer outage window. They re-seeded overnight and cut over late. Nobody got fired, but the replication pipeline got a new rule: the base snapshot is sacred, and retention policies must be automated and prefix-aware.
Lesson: Incremental replication is about snapshot lineage, not about “the latest state.” If the destination loses the base, you don’t have replication—you have a time bomb.
Mini-story #2: The optimization that backfired
A different team wanted speed. They had a 40Gb link and a backlog of datasets to replicate to a DR site. Someone suggested compressing everything “because it’s always faster.” They wrapped zfs send in aggressive compression and declared victory after one test on a text-heavy dataset.
Then they pointed it at media archives and VM images. CPU went to the ceiling on both ends, SSH threads fought for cycles, and throughput dropped below what the plain, uncompressed stream could do. Worse: the receive process started falling behind enough to trigger operational noise—timeouts, monitoring alerts, and a cascade of “is the DR site down?” messages.
The fix wasn’t heroic. They ran a short benchmark per dataset class and set policy: compress on the wire for logs and database dumps; don’t compress for already-compressed blobs; prefer raw sends for encrypted datasets; and limit concurrency based on destination pool write capacity, not network speed. The pipeline became stable and predictably fast—less exciting, more effective.
Lesson: Optimizing one stage of the pipeline can starve another. Compression is a throughput trade: network saved, CPU spent. Spend CPU only where it buys you real time.
Mini-story #3: The boring but correct practice that saved the day
The best replication story I can tell is the one where nothing happened. A storage team had a routine: before any major receive, they used -u to prevent mounts, forced a quarantine mountpoint, and ran a scrub within 24 hours. They also tagged replication snapshots with a strict prefix and refused “manual snapshot cleanup” on the destination.
One quarter, the DR site suffered a messy power event. Most systems came back, but one disk shelf had a controller problem that caused intermittent IO errors. The pool stayed online, but it was not happy. The replication job still ran and claimed success for some datasets—until the scrub caught checksum errors on recently received blocks.
Because they scrubbed routinely, they detected the issue before a real disaster. They paused replication, fixed the hardware, re-sent the affected incrementals using the preserved snapshot lineage, and validated again. When an unrelated production incident happened weeks later, the DR copy was actually usable—because the team treated “validation” as part of replication, not a luxury.
Lesson: The boring practices—no auto-mount surprises, consistent snapshot naming, periodic scrubs—aren’t ceremony. They’re the difference between DR as a slide deck and DR as a working system.
Fast diagnosis playbook
When a replication is slow or failing, you don’t have time for interpretive dance. Here’s the quickest route to the bottleneck, in the order that usually pays off.
First: confirm it’s actually making progress (and not retrying)
cr0x@dst:~$ ps aux | egrep 'zfs receive|ssh' | grep -v egrep
root 21844 2.1 0.1 24548 9120 ? Ss 09:20 0:01 zfs receive -u tank/recv
root 21843 0.9 0.0 16720 6340 ? Ss 09:20 0:00 ssh -oBatchMode=yes src zfs send -R tank/prod@replica-2025-12-25-0600
Interpretation: If you see repeated short-lived processes, you’re flapping due to an error. If the processes are long-lived, you’re probably slow, not failing.
Second: check whether the destination is the throttle (pool busy or unhealthy)
cr0x@dst:~$ zpool iostat -v tank 2 3
capacity operations bandwidth
pool alloc free read write read write
------------------------------------------ ----- ----- ----- ----- ----- -----
tank 9.40T 8.90T 10 980 12M 820M
mirror-0 9.40T 8.90T 10 980 12M 820M
sda - - 5 490 6M 410M
sdb - - 5 490 6M 410M
------------------------------------------ ----- ----- ----- ----- ----- -----
Interpretation: If writes are pinned near the physical ceiling of the pool, that’s your bottleneck. If writes are low but the send is slow, look elsewhere (SSH CPU, source reads, network).
Third: check SSH CPU saturation
cr0x@src:~$ top -b -n 1 | head -n 15
top - 09:24:11 up 31 days, 6:02, 2 users, load average: 18.22, 16.91, 14.77
Tasks: 312 total, 2 running, 310 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.2 us, 1.1 sy, 0.0 ni, 7.4 id, 88.0 wa, 0.0 hi, 0.3 si, 0.0 st
MiB Mem : 257982.3 total, 11230.5 free, 78112.9 used, 170638.9 buff/cache
Interpretation: High wa means IO wait (source reads likely bottleneck). High user/system with ssh processes on top suggests crypto overhead.
Fourth: check snapshot lineage errors (the “it failed instantly” class)
cr0x@dst:~$ sudo zfs receive -u tank/recv
cannot receive incremental stream: destination tank/recv/prod has been modified
Interpretation: This usually means the destination dataset diverged (writes happened, snapshots deleted, or you received into the wrong place). Fix lineage, don’t brute-force retries.
Fifth: check if encryption mode mismatch is blocking you
cr0x@dst:~$ zfs get -o name,property,value -r tank/recv-secure | egrep 'encryptionroot|keystatus'
tank/recv-secure/secure encryptionroot tank/recv-secure/secure
tank/recv-secure/secure keystatus unavailable
Interpretation: keystatus unavailable is normal if keys aren’t loaded. It becomes a problem if your workflow expects to mount immediately. Decide whether you want raw replication (keys managed separately) or decrypted replication (keys loaded at source and data sent as plaintext within ZFS stream).
Common mistakes (symptoms + fixes)
Mistake 1: Receiving into the wrong dataset path
Symptom: Receive completes, but datasets appear under an unexpected hierarchy (or overwrite a test dataset).
Fix: Always stage receives into a clearly named namespace (e.g., tank/recv) and use -u. Verify with zfs list -r before mounting or renaming.
Mistake 2: Deleting snapshots on the destination “to save space”
Symptom: Incremental receives fail with “missing snapshot” or “destination has been modified.”
Fix: Retention must be policy-driven and replication-aware. Only delete replication snapshots according to a schedule that preserves the base required by the next incremental.
Mistake 3: Not using -u and getting surprise mounts
Symptom: After receive, new filesystems mount automatically, sometimes on paths used by production services.
Fix: Use zfs receive -u and override mountpoint and/or set canmount=off during receive. Mount intentionally later.
Mistake 4: Assuming compression always helps
Symptom: CPU pegged, throughput worse than expected, receive lagging.
Fix: Benchmark. Use compression only where data is compressible. Consider lighter compression levels if CPU is the limiter.
Mistake 5: Ignoring pool feature flags and version compatibility
Symptom: Receive fails on older systems, or replicated pool can’t be imported where you need it.
Fix: Align ZFS versions/features across sites before you build DR assumptions. If you must interop with older ZFS, avoid enabling features that break compatibility.
Mistake 6: Forgetting that -R brings properties and descendants
Symptom: Mountpoints, quotas, reservations, or other properties appear unexpectedly on the destination.
Fix: Use receive-side -o overrides where appropriate, and audit critical properties after receive with zfs get. Keep a “property contract” for replicated datasets.
Mistake 7: Not planning for resume
Symptom: A transient network drop forces a full re-send and blows the window.
Fix: Use ZFS resume tokens where supported. Design jobs to detect tokens and resume automatically rather than restarting from scratch.
Checklists / step-by-step plan
Checklist A: One-time seeding (full replication) that won’t hurt you later
- Verify pool health on both ends (
zpool status), fix errors first. - Pick a naming convention for replication snapshots (
replica-YYYY-MM-DD-HHMM). - Create a recursive snapshot on the source for the initial seed.
- Estimate size with
zfs send -nP -Rto understand time and bandwidth needs. - Receive into a staging namespace with
-uand safe properties (mountpoint/canmount). - Verify snapshots exist on destination and dataset tree matches expectations.
- Scrub destination pool within 24 hours of the seed for confidence.
Checklist B: Ongoing incremental replication (the daily driver)
- Create new snapshots on a schedule (hourly/daily), always with the replication prefix.
- Send incrementals using a known base snapshot; prefer
-Iif you want to include intermediate snapshots. - Use
-uto keep receives non-disruptive; mount only during controlled cutovers or tests. - Monitor for receive resume tokens and auto-resume if a job is interrupted.
- Apply retention policies to replication snapshots on both ends, preserving lineage for the next incremental.
Checklist C: Cutover plan (because “we’ll just switch it” is not a plan)
- Quiesce application writes (maintenance mode, database flush, or service stop—whatever is correct for your workload).
- Take final snapshots and run a last incremental send.
- On destination: set final mountpoints, load encryption keys if needed, and validate datasets exist.
- Mount datasets and start services; validate with application-level checks, not just filesystem mounts.
- Keep the old source read-only and retain snapshots until you’re confident in the new state.
FAQ
1) Is zfs send | zfs receive always faster than rsync?
No, but it’s often faster at scale and usually more consistent. It shines when you can use incrementals and when filesystem semantics (permissions, hardlinks, xattrs) matter. If you only need a small subset of files or you can’t use snapshots, rsync might be simpler.
2) Should I always use -R?
Not always. -R is great for replicating a dataset tree with properties and snapshots. If you only need one dataset without descendants or you want tighter control over which properties travel, a non-recursive send can be safer.
3) What’s the difference between -i and -I again?
-i sends from one snapshot to another (a single delta). -I sends an incremental chain and includes intermediate snapshots between base and target. Use -I when you want the destination to retain the same snapshot history.
4) How do I know which snapshot to use as the incremental base?
Use the most recent replication snapshot that exists on both source and destination. Operationally, that means: the last successfully received snapshot with your replication prefix. Don’t guess—list snapshots on both sides and match names.
5) Can I resume an interrupted transfer safely?
Yes, if your ZFS supports resume tokens and you haven’t destroyed the partial receive state. Check receive_resume_token on the destination dataset; if present, send with zfs send -t <token> and receive again.
6) Does raw send (-w) mean the data stays encrypted in transit?
Raw send means ZFS sends encrypted blocks and encryption metadata, so the ZFS stream does not contain plaintext. Many people still use SSH, which encrypts transport too—fine, but not required for confidentiality when the stream itself is raw-encrypted.
7) Why did my incremental receive fail with “destination has been modified”?
Because it has. That can mean actual writes to the destination dataset, a rollback, a property change that implies divergence in some contexts, or missing snapshots in the expected lineage. The fix is to re-establish a common snapshot base—sometimes by rolling back or by doing a new full send.
8) How do I prevent replicated datasets from mounting automatically?
Use zfs receive -u. For extra safety, set canmount=off and a quarantine mountpoint during receive, then change properties and mount intentionally later.
9) Is it safe to replicate production databases with ZFS snapshots?
It’s safe for filesystem consistency, not automatically application consistency. For databases, coordinate snapshots with the database’s own consistency mechanism (flush/checkpoint/backup mode) if you need crash-consistent vs application-consistent guarantees.
10) How do I validate that the DR copy is usable?
At minimum: ensure snapshots are present, run periodic scrubs, and perform test restores or mounts in an isolated environment. A DR plan without restore drills is just a nice story.
Conclusion
zfs send/zfs receive is one of the few tools that can move terabytes quickly without turning correctness into a hope-and-prayer exercise. But ZFS is strict in the way good systems are strict: snapshot lineage matters, properties matter, and the receiver is not a trash can you pour streams into.
If you want it fast, treat replication as a pipeline and measure where you’re throttled. If you want it reliable, treat snapshot naming, retention, resume behavior, and post-receive validation as part of the system—not optional extras. Do it right and you get something rare: speed you can trust.