Compression in ZFS is one of those rare knobs that can make storage faster, cheaper, and quieter—all at once. It can also turn a perfectly healthy system into a CPU-bound space heater if you treat “zstd-19” like a personality trait. The trick is to pick levels like an SRE: based on measurements, workload shape, and failure modes, not vibes.
This is a field guide for running zstd compression in production ZFS: how the levels behave, what “good” looks like, what to check when it hurts, and how to tune without accidentally building a benchmarking hobby farm. We’ll stay practical: commands you can run today, how to interpret them, and what decisions they should change.
Why compression is a performance feature
ZFS compression is inline. That means on writes, ZFS compresses blocks before they hit disk; on reads, it decompresses blocks after they come off disk and before they land in your application’s buffers. People file compression under “capacity savings,” but in practice it’s frequently a performance tool.
Fewer bytes moved is often faster than fewer CPU cycles spent
Storage systems are usually bounded by one of three things: latency, bandwidth, or CPU. Compression trades CPU for fewer bytes on the wire and fewer bytes on disk. If your workload is I/O-bound (common), that trade is a win.
Here’s the mental model that keeps you honest:
- If your storage is the bottleneck, compression can increase effective throughput and reduce read amplification.
- If your CPU is the bottleneck, aggressive compression can tank performance and cause cascading timeouts.
- If your workload is latency-sensitive, you care about p99 and tail spikes, not average throughput.
Compression is also a quiet defragmenter of fate: fewer blocks written means fewer allocations, fewer metaslab updates, and sometimes less fragmentation pressure over time. That isn’t magic and it isn’t guaranteed, but it’s real enough to show up in long-lived pools.
One joke before we get serious: compression is like dieting—everyone loves the “before/after,” nobody loves counting the CPU calories.
How zstd behaves in ZFS (and what levels really mean)
Zstandard (zstd) is designed to be fast at low levels and capable of high compression at higher levels. In ZFS, you typically set compression at the dataset level using compression=zstd or compression=zstd-N. The level affects how hard ZFS tries to find compressible patterns.
What ZFS compresses: recordsize-sized blocks (usually)
ZFS compresses individual blocks, typically up to the dataset’s recordsize (for filesystems) or volblocksize (for zvols). This is a major reason “compression level” is not a universal answer: compressing 128K blocks for a file server is a different game than compressing 16K blocks for a database, and wildly different than 8K blocks for a zvol backing a VM disk.
Levels aren’t linear—and “better ratio” can be worse
zstd levels increase CPU cost faster than they increase compression ratio. The curve is steep: low levels are often “almost free,” while high levels can be dramatically more expensive for modest additional savings. On modern CPUs, decompression is typically much cheaper than compression, which matters for read-heavy workloads.
Also: a better compression ratio isn’t always better for ZFS. When blocks get smaller, you reduce I/O, but you might increase CPU time, reduce prefetch efficiency, or change how ARC and L2ARC behave. The right level is the one that makes your workload faster or cheaper without moving the bottleneck somewhere worse.
zstd in ZFS is not “one size fits all”
Real-world workloads have personalities:
- VM images: lots of random reads/writes, mixed compressibility, often benefit from moderate compression but can expose CPU overhead quickly.
- Databases: structured data, log files, and indexes vary; sometimes you get great gains, sometimes almost nothing. Latency tails matter.
- Backups: typically sequential, highly compressible (especially text, JSON, CSV), and tolerant of CPU use—until restore time becomes a business event.
- Media: already compressed; compression mostly wastes CPU, though metadata and sidecar files may still benefit.
Facts & historical context you can use in meetings
These are the kind of concrete points that help when someone asks, “Why are we changing compression again?”
- ZFS compression has been around for a long time—it’s not a trendy add-on. Early ZFS often used LZJB, then lz4 became the practical default for many deployments because it was fast enough to be “always on.”
- zstd was created by Yann Collet (the same engineer behind LZ4), designed to offer a spectrum: lz4-like speed at low levels and strong ratios at higher levels.
- Inline compression changes the physical layout: data is stored compressed on disk. You’re not “saving space later”; you’re changing how many sectors hit media right now.
- Modern CPUs can decompress absurdly fast relative to spinning disks and even many SSD arrays; decompression often isn’t the limiter—compression is.
- ZFS reports compression per block and per dataset, but
compressratiois not a benchmark; it includes quirks like metadata, padding, and how blocks were rewritten over time. - Compression interacts with recordsize: larger blocks can compress better, but are more expensive to rewrite and can increase write amplification for small random writes.
- Dedup is not compression and historically has caused more outages than it has saved budgets when enabled casually; compression is the safer lever.
- ZFS “copies” and checksums everything (copy-on-write + end-to-end checksumming). Compression must fit into that pipeline, which is why CPU overhead can show up as latency spikes under load.
Picking zstd levels by workload (without lighting up CPUs)
The default stance: start conservative, then earn your way up
If you want one opinionated baseline: use zstd at a low level for most datasets, then selectively increase it where you have evidence it helps. In many production environments, that means compression=zstd (which maps to a default level) or an explicit low level like zstd-1 to zstd-3 for general-purpose filesystems.
Why? Because you can usually get a meaningful reduction in bytes written and read with minimal CPU impact. That reduction improves cache efficiency too: ARC and page cache effectively become bigger when the same RAM holds more logical data.
Workload-driven recommendations (practical, not doctrinal)
General-purpose filesystems (home dirs, build artifacts, logs)
Start at zstd-1 to zstd-3. These levels tend to capture “easy wins” (text, JSON, binaries with zeroed regions) without turning compression into a CPU job.
Logs are particularly fun: they compress extremely well, and log bursts can be intense. Low-level zstd usually keeps up; higher levels may become visible in p99 write latency during incident storms—exactly when you least want extra latency.
VM storage (zvols or filesystems storing qcow2/raw)
Use zstd-1 or zstd-2 first. VM I/O is often random and latency sensitive. Compression can help because VM images contain a lot of zeros and repetitive filesystem structures, but the write pattern can expose compression CPU overhead quickly.
If your hypervisor host is already CPU-loaded (overcommit, encryption, network overlays), keep compression cheap. If the storage array is the bottleneck and hosts have CPU headroom, you can test zstd-3 or zstd-4 carefully.
Databases
Be conservative and measure p99. DB workloads tend to be a mix: WAL/redo logs (often compressible), data files (maybe), and indexes (depends). The cost of extra CPU in the storage path can show up as tail latency, which databases translate into “why is the app slow?” tickets.
Start with zstd-1 or zstd-2 for data, maybe higher for archival tablespaces if your DB layout supports separation.
Backup repositories
Here you can justify higher levels—sometimes. If the backup workload is sequential and the repository is write-heavy during a window, zstd-3 to zstd-6 might make sense if you have CPU headroom and the storage is expensive or slow. But remember restores: decompression is usually cheap, yet the restore process can be CPU-heavy in other ways (checksums, network, encryption).
If your backup software already compresses (or deduplicates and compresses), ZFS compression may add little. Measure before you declare victory.
Already-compressed media (JPEG, MP4, ZIP archives)
Use low or don’t bother, but “don’t bother” is not mandatory. ZFS will try to compress each block and may decide it’s not worth it; but the attempt itself costs CPU. On CPU-constrained systems serving media, set compression low. On general-purpose pools, leaving low-level compression on is often fine because the “not compressible” fast-path is reasonably efficient.
CPU “burn” is usually about concurrency, not a single thread
In production, compression rarely kills you because one block took too long. It kills you because thousands of blocks are being compressed concurrently, and suddenly your storage nodes look like batch processing servers. ZFS will happily use CPU to keep up with I/O, and if you’re running on the edge (common), the edge will move.
Second joke, and that’s the quota: the best compression level is the one that doesn’t turn your storage appliance into a distributed CPU benchmark.
Three corporate-world mini-stories
1) Incident caused by a wrong assumption: “Compression only affects disk usage”
A company I worked with ran an internal artifact repository on ZFS. It was a classic “works fine until it doesn’t” setup: devs push a lot during business hours, CI is noisy, and the storage pool was sized for capacity with “enough” CPU because the box looked overpowered on paper.
Someone changed the main dataset from compression=lz4 to compression=zstd-15 after a capacity review. The assumption was simple and very human: “compression is a storage thing.” The change went out during a quiet period, initial tests looked fine, and the compressratio number improved enough to make everyone feel clever.
Two days later, a big product branch cut happened. CI became a firehose: tarballs, containers, logs, and a lot of small files. The storage didn’t run out of space—it ran out of time. Write latency spiked, application threads piled up, and the artifact service started returning timeouts. Engineers blamed the network, then DNS (a rite of passage), and only later noticed the storage nodes sitting at high CPU with system time climbing.
The resolution wasn’t complicated: drop back to a low zstd level, restart the affected services to clear backlog, and implement a change process for storage properties. The lesson stuck because it was expensive in the way that hurts: no data loss, just a day of engineering time and a lot of “why is everything slow?” noise.
The postmortem conclusion that mattered: compression is part of the I/O pipeline. Treat it like you would treat enabling encryption or changing recordsize—measure, stage, and watch latency tails.
2) An optimization that backfired: “Let’s crank it up for better ratios”
In another environment, a team ran a ZFS-backed VM platform. Storage cost was under scrutiny, so an engineer proposed raising compression levels across the board. The idea looked great on a spreadsheet: fewer terabytes, fewer disks, fewer future purchases. The platform had CPU headroom—until it didn’t.
The backfire came from an interaction nobody modeled: the busiest hosts were also running encryption at rest and doing a lot of snapshot churn. Compression, encryption, checksumming, snapshot metadata—each individually “fine,” together forming a pipeline that started to amplify latency under load. The first symptom wasn’t even storage alarms; it was VM guest time drift and occasional filesystem complaints when guests experienced long I/O stalls.
They tried to fix it the way well-meaning engineers often do: tune more knobs. They increased ARC, tweaked prefetch, and changed queue depths. It helped a little, but the underlying problem was CPU saturation in the storage stack during peak write bursts. Compression at higher levels was the multiplier.
The eventual fix was almost boring: reset most datasets to a low zstd level, keep a higher level only for cold, compressible volumes, and add a dashboard that correlated latency (p95/p99), CPU, and IOPS with compression settings. The capacity savings decreased—but the platform stopped generating spooky “VM is haunted” tickets.
In the retro, the best line was: “We optimized for a number we could brag about, not for the thing users feel.” That’s a good rule for life, not just ZFS.
3) A boring but correct practice that saved the day: “Per-dataset policies + canaries”
A more mature organization treated ZFS properties like application configuration: versioned, reviewed, and rolled out in stages. Not because they loved process, but because they’d been burned before. They had a clear policy: general datasets get low zstd; special datasets (backups, archives) can use higher levels if they pass benchmarks; database datasets must demonstrate no p99 regression.
They also used canaries: one storage node and a small slice of workloads would get the change first. The key wasn’t perfection; it was visibility. They watched a handful of signals: write latency percentiles, CPU utilization (user/system), and application-level timeouts. If any of those moved the wrong way, rollback was immediate and uncontroversial because it was expected.
During a hardware refresh, they moved from an older CPU generation to a newer one and took the opportunity to revisit compression. Because they had baseline metrics, they could prove that zstd-3 was safe for a set of workloads that previously needed zstd-1. That bought real capacity and throughput improvements with zero drama.
The day it “saved the day” was a surprise growth event: a new team started storing large volumes of semi-structured logs. On a less disciplined platform, that would have caused either a capacity emergency or a performance incident. Here, the dataset already had the right recordsize and compression level, and the platform absorbed the growth like it had been expecting it—which, in a sense, it had.
Practical tasks: commands + interpretation (12+)
Everything below is meant to be runnable on a typical Linux system with ZFS utilities available. Output varies by distro and ZFS version; the interpretation patterns are the point.
Task 1: See current compression settings across datasets
cr0x@server:~$ zfs get -r -o name,property,value,source compression tank
NAME PROPERTY VALUE SOURCE
tank compression zstd local
tank/vm compression zstd-2 local
tank/db compression zstd-1 local
tank/backups compression zstd-6 local
tank/media compression off localInterpretation: This tells you what’s set and where it came from (local vs inherited). If you see a surprise high level inherited from a parent, that’s how “small change” turns into “platform-wide incident.”
Task 2: Check actual compression results (don’t guess)
cr0x@server:~$ zfs get -o name,used,logicalused,compressratio -r tank
NAME USED LOGICALUSED RATIO
tank 4.20T 6.10T 1.45x
tank/vm 1.80T 2.10T 1.17x
tank/db 900G 1.05T 1.19x
tank/backups 1.10T 2.70T 2.45x
tank/media 400G 410G 1.02xInterpretation: logicalused vs used gives you a reality check. Backups compress well; media doesn’t. If compressratio is near 1.0x, a high zstd level is mostly CPU cosplay.
Task 3: Find datasets that are wasting CPU (high level, low ratio)
cr0x@server:~$ zfs get -H -o name,value compression,compressratio -r tank | paste - - | head
tank zstd tank 1.45x
tank/vm zstd-2 tank/vm 1.17x
tank/db zstd-1 tank/db 1.19x
tank/backups zstd-6 tank/backups 2.45x
tank/media off tank/media 1.02xInterpretation: This quick-and-dirty view helps you spot “zstd-12 with 1.05x.” If you find that, you’ve likely found free CPU.
Task 4: Change compression safely (and understand what it does)
cr0x@server:~$ sudo zfs set compression=zstd-2 tank/db
cr0x@server:~$ zfs get compression tank/db
NAME PROPERTY VALUE SOURCE
tank/db compression zstd-2 localInterpretation: This affects newly written blocks. Existing data stays at its old compression until rewritten (via normal churn or a deliberate rewrite).
Task 5: Measure I/O latency and throughput quickly with iostat
cr0x@server:~$ iostat -x 1 5
Linux 6.6.0 (server) 12/24/2025 _x86_64_ (32 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
22.10 0.00 8.50 1.20 0.00 68.20
Device r/s w/s rkB/s wkB/s r_await w_await aqu-sz %util
nvme0n1 120.0 900.0 6400.0 51200.0 0.45 2.10 1.20 82.0Interpretation: If %util is high and awaits rise, you’re storage-bound. If awaits are fine but app is slow, you may be CPU-bound (compression or checksumming) or blocked elsewhere.
Task 6: Watch ZFS pool-level I/O and latency
cr0x@server:~$ zpool iostat -v 1 5
capacity operations bandwidth
pool alloc free read write read write
-------------------------- ----- ----- ----- ----- ----- -----
tank 4.20T 1.80T 800 1200 90M 140M
mirror 4.20T 1.80T 800 1200 90M 140M
nvme0n1 - - 400 600 45M 70M
nvme1n1 - - 400 600 45M 70M
-------------------------- ----- ----- ----- ----- ----- -----Interpretation: Bandwidth and ops tell you whether you’re pushing lots of small I/O or fewer big ones. Compression tends to matter more when you’re bandwidth-bound; it can hurt more when you’re IOPS- and latency-bound with CPU tight.
Task 7: Check ARC pressure and memory health
cr0x@server:~$ arcstat 1 5
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
12:00:01 900 120 13 40 4 60 7 20 2 48.0G 64.0G
12:00:02 880 140 15 55 6 65 7 20 2 48.1G 64.0GInterpretation: Compression changes effective cache density. If your miss rate is high, even a modest compression ratio can reduce physical reads by fitting more logical content into ARC.
Task 8: Check per-dataset recordsize and why it matters for compression
cr0x@server:~$ zfs get -o name,property,value recordsize tank/vm tank/db tank/backups
NAME PROPERTY VALUE
tank/vm recordsize 128K
tank/db recordsize 16K
tank/backups recordsize 1MInterpretation: Larger recordsize can improve compression ratio for sequential workloads (like backups) but can increase write amplification for random small writes (like databases). Don’t “standardize” this property unless you enjoy avoidable incidents.
Task 9: Confirm what’s actually compressible (quick sampling)
cr0x@server:~$ zfs get -o name,compression,compressratio -r tank | egrep 'tank/(vm|db|backups|media)'
tank/vm zstd-2 1.17x
tank/db zstd-1 1.19x
tank/backups zstd-6 2.45x
tank/media off 1.02xInterpretation: A dataset that consistently stays near 1.0x is telling you a story: either the data is incompressible, or your block size/workload shape is preventing gains. Raising the zstd level rarely fixes that.
Task 10: Force a rewrite (carefully) to apply new compression
cr0x@server:~$ sudo zfs snapshot tank/backups@pre-recompress
cr0x@server:~$ sudo rsync -a --inplace --info=progress2 /tank/backups/ /tank/backups/
cr0x@server:~$ zfs get -o name,used,logicalused,compressratio tank/backups
NAME USED LOGICALUSED RATIO
tank/backups 1.05T 2.70T 2.57xInterpretation: Rewriting data can apply new compression, but it also generates I/O, fragmentation risk, and potentially a lot of churn. Snapshot first, schedule it, and watch pool health. (And yes, rsync to itself is a blunt instrument; use it with care and understanding.)
Task 11: Observe CPU saturation during load
cr0x@server:~$ mpstat -P ALL 1 3
Linux 6.6.0 (server) 12/24/2025 _x86_64_ (32 CPU)
12:05:01 PM CPU %usr %sys %iowait %irq %soft %idle
12:05:02 PM all 72.0 18.0 0.5 0.0 0.5 9.0
12:05:02 PM 10 95.0 4.0 0.0 0.0 0.0 1.0Interpretation: If CPU is pinned while iowait is low, the system is not “waiting on disk”—it’s doing work. Compression, checksum, encryption, and kernel overhead are usual suspects.
Task 12: Check pool health and error counters before blaming compression
cr0x@server:~$ zpool status -v tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 02:14:33 with 0 errors on Sun Dec 21 03:00:00 2025
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
nvme0n1 ONLINE 0 0 0
nvme1n1 ONLINE 0 0 0
errors: No known data errorsInterpretation: Compression tuning won’t fix a drive throwing errors or a pool that’s constantly resilvering. Always verify baseline health.
Task 13: Inspect dataset properties that often travel with compression decisions
cr0x@server:~$ zfs get -o name,property,value atime,sync,logbias,primarycache,secondarycache -r tank/vm | head -n 12
NAME PROPERTY VALUE
tank/vm atime off
tank/vm sync standard
tank/vm logbias latency
tank/vm primarycache all
tank/vm secondarycache allInterpretation: People often change compression alongside other knobs and then can’t attribute cause. Capture these properties before experiments so you can roll back cleanly.
Task 14: Validate what’s being sent during replication (compressed matters)
cr0x@server:~$ sudo zfs send -nPv tank/db@hourly-001 | head
send from @ to tank/db@hourly-001 estimated size is 12.3G
total estimated size is 12.3GInterpretation: Replication size is influenced by changed blocks and their compressed size on disk. If you tune compression and rewrite data, replication traffic can change materially—sometimes for the better, sometimes as an unexpected bandwidth bill.
Fast diagnosis playbook
This is the “it’s slow right now” order of operations. The goal is to find the bottleneck in minutes, not to win an argument about compression ideology.
First: decide whether you’re storage-bound or CPU-bound
- Check CPU saturation: if CPUs are pegged and iowait is low, suspect compression/checksum/encryption.
- Check device awaits and utilization: if awaits climb and devices are hot, you’re storage-bound (or queueing too deep).
- Check application symptoms: timeouts during bursts often correlate with p99 I/O latency spikes.
cr0x@server:~$ mpstat 1 3
cr0x@server:~$ iostat -x 1 3
cr0x@server:~$ zpool iostat 1 3Second: identify which datasets are involved
Don’t tune the whole pool because one dataset is misbehaving.
cr0x@server:~$ zfs list -o name,used,logicalused,compressratio,mountpoint -r tank | head -n 20What you’re looking for: the busiest datasets, the ones with low ratios but high compression levels, and any dataset storing latency-sensitive workloads.
Third: correlate compression level with workload shape
Ask: random write heavy? sequential? already compressed? lots of small files? VM zvols? DB with 16K recordsize? Then check the actual properties.
cr0x@server:~$ zfs get -o name,compression,recordsize,volblocksize,sync -r tank/vm tank/db tank/backupsFourth: make the smallest safe change
If you suspect compression CPU overhead:
- Lower the compression level on the affected dataset first.
- Don’t rewrite everything during the incident.
- Let new writes take the new setting and re-evaluate.
cr0x@server:~$ sudo zfs set compression=zstd-1 tank/vm
cr0x@server:~$ zfs get compression tank/vmFifth: confirm improvement with the same metrics you used to declare pain
cr0x@server:~$ iostat -x 1 5
cr0x@server:~$ zpool iostat 1 5Interpretation: Improvement should show up as reduced write latency, reduced CPU saturation, fewer timeouts. If not, compression wasn’t the bottleneck—move on.
Checklists / step-by-step plan
A sane rollout plan for changing zstd levels
- Inventory current state: record compression, recordsize/volblocksize, sync settings, and current ratios.
- Choose candidate datasets: pick ones with high I/O cost or high capacity cost, not everything.
- Define success metrics: pick p95/p99 latency, CPU utilization, throughput, and error rates.
- Canary first: one dataset, one node, or one tenant.
- Change one variable: don’t mix compression level changes with recordsize changes in the same deploy.
- Observe for a full business cycle: at least one peak period.
- Roll forward or roll back: based on measured impact, not on compressratio alone.
A step-by-step method to pick a level for a new dataset
- Classify the workload: VM, DB, backup, logs, media, mixed.
- Set recordsize appropriately (don’t default blindly).
- Start low:
compression=zstd-1orzstd-2. - Measure compressibility after real data lands (not synthetic).
- If ratio is good and CPU headroom exists, test one higher level.
- Stop when marginal gains flatten or latency tails start to move.
Common mistakes (symptoms and fixes)
Mistake 1: Setting a high zstd level globally “because storage is expensive”
Symptoms: sudden CPU saturation on storage nodes, increased p99 write latency, application timeouts during bursts, and a lot of finger-pointing at “the network.”
Fix: drop to a low level on hot datasets first (VMs, DBs). Keep higher levels only for cold/sequential datasets that prove they benefit. Validate with p99 and CPU metrics.
Mistake 2: Judging success purely by compressratio
Symptoms: “We saved 20% space” followed by “Why are deployments slower?” or “Why are our VMs stalling?”
Fix: track workload outcomes: I/O latency percentiles, throughput under peak, and CPU headroom. Capacity savings are real, but they aren’t the only KPI.
Mistake 3: Forgetting that changing compression does not recompress old blocks
Symptoms: you change compression level and nothing seems to happen; ratios don’t move; teams conclude “zstd is useless.”
Fix: understand that only new writes use the new setting. If you need old data recompressed, plan a controlled rewrite (and expect extra I/O and snapshot implications).
Mistake 4: Using a “one recordsize fits all” standard
Symptoms: databases slow down after setting 1M recordsize “for better compression,” or VM latency spikes after changing recordsize on a dataset storing images.
Fix: set recordsize per workload. Compression and recordsize are coupled; tune them as a pair, but change one at a time.
Mistake 5: Blaming compression for a failing pool
Symptoms: performance degradation that coincides with rising read/write/checksum errors, scrubs taking forever, or resilvers in progress.
Fix: check zpool status and SMART/NVMe health first. A pool under repair will be slow regardless of compression.
Mistake 6: Overlooking the interaction with encryption and sync writes
Symptoms: CPU spikes and latency tails after enabling encryption and raising zstd level; synchronous workloads get worse.
Fix: treat compression level as part of a pipeline. If encryption is enabled, be more conservative on compression for hot datasets. Validate sync-heavy workloads specifically.
FAQ
1) Should I enable zstd compression everywhere?
Enable it widely at a low level if your CPUs have headroom and you want simpler operations. But “everywhere at a high level” is how you create CPU-bound storage. If you have very CPU-constrained systems serving mostly incompressible data, consider keeping compression low or off for those datasets.
2) What zstd level is the best default?
For many general-purpose datasets, zstd-1 to zstd-3 is the sweet spot: meaningful savings, low overhead. Use higher levels only when you’ve measured both improved ratios and acceptable latency.
3) Why did my compressratio not change after I set a new level?
Because existing blocks keep their original compression. Only newly written blocks use the new setting. To change historical data, you need churn or a planned rewrite (and you should snapshot first).
4) Does compression help reads or writes more?
Both can benefit. Writes may improve because fewer bytes hit disk; reads may improve because fewer bytes come off disk and fit better in caches. But compression costs CPU on write (compression) and on read (decompression). In many systems, decompression is cheap; compression is the cost center.
5) Is zstd always better than lz4 on ZFS?
No. lz4 is extremely fast and often “good enough.” zstd at low levels can be close in speed while offering better ratios, but the right choice depends on CPU headroom and workload. If you’re latency-sensitive and CPU-tight, lz4 can still be the safest default.
6) How do I know if I’m CPU-bound due to compression?
Look for high CPU usage with low iowait, rising write latency during heavy write periods, and improvement when you reduce compression level on the hot dataset. Use mpstat, iostat -x, and zpool iostat together to triangulate.
7) Will higher compression levels reduce replication bandwidth?
Often yes, because ZFS send streams reflect on-disk block sizes, and compressed blocks are smaller. But if you rewrite a lot of data to apply new compression, you may temporarily increase replication traffic due to higher churn. Measure the before/after across normal cycles.
8) Should I recompress old datasets after changing levels?
Only if you have a business reason (capacity pressure, expensive storage, replication constraints) and a maintenance window. Rewrites generate heavy I/O and can interact with snapshots and fragmentation. In many cases, letting natural churn slowly migrate blocks is the safer approach.
9) How does recordsize affect zstd level choice?
With larger blocks, you often get better compression ratios, and higher zstd levels may yield slightly more benefit. With small blocks and random writes, higher levels can cost more CPU for minimal gain. This is why VM and DB datasets tend to stay at low levels.
10) What’s the most production-safe way to experiment?
Per-dataset changes, canaries, and one variable at a time. Keep a rollback plan: you can always lower compression for new writes immediately, and you can defer recompression of old blocks until you’re confident.
Conclusion
Picking zstd compression levels in ZFS is not about finding the “best number.” It’s about placing the bottleneck where you can afford it. Low zstd levels often deliver the best return: solid space savings, better effective bandwidth, and minimal CPU pain. Higher levels can be justified—especially for sequential, cold, or backup-heavy datasets—but only when you’ve measured that the marginal ratio gains are worth the CPU and latency risk.
If you remember one operational rule: tune compression like you tune anything in production—small scope, measurable outcomes, staged rollout, and a rollback that doesn’t require heroics. Your future self, on the worst day of the quarter, will appreciate it.