People talk about ZFS compression=lz4 like it’s a universal coupon: turn it on and everything gets faster and smaller. In production, it often does feel like free money—until it doesn’t. The catch is that “free” depends on what you’re bottlenecked on, how your blocks are shaped, and what you think ZFS is actually doing under the hood.
This piece is written from the perspective of someone who has watched storage graphs at 2 a.m., argued with CPU schedulers, and learned (the hard way) that compression is both a performance feature and a systems design choice. We’ll cover when lz4 is effectively free, when it costs you, and how to diagnose it quickly without superstition.
What “free” really means in ZFS compression
In systems, “free” never means zero cost. It means the cost is paid somewhere you have surplus: spare CPU cycles, unused memory bandwidth, slack in the pipeline, or simply less pain than the alternative. With ZFS compression, the basic trade is:
Spend CPU to save I/O. That saved I/O can be disk bandwidth, IOPS, queue depth, or even time waiting on parity/RAID math and device latency. If your storage is the bottleneck, compression can make the system faster by shipping fewer bytes through the slowest part of your stack. If your CPU is the bottleneck, compression can make the system slower by adding work on the critical path.
lz4 is popular because its decompression is extremely fast and predictable, and its compression is fast enough that in many real-world storage-bound systems it feels “free.” But ZFS isn’t compressing a file; it compresses blocks (records), and the performance depends on block size, access pattern, data entropy, and how your workload hits ARC and the disks.
Here’s a useful mental model: ZFS compression is like packing a moving truck. If the highway is jammed, packing tightly gets you there sooner. If you’re late because you’re still boxing up the kitchen, packing more carefully makes you later. Same trip, different bottleneck.
Joke #1: Compression is like shaving—when it’s going well you feel smooth, and when it’s going poorly you’re bleeding and pretending it’s “just a small nick.”
Interesting facts & historical context
Some context points that tend to make compression decisions clearer:
- ZFS compression is per-dataset and online. You can enable it without rewriting existing blocks; it applies to newly written blocks.
- lz4 became the de facto default in many ZFS shops because it’s designed for speed and low CPU overhead, not maximum ratio.
- ZFS stores blocks already-compressed on disk, and (critically) those blocks are also stored compressed in ARC; decompression happens on demand.
- “Copies” and “parity” happen after compression. You generally want to compress first so you mirror/parity fewer bytes (the exact layering depends on implementation details, but the net effect is: fewer bytes written).
- Checksums cover the logical data. ZFS validates integrity regardless of compression; decompression is not a “best effort.”
- Not all data compresses. Encrypted, already-compressed media, and many modern columnar formats are often near-incompressible at the block level.
- Recordsize matters more than most people think. Compression works on the record; a mismatch between recordsize and I/O pattern can dominate performance.
- Compressed ARC can be a hidden win. If your working set is compressible, ARC effectively holds more logical data per GB of RAM.
- Historically, gzip levels were used to squeeze space on expensive disks. That made sense when CPUs were idle and storage was tight; today the cost profile is different, but the instinct lingers.
How ZFS compression actually works (the parts that matter)
Compression is a write-path decision
ZFS compression happens when blocks are written. For a dataset with compression=lz4, ZFS attempts to compress each block and only keeps the compressed version if it’s smaller by a threshold (implementation-specific, but think “don’t bother if savings are trivial”). If the block doesn’t compress meaningfully, it’s stored uncompressed. This matters because “compression enabled” doesn’t mean “everything is compressed.”
Blocks, recordsize, and why your “file” intuition misleads you
ZFS is a transactional, copy-on-write filesystem. Files are represented as a set of blocks (records). For a typical dataset, the recordsize might be 128K. That means sequential writes can be assembled into 128K records, and each 128K record gets its own compression attempt. Random I/O can fracture that story.
If your application does 8K random writes into a 128K record, ZFS still has to manage the record (often via read-modify-write semantics at some layer). Compression can then become part of a larger “small random write amplification” problem. It’s not compression’s fault, but it’s compression’s CPU time on top of an already-expensive operation.
ARC stores compressed data (and this is a big deal)
When ZFS reads a compressed block from disk, it can keep it in ARC in compressed form. That means the same RAM can cache more logical data, improving hit rates for workloads with compressible data. If you’ve ever seen a system “magically” improve after enabling lz4, this is often why: not only is disk doing less work, ARC is effectively larger.
Decompression is usually cheap; compression cost is workload-dependent
lz4 decompression is famously fast. In many workloads, the decompression is lost in the noise compared to disk latency. Compression on writes can be the more meaningful cost, especially for high-throughput write-heavy systems where the CPU is already busy with checksums, encryption (if enabled), network, or application threads.
Compression interacts with encryption, dedup, and special vdevs
Three interactions to keep straight:
- Encryption: compress before encrypt. Encrypting first destroys compressibility. Most ZFS stacks compress then encrypt when both are enabled, which is what you want.
- Dedup: dedup wants identical blocks. Compression can help or hurt depending on data; but dedup is its own beast and usually the larger risk in production.
- Special vdevs (metadata/small blocks): compressing small blocks can change the I/O mix and metadata patterns. Sometimes that’s good; sometimes it shifts pressure onto the special vdev.
When lz4 is effectively free (and sometimes faster)
lz4 feels free when CPU is not the limiting factor and I/O is. That’s common because storage is slow, latency is brutal, and many workloads move an embarrassing amount of redundant bytes.
1) Storage-bound workloads: slow disks, busy arrays, high latency
If your pool is near saturation—high device utilization, rising latencies, queues backing up—compression can reduce bytes written and read. The disk sees fewer sectors, parity math touches fewer bytes, and the whole pipeline shortens. This is the classic “fewer bytes through the straw.”
I’ve seen a noisy analytics node go from “always behind” to “mostly fine” just by enabling lz4 on its scratch dataset. Not because lz4 is magical, but because the scratch data was extremely repetitive and the disks were the bottleneck.
2) Compressible data: text, logs, JSON, CSV, VM images with zeros
Compression wins when data has patterns: repeated strings, whitespace, timestamps, zero pages, and generally low entropy. Logs and text formats are the poster children. VM images often contain large regions of zeros or lightly-used space, which compresses well even when the guest thinks it’s “allocated.”
In these cases, compression can reduce write amplification and improve read caching density. Your ARC hit ratio can improve because cached blocks represent more logical data.
3) Network or replication bound: send/receive and backups
If you replicate with zfs send or ship snapshots across the network, storing blocks compressed can reduce bytes transferred (depending on send flags and implementation). Even when the send stream isn’t “raw compressed blocks” in your setup, the overall effect often trends toward less data churn because fewer bytes are written in the first place.
4) When “CPU is cheap” is actually true
On many servers, CPUs are underutilized most of the day. Storage isn’t. If you’re running on a modern CPU with headroom, lz4 is typically a good trade: burn a bit of CPU to save expensive I/O time and device wear.
5) ARC efficiency and latency tail wins
Compression can improve the long tail of latency. When ARC holds more, more reads are served from memory. You don’t just reduce average latency; you shave off the “sometimes it hits disk and everything stalls” moments that ruin p99s. This is subtle but operationally valuable.
When lz4 isn’t free (and can be painful)
Compression stops being “free” when the extra CPU work lands on your critical path, or when the data doesn’t compress enough to justify the effort.
1) CPU-bound systems: high PPS networking, TLS, busy databases, heavy checksumming
If your system is already CPU-saturated—application threads pegged, interrupt storms, encryption overhead, checksum-heavy patterns—adding compression can push you over the edge. The symptom is often: disks are not busy, but throughput is still poor and CPU is high.
On some nodes, you can watch a single dataset flip the system from “fine” to “mysteriously slow” during a bulk load. Turn off compression and throughput rises—not because storage got faster, but because the CPU stopped doing extra work.
2) Incompressible data: already-compressed media, encrypted blobs
JPEG, H.264/H.265 video, MP3/AAC, many modern backup archives, and encrypted data are often effectively random from a compressor’s point of view. ZFS will attempt compression and frequently store the block uncompressed anyway. The attempt itself has overhead.
lz4 is fast, so the “attempt tax” is usually small, but at high throughput it’s real. If you’re writing tens of GB/s to fast NVMe, even small per-block overhead can add up.
3) Small random writes and recordsize mismatch
If you have a workload doing 4K–16K random writes into large records, you may already be paying a read-modify-write tax. Compression adds CPU, and it can also complicate how quickly ZFS can assemble, checksum, and commit transaction groups.
The fix might not be “disable compression.” It might be “use an appropriate recordsize or volblocksize for that dataset/zvol,” because you’re losing far more in amplification than you’ll ever lose or gain from lz4.
4) When you misread compressratio and optimize the wrong thing
compressratio is a seductive number. People chase it like a KPI. But it is not a direct measure of savings across all paths, and it can be misleading with mixed workloads, snapshots, and blocks that were written before compression was enabled.
5) Hot-path latency sensitivity on write-heavy workloads
Some systems care about write latency more than throughput: synchronous writes, fsync-heavy applications, certain database configurations. Compression adds CPU work before blocks are committed. If you’re already tuned tightly, lz4 can push p99 write latency up a notch.
Joke #2: The only thing more compressed than your ZFS blocks is your schedule during an outage.
Workload guidance: databases, VMs, logs, backups, media
Databases (PostgreSQL/MySQL and friends)
Databases are where people get nervous, and sometimes they’re right to be nervous. The outcome depends on I/O pattern, block size, and whether you’re already CPU-bound.
- Read-heavy with a storage bottleneck: lz4 often helps. Less read I/O, better ARC density.
- Write-heavy, CPU-bound: lz4 can hurt. Watch CPU, txg commit behavior, and latency.
- Random I/O: set
recordsizesensibly (often 16K for some DB files), and don’t use a one-size-fits-all dataset.
Operational note: many database pages are partially compressible (repeated structures, zeros). lz4 can give modest ratios but meaningful I/O savings.
VM images and container layers
VM disks can be spectacularly compressible, especially thin-provisioned images with lots of zeros and repeated OS blocks. lz4 is usually a win, and the ARC “effective size” boost can be dramatic.
The main gotcha is volblocksize for zvols: choose it to match guest I/O patterns. Compression won’t save you from a pathological block size choice.
Logs, metrics, traces
Text-heavy telemetry is the easiest call: lz4 is typically a win. You’re mostly write-heavy, and data compresses well. The savings translate into lower disk wear and faster replay/reads for recent data.
Backups and archives
If you store backups that are already compressed (many are), lz4 won’t do much. But enabling lz4 still isn’t crazy: ZFS will store incompressible blocks uncompressed, and the overhead is typically small. Still, on extremely high ingest rates (backup windows on NVMe targets), consider benchmarking.
Media repositories
For photo/video collections, lz4 rarely improves space much. The value is usually in uniform policy (“compression on everywhere”) and the occasional win for thumbnails, metadata, subtitles, or sidecar files. If your media servers are CPU-starved and serving high throughput, consider disabling compression on those datasets.
Three corporate-world mini-stories
Mini-story #1: The incident caused by a wrong assumption
The storage team rolled out a new ZFS-backed NFS tier for a mixed workload: build artifacts, logs, and a database export pipeline. Someone made a reasonable-sounding assumption: “lz4 is free; just turn it on everywhere.” It went through change review because it was considered low-risk—after all, it’s just a dataset property.
Two weeks later, during a quarterly close, the export pipeline started missing its window. Alerts fired for application lag, not storage. The first responders looked at the pool: disks weren’t pegged, latency wasn’t terrible, ARC was healthy. It didn’t smell like storage.
But CPU on the storage head was high and spiky, especially in kernel time. The write workload was mostly incompressible (already-compressed exports), but ZFS still had to attempt compression for each record. Worse, the pipeline wrote large sequential streams at high speed, which pushed compression attempts onto the hot path at a time when the CPU was already busy servicing NFS threads and checksums.
The fix wasn’t heroic: disable compression on the export dataset, keep it on for logs and artifacts. The window returned. The lesson that stuck wasn’t “compression is bad.” It was “compression is a workload decision, not a religion.” And yes, the postmortem included the phrase “free, unless it isn’t,” which is how you can tell engineers wrote it.
Mini-story #2: The optimization that backfired
A platform team wanted to reduce storage growth. They had a pool that looked healthy, but capacity charts were climbing. A well-meaning optimization sprint decided to crank compression harder: switch from lz4 to gzip-9 on several datasets because “better ratio equals less cost.” The change was made after a small test on a quiet staging node.
In production, the storage saved space—no argument there. But the week after, developers started reporting intermittent slowness: CI jobs timing out, container pulls stalling, “random” latency spikes. Storage graphs showed increased CPU, especially during peak write and snapshot activity. The pool wasn’t out of IOPS; it was out of patience.
The backfire came from two places. First, gzip-9 is significantly heavier than lz4, especially on writes. Second, the workload wasn’t dominated by cold data. It was active data with frequent writes and reads, so the CPU tax happened all day, not just once. The end result: they saved disk, but paid with productivity and operational noise.
The team rolled back to lz4 for hot datasets, kept stronger compression only for cold/archive datasets where latency didn’t matter, and documented a rule: optimize for the bottleneck you actually have. “Disk is expensive” is true; “CPU is free” is not a universal law.
Mini-story #3: The boring but correct practice that saved the day
A different org ran a ZFS fleet with a dull policy that never made it into presentations: every dataset had explicit properties set (compression, recordsize, atime, logbias where relevant), and every major change came with a before/after benchmark using production-like I/O patterns. It wasn’t fancy; it was consistent.
One day, a new application rolled out that wrote high-rate telemetry. Everyone expected it to be fine—text compresses well, right? But the engineers did the boring thing: they created a dedicated dataset, set compression=lz4, used a sensible recordsize, and ran a controlled load test with iostat and CPU profiling.
The test surfaced a surprise: not the compressor, but synchronous write behavior and a misconfigured client mount option were dominating latency. Because they had baseline numbers, they didn’t waste two days arguing about compression. They fixed the sync behavior where appropriate, kept lz4 on, and shipped the app without drama.
In the incident review, compression barely came up—exactly the point. The practice that saved the day was disciplined measurement. In production, boring beats clever when the pager is involved.
Practical tasks: commands, outputs, interpretation
Below are real tasks you can run on a ZFS system. The goal is not to memorize commands; it’s to tie observations to decisions.
Task 1: Check dataset compression settings
cr0x@server:~$ zfs get -o name,property,value,source compression tank
NAME PROPERTY VALUE SOURCE
tank compression lz4 localInterpretation: Confirms what’s set and whether it’s inherited. If your dataset inherits from a parent you forgot about, you’ll chase ghosts.
Task 2: Find which datasets differ from your policy
cr0x@server:~$ zfs get -r -s local,inherited compression tank | sed -n '1,20p'
NAME PROPERTY VALUE SOURCE
tank compression lz4 local
tank/db compression lz4 inherited from tank
tank/media compression off local
tank/backups compression lz4 inherited from tank
tank/exports compression off localInterpretation: You want intentional exceptions (like media or already-compressed exports), not accidental ones.
Task 3: See compression effectiveness per dataset
cr0x@server:~$ zfs get -o name,used,logicalused,compressratio -r tank | sed -n '1,12p'
NAME USED LOGICALUSED RATIO
tank 1.20T 1.85T 1.54x
tank/db 420G 560G 1.33x
tank/logs 180G 620G 3.44x
tank/media 300G 305G 1.02xInterpretation: logicalused vs used hints at real savings. Media barely compresses; logs compress a lot.
Task 4: Understand that existing data won’t “retro-compress”
cr0x@server:~$ zfs set compression=lz4 tank/olddata
cr0x@server:~$ zfs get -o name,compressratio tank/olddata
NAME RATIO
tank/olddata 1.01xInterpretation: If you changed compression on a dataset full of old blocks, the ratio won’t change much until new writes happen.
Task 5: Measure I/O and latency at the pool level
cr0x@server:~$ zpool iostat -v 1 5
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 6.30T 2.70T 820 1300 110M 240M
raidz2-0 6.30T 2.70T 820 1300 110M 240M
sda - - 110 180 14.0M 30.1M
sdb - - 105 175 13.8M 29.8M
sdc - - 115 185 14.2M 30.4M
sdd - - 108 182 14.0M 30.2M
sde - - 112 178 14.1M 30.0MInterpretation: If disks are busy and bandwidth is high, compression may help by reducing bytes. If disks are idle but performance is poor, your bottleneck is elsewhere (CPU, sync writes, network, app).
Task 6: Watch per-dataset I/O with ZFS iostat
cr0x@server:~$ zfs iostat -r -v tank 1 5
capacity operations bandwidth
dataset used avail read write read write
--------------- ---- ----- ----- ----- ----- -----
tank 1.20T 2.70T 420 610 52.0M 90.0M
tank/db 420G 2.70T 210 240 18.0M 22.0M
tank/logs 180G 2.70T 20 280 1.2M 38.0M
tank/media 300G 2.70T 180 40 32.0M 6.0MInterpretation: Identify who’s actually pushing I/O. Then target compression decisions by dataset instead of arguing in the abstract.
Task 7: Verify recordsize and why it matters
cr0x@server:~$ zfs get -o name,property,value recordsize tank/db tank/logs tank/media
NAME PROPERTY VALUE
tank/db recordsize 16K
tank/logs recordsize 128K
tank/media recordsize 1MInterpretation: DB dataset uses smaller records (often better for random I/O). Logs default to 128K. Media uses 1M to favor sequential throughput. Compression operates per record.
Task 8: Check zvol volblocksize for VM disks
cr0x@server:~$ zfs get -o name,property,value volblocksize,compression tank/vmdata/vm-101
NAME PROPERTY VALUE
tank/vmdata/vm-101 volblocksize 16K
tank/vmdata/vm-101 compression lz4Interpretation: A zvol’s block size is set at creation and shapes I/O. Compression “wins” are capped if your block size causes amplification.
Task 9: Observe CPU saturation during write-heavy events
cr0x@server:~$ mpstat -P ALL 1 5
Linux 6.1.0 (server) 12/24/2025 _x86_64_ (16 CPU)
12:00:01 PM CPU %usr %sys %iowait %idle
12:00:01 PM all 62.10 22.40 0.80 14.70
12:00:02 PM all 68.90 24.10 0.30 6.70Interpretation: Low iowait with low idle often means CPU-bound. If performance is bad but iowait is near zero, turning off compression might help—if the workload is incompressible or if write throughput is the goal.
Task 10: Estimate compressibility with a quick sample (safely)
cr0x@server:~$ dd if=/tank/logs/app.log bs=1M count=256 2>/dev/null | lz4 -q -c | wc -c
41234567Interpretation: Compare against original bytes (256 MiB). If compressed bytes are dramatically smaller, lz4 is likely worthwhile. This is a rough check; ZFS works per record and may differ.
Task 11: Confirm what’s actually compressed on disk (indirectly)
cr0x@server:~$ zdb -DD tank/logs | sed -n '1,20p'
DDT-sha256-zap-duplicate: 0 entries, size 0 on disk, 0 in core
Dataset tank/logs [ZPL], ID 54, cr_txg 12345, ...
Blocks Logical Physical compress% copies% dedup% checksum
------ ------- -------- --------- ------- ------ --------
... 620G 180G 70.9% 100.0% 100.0% 100.0%Interpretation: This kind of output varies by platform, but the gist is: physical vs logical block accounting. Use it carefully; it’s powerful but sharp.
Task 12: Test write throughput with and without compression (controlled)
cr0x@server:~$ zfs create -o compression=off tank/bench_off
cr0x@server:~$ zfs create -o compression=lz4 tank/bench_lz4
cr0x@server:~$ fio --name=write_test --directory=/tank/bench_off --rw=write --bs=128k --size=8g --numjobs=4 --iodepth=16 --direct=1
...
WRITE: bw=820MiB/s (860MB/s), iops=6560, runt=10000msec
cr0x@server:~$ fio --name=write_test --directory=/tank/bench_lz4 --rw=write --bs=128k --size=8g --numjobs=4 --iodepth=16 --direct=1
...
WRITE: bw=980MiB/s (1027MB/s), iops=7840, runt=10000msecInterpretation: If lz4 improves throughput, you were likely I/O-bound and the data compresses. If it reduces throughput while disks are not saturated, you’re paying CPU for little gain.
Task 13: Check ARC stats for caching effects (Linux)
cr0x@server:~$ arcstat 1 5
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
12:01:01 1200 180 15 20 2% 160 13% 0 0% 64G 96G
12:01:02 1180 140 12 18 2% 122 10% 0 0% 64G 96GInterpretation: Lower miss rates after enabling compression can be a real performance win. Compression can make ARC “feel bigger” for compressible datasets.
Task 14: Check whether a dataset is doing sync writes (and why you care)
cr0x@server:~$ zfs get -o name,property,value sync logbias tank/db
NAME PROPERTY VALUE
tank/db sync standard
tank/db logbias latencyInterpretation: If your issue is sync write latency, compression might not be the primary lever. SLOG health, sync settings, and application fsync behavior may dominate.
Fast diagnosis playbook
This is the triage sequence I use when someone says “ZFS got slow after we enabled lz4” or “lz4 didn’t help like the internet promised.” The point is to find the bottleneck quickly, not to debate ideology.
First: Determine the bottleneck class (CPU vs I/O vs latency source)
- Is CPU saturated? Check system-wide CPU and run queue. If CPU is pinned and iowait is low, you’re not disk-bound.
- Are disks saturated? Look at pool/device utilization, latency, queueing. If disks are pegged, compression can help (if data compresses).
- Is the pain sync latency? If the workload is fsync-heavy, check SLOG, sync settings, and latency distribution.
Second: Verify compressibility and where the bytes are
- Check
logicalusedvsusedandcompressratiofor the impacted dataset. - Identify the top I/O dataset using
zfs iostator per-dataset metrics. - Sanity-check the data type (logs vs media vs encrypted blobs). If it’s incompressible, don’t expect magic.
Third: Validate block sizing and access pattern
- Check
recordsize(filesystems) orvolblocksize(zvols). Mismatch causes amplification that dwarfs compression effects. - Determine random vs sequential I/O using workload knowledge or fio reproduction. Random small writes + big records is a classic trap.
Fourth: Decide the smallest safe change
- If CPU-bound and data incompressible: disable compression on that dataset, not the whole pool.
- If I/O-bound and data compressible: keep lz4, consider improving recordsize, and verify ARC behavior.
- If results are ambiguous: carve out a test dataset and benchmark with representative I/O.
Common mistakes: symptoms and fixes
Mistake 1: Enabling lz4 and expecting old data to shrink
Symptom: You set compression=lz4, but compressratio stays near 1.0x and space doesn’t drop.
Why: Existing blocks aren’t rewritten. ZFS compresses newly written blocks.
Fix: Rewrite the data (e.g., zfs send | zfs receive to a fresh dataset, or copy within the dataset with care). Do it intentionally; don’t “rm -rf and recopy” in production without a plan.
Mistake 2: Using compressratio as a performance metric
Symptom: A dataset shows 2.0x ratio, but performance is worse; or ratio is 1.1x but performance improves.
Why: Ratio measures storage accounting, not latency/throughput. ARC effects, I/O patterns, and CPU can dominate.
Fix: Measure what you care about: throughput, p95/p99 latency, CPU, iowait, device busy, ARC hit rate.
Mistake 3: Assuming incompressible data has zero cost
Symptom: CPU increases during big writes of media/archives; disk isn’t busy; throughput drops slightly.
Why: ZFS still tries to compress blocks before deciding it’s not worth it.
Fix: Disable compression on that dataset if CPU is precious and the workload is proven incompressible. Or keep it if the overhead is acceptable and policy simplicity matters.
Mistake 4: One dataset for everything
Symptom: You can’t tune recordsize, sync, or compression without hurting some workload.
Why: Dataset properties are the tuning boundary. Mixing VM images, DBs, logs, and media in one dataset forces compromise.
Fix: Split datasets by workload class and set properties explicitly.
Mistake 5: Ignoring recordsize/volblocksize and blaming compression
Symptom: Random write workload is slow; enabling/disabling compression barely changes it; latency spikes persist.
Why: Block size mismatch causes read-modify-write and fragmentation patterns that swamp compression effects.
Fix: Use appropriate block sizes (often smaller for DB, sensible for VM). Benchmark and validate.
Mistake 6: Switching to heavy compression on hot data
Symptom: CPU rises, p99 latency gets ugly, users complain “it feels laggy,” but you saved space.
Why: gzip levels are not lz4. You traded disk for CPU on a hot path.
Fix: Use lz4 for hot data. Use stronger compression only for cold/archive datasets where latency doesn’t matter.
Checklists / step-by-step plan
Checklist A: Deciding whether to enable lz4 on a dataset
- Identify the workload type (logs, DB, VM, media, backups, encrypted blobs).
- Check current bottleneck (CPU vs disk vs network vs sync latency).
- Estimate compressibility (sample data, prior knowledge, existing ratios).
- Confirm block sizing:
recordsizefor files,volblocksizefor zvols. - Enable
compression=lz4on a test dataset first if risk is non-trivial. - Benchmark with representative I/O and measure latency percentiles, not just throughput.
- Roll out to production datasets with intentional exceptions and monitoring.
Checklist B: Rolling out lz4 safely in production
- Document current dataset properties:
zfs get -r all(filtered as needed). - Pick a canary dataset and enable lz4 there first.
- Monitor CPU, pool iostat, ARC miss rate, and application p95/p99 latency for at least one business cycle.
- Expand to other datasets by workload class (logs first is usually low drama).
- Keep explicit exceptions (media, export blobs) rather than disabling globally.
- Write a rollback plan: know the command and the expected “it only affects new writes” behavior.
Checklist C: If performance got worse after enabling lz4
- Confirm the time correlation (did performance degrade exactly when new writes with compression began?).
- Check CPU saturation; compare to pre-change baselines if you have them.
- Check dataset compressibility: if ratio stays ~1.0x, lz4 is likely doing work for nothing.
- Identify which dataset is hot; don’t disable compression everywhere.
- Validate recordsize/volblocksize; fix the bigger problem first.
- Disable compression on the affected dataset if CPU-bound and incompressible, and re-test.
FAQ
1) Should I enable compression=lz4 on every ZFS dataset?
Often yes for general-purpose datasets, but “every” should include intentional exceptions. Media repositories and already-compressed export blobs are common candidates for compression=off if CPU is tight.
2) Why does lz4 sometimes make things faster?
Because you reduce I/O: fewer bytes written/read means fewer disk operations and better cache density. If you’re storage-bound, spending a bit of CPU to reduce I/O can improve throughput and latency tails.
3) Why does lz4 sometimes make things slower?
If you’re CPU-bound, compression adds work on the critical path. Also, if data is incompressible, you may pay the attempt overhead without saving I/O.
4) Does enabling compression compress existing files?
No. It applies to newly written blocks. Existing blocks remain as they are until rewritten.
5) Is compressratio the best way to judge success?
It’s a helpful clue but not the final verdict. Use it alongside application latency, pool/device utilization, CPU load, and ARC hit/miss behavior.
6) What about compression=zstd?
zstd can offer better ratios at reasonable speeds depending on level and platform support. But the topic here is lz4’s “default win” profile. If you’re considering zstd, benchmark with production-like I/O and treat compression level as a tuning knob, not a checkbox.
7) Does compression help ARC?
Yes, often. Compressed blocks can be cached in compressed form, letting ARC store more logical data per unit RAM—if the data compresses.
8) Can compression reduce disk wear on SSDs?
It can, by writing fewer physical bytes when data compresses. The real-world impact depends on workload and how compressible the write stream is.
9) Should I disable compression for encrypted datasets?
If you’re using ZFS native encryption, compression typically happens before encryption, so you can still get compression benefits. If the data is already encrypted at the application layer (random-looking), compression won’t help much and may be overhead.
10) If lz4 is so good, why not use gzip everywhere for better space savings?
Because hot-path CPU and latency matter. gzip can be great for cold archives; it can be a tax collector for active workloads. Use the right tool for the dataset’s temperature.
Conclusion
ZFS lz4 compression is “free” when it trades surplus CPU for scarce I/O and your data actually compresses. It’s not free when CPU is already your limiting reagent, when your data is incompressible, or when block sizing and access patterns are the real villains wearing compression’s jacket.
The winning approach in production is boring and repeatable: split datasets by workload, set properties intentionally, measure the right bottleneck signals, and keep a small diagnosis playbook you can run at 2 a.m. When you do that, lz4 stops being folklore and becomes what it really is: a practical lever you can pull with confidence.