L2ARC has a reputation: “Add an SSD and your ZFS pool gets fast.” Sometimes that’s true. Sometimes it’s the storage equivalent of putting a spoiler on a delivery van.
The ugly secret is that the best L2ARC configuration for plenty of real systems is… to cache nothing at all.
This piece is about secondarycache—the dataset property that decides whether blocks are eligible for L2ARC—and the operational reality behind it.
We’ll talk about why L2ARC can be neutral, why it can be actively harmful, and how to prove (not guess) what’s happening on your boxes.
What secondarycache really does
ZFS has two primary cache concepts that people conflate:
- ARC (Adaptive Replacement Cache): lives in RAM, caches frequently and recently used data (and metadata) automatically.
- L2ARC: an extension of ARC that stores eligible cached blocks on a fast device (typically SSD/NVMe).
The property that gates whether a dataset’s blocks are eligible for L2ARC is secondarycache. It’s a per-dataset setting, inheritable, and it controls what ZFS is allowed to write out to L2ARC.
Values you’ll actually use
secondarycache=all: both data and metadata may enter L2ARC.secondarycache=metadata: only metadata is eligible (often surprisingly effective for certain workloads).secondarycache=none: nothing from this dataset is eligible for L2ARC.
The punchline: setting secondarycache=none is not “turning caching off.” ARC still caches.
You are only telling ZFS: “Do not spend L2ARC bandwidth and bookkeeping on this dataset.”
If you’ve ever watched a system melt down because “we added cache,” you know why that matters.
L2ARC isn’t free; it costs CPU, memory for metadata, and write bandwidth to populate.
Joke #1: L2ARC is like a gym membership—owning it doesn’t make you faster, using it correctly might.
Interesting facts & a little history
A few concrete nuggets that explain why L2ARC behaves the way it does today:
- L2ARC arrived early and evolved slowly. The design originated in the Solaris era, when SSDs were smaller, slower, and much more expensive than today.
- L2ARC is a read cache, populated by writes. It is filled by copying ARC-evicted buffers out to the L2ARC device—meaning it consumes write I/O even if your workload is read-heavy.
- Historically, L2ARC contents vanished on reboot. For a long time, L2ARC always started “cold” after a restart; warm-up time could be hours. Persistent L2ARC arrived much later in some implementations, but operationally you still must plan for cold-cache behavior.
- L2ARC needs RAM to index what’s on disk. The cache device is useless without in-memory headers that say what’s stored where; a “big” L2ARC can quietly eat meaningful memory.
- ZFS prefetch can sabotage naive caching. Sequential reads may pull in data that looks like it should be cached but will never be reused; L2ARC can become a museum of blocks you looked at once.
- ARC is usually the best cache you’ll ever buy. RAM latency and bandwidth still dominate, and ARC is smarter than most people give it credit for—especially for metadata-heavy workloads.
- L2ARC is not a write cache. People keep trying to use it as one. That’s what SLOG (separate log) is for, and even that only helps synchronous writes.
- Compression changes the math. Caching compressed blocks means more logical data per byte, but also different CPU tradeoffs; sometimes compression makes ARC so efficient that L2ARC becomes marginal.
- Metadata caching can be the real win. File servers and VM stores often benefit more from caching indirect blocks, dnodes, and directories than from caching file contents.
L2ARC in plain English (and why it’s tricky)
ARC sits in RAM and keeps what it thinks you’ll need next. When ARC runs out of room, it evicts buffers.
Some of those evicted buffers can be written to L2ARC. Later, if the same block is requested again and it’s no longer in ARC,
ZFS can fetch it from L2ARC instead of going to your pool disks.
That sounds straightforward until you run production workloads, because “later” is doing a lot of work in that sentence.
L2ARC only helps when:
- the workload has re-reads of the same blocks,
- ARC is too small to keep them hot,
- the L2ARC device is fast enough to beat the pool,
- and populating L2ARC doesn’t steal resources needed elsewhere.
The two invisible costs: fill traffic and metadata
The first cost is fill traffic. L2ARC is populated by writing to it. Those writes compete with your pool’s normal work:
CPU cycles, memory bandwidth, and sometimes the same I/O scheduler and interrupt budget.
The second cost is metadata overhead. ZFS must remember what’s on L2ARC. That index lives in RAM. If you’re already short on memory
(the reason you wanted L2ARC in the first place), adding a large L2ARC can be like buying a bigger fridge and then complaining the kitchen is smaller.
Why “cache everything” is rarely the best policy
Most production datasets contain a mix of access patterns:
- Streaming/sequential reads (backups, media, ETL): terrible for L2ARC.
- Random re-reads (databases, VM images, small-file workloads): potentially great for L2ARC.
- Metadata churn (build servers, CI workspaces): sometimes best served by metadata-only caching or just more RAM.
The secondarycache property lets you be selective.
The best operators I know don’t “enable L2ARC”; they decide which datasets are allowed to use it, and which ones are politely told to keep out.
When caching nothing is the correct choice
“L2ARC should cache nothing” doesn’t mean “L2ARC is useless.” It means: for a particular dataset (or a whole pool), the cost of caching exceeds the benefit.
Here are the scenarios where secondarycache=none is not only reasonable, it’s often the cleanest fix.
1) The workload is mostly sequential or one-and-done
Think: backup targets, log archives, video files, data lake ingestion, nightly exports. ZFS prefetch reads ahead, ARC sees a flood of data, and L2ARC
eagerly starts writing evicted buffers. But nothing is reread. You’ve built a cache for a workload that doesn’t repeat.
Symptoms:
- L2ARC write I/O is high; L2ARC hit rate stays low.
- Pool disks show extra activity even though the “cache SSD” is busy.
- Latency is worse during heavy scans because the system is doing extra work.
2) You’re memory constrained and L2ARC steals the last good RAM
ARC needs RAM. The OS needs RAM. Applications need RAM. L2ARC also needs RAM (for headers and book-keeping).
If you’re already close to swapping or you see ARC constantly shrinking under pressure, adding L2ARC can make performance more erratic.
This is where secondarycache=none is a scalpel: you can keep L2ARC for the datasets that actually benefit, while preventing “cache thrash”
from bulk/streaming datasets.
3) The cache device is fast, but not fast enough (or not consistent enough)
Not all SSDs are equal. Some look great in a vendor spec and then fall apart under sustained writes, mixed workloads, or high queue depth.
L2ARC creates sustained writes. If the device’s write behavior is ugly, L2ARC fill can create latency spikes.
If your pool is already on NVMe, a “cache NVMe” might not be meaningfully faster than the pool.
In that case, you pay L2ARC overhead to replace one fast read with another fast read. It’s not a crime, but it is a waste.
4) Your real bottleneck is not reads
I’ve watched teams chase L2ARC hits while the actual problem was:
- synchronous write latency (needs SLOG tuning, recordsize alignment, or application changes),
- CPU saturation from compression/checksumming,
- network latency,
- fragmentation and small record sizes,
- or simply “too many clients.”
In those cases, L2ARC is a side quest. Setting secondarycache=none on irrelevant datasets prevents L2ARC from consuming resources
while you fix the real issue.
5) You’re caching data that should be “special” instead
L2ARC is a read cache. A special vdev is different: it can permanently store metadata (and optionally small blocks) on fast media.
If your pain is metadata latency (directory traversals, small random reads, VM metadata), the “right” acceleration might be special vdev—not L2ARC.
That doesn’t mean “always deploy special vdev.” It means: if you’re using L2ARC to paper over a metadata problem, you may be paying continuous overhead
for a benefit that could be made structural.
6) You don’t have a stable reuse window
L2ARC shines when your working set is larger than RAM but still has locality—meaning blocks get reread within minutes/hours.
Some workloads have a working set larger than RAM and no useful reuse. You can’t cache your way out of entropy.
Joke #2: If your workload is pure streaming, L2ARC is basically a very expensive souvenir shop—lots of inventory, nobody comes back.
Fast diagnosis playbook (first, second, third)
When performance is bad and L2ARC is in the mix, you need a fast, repeatable triage routine. Here’s what I check, in order, when someone pings “storage is slow”
and there’s an L2ARC device proudly installed.
First: is it a read problem, a write problem, or a latency problem?
- Look at pool latency and ops: are reads slow, writes slow, or both?
- Check whether the app is blocked on I/O or CPU.
- Confirm whether the slowdown correlates with sequential scans, backups, replication, or scrubs.
Second: is ARC already doing the job?
- If ARC hit ratio is high and stable, L2ARC often won’t help.
- If ARC is under memory pressure, fix that before “adding cache.”
Third: is L2ARC actually serving reads (hits), or just being written to?
- Check L2ARC hit rate, feed rate, and the ratio of L2ARC reads to writes.
- If L2ARC is busy writing but not producing hits, clamp it down with
secondarycache.
Fourth: does the cache device show bad behavior?
- Watch device latency during fill.
- Look for sustained high utilization, queueing, or periodic stalls.
- Verify it’s not sharing lanes/controllers with other hot devices.
Fifth: narrow the blast radius
- Set
secondarycache=noneon obvious streaming datasets first. - Keep metadata caching for datasets where it helps.
- Measure again. Don’t “feel” your way through this.
Hands-on tasks: commands + interpretation
The goal here is not “run these commands once.” It’s to build a habit: observe, change one thing, observe again.
Below are practical tasks you can do on a real ZFS host. Output varies by platform, but the intent and interpretation hold.
Task 1: Confirm which datasets are eligible for L2ARC
cr0x@server:~$ zfs get -r -o name,property,value,source secondarycache tank
NAME PROPERTY VALUE SOURCE
tank secondarycache all default
tank/backups secondarycache all inherited from tank
tank/vmstore secondarycache all inherited from tank
tank/logarchive secondarycache all inherited from tank
Interpretation: if you see streaming datasets like tank/logarchive inheriting all, that’s a red flag.
You are allowing L2ARC to ingest the least cacheable data on the system.
Task 2: Set streaming datasets to cache nothing
cr0x@server:~$ sudo zfs set secondarycache=none tank/logarchive
cr0x@server:~$ sudo zfs set secondarycache=none tank/backups
Interpretation: this doesn’t purge ARC; it simply stops new blocks from those datasets being admitted to L2ARC going forward.
You’ve reduced future L2ARC churn.
Task 3: Try metadata-only caching for “lots of filenames, not much reread data”
cr0x@server:~$ sudo zfs set secondarycache=metadata tank/home
cr0x@server:~$ zfs get secondarycache tank/home
NAME PROPERTY VALUE SOURCE
tank/home secondarycache metadata local
Interpretation: metadata-only is a pragmatic middle ground. For file servers, it can improve directory traversal and stat-heavy workloads
without filling L2ARC with file contents that won’t be reread.
Task 4: Verify you actually have an L2ARC device attached
cr0x@server:~$ zpool status -v tank
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
cache
nvme0n1p1 ONLINE 0 0 0
errors: No known data errors
Interpretation: L2ARC devices appear under cache. If there’s no cache vdev, secondarycache still matters only in the sense of future policy.
Task 5: Check whether the pool is IOPS-bound or bandwidth-bound
cr0x@server:~$ zpool iostat -v tank 1 5
capacity operations bandwidth
pool alloc free read write read write
-------------------------- ----- ----- ----- ----- ----- -----
tank 6.12T 5.88T 410 120 52.1M 14.2M
raidz2-0 6.12T 5.88T 410 120 52.1M 14.2M
sda - - 80 25 10.6M 3.1M
sdb - - 85 24 10.5M 3.0M
sdc - - 78 22 10.2M 2.8M
sdd - - 82 25 10.8M 3.2M
-------------------------- ----- ----- ----- ----- ----- -----
Interpretation: if you’re seeing large bandwidth with low IOPS, it smells like sequential streaming. That’s where L2ARC often does nothing useful.
If IOPS are high and reads are random, L2ARC might help—if the reuse exists.
Task 6: Watch system-level I/O latency and identify the actual pain
cr0x@server:~$ iostat -x 1 5
avg-cpu: %user %nice %system %iowait %steal %idle
12.30 0.00 4.10 8.90 0.00 74.70
Device r/s w/s rkB/s wkB/s await r_await w_await %util
sda 95.0 28.0 11520.0 3200.0 18.2 16.9 22.1 96.0
nvme0n1 5.0 900.0 80.0 51200.0 2.1 1.2 2.2 82.0
Interpretation: look at await and %util. Here the NVMe is doing a lot of writes (likely L2ARC feed),
while the HDDs are near saturation. If L2ARC isn’t producing hits, those writes are just extra work.
Task 7: Check ARC and L2ARC counters (common on Linux with OpenZFS)
cr0x@server:~$ kstat -p zfs:0:arcstats:hits zfs:0:arcstats:misses zfs:0:arcstats:l2_hits zfs:0:arcstats:l2_misses
zfs:0:arcstats:hits 389002134
zfs:0:arcstats:misses 54200121
zfs:0:arcstats:l2_hits 120934
zfs:0:arcstats:l2_misses 3812490
Interpretation: ARC hits dwarf misses (good), and L2ARC hits are tiny compared to L2 misses (bad). In this shape, L2ARC is not providing value.
You should consider restricting secondarycache and/or removing L2ARC entirely if it causes overhead.
Task 8: Calculate quick-and-dirty hit ratios
cr0x@server:~$ python3 - <<'PY'
hits=389002134
misses=54200121
l2hits=120934
l2misses=3812490
print("ARC hit ratio:", hits/(hits+misses))
print("L2ARC hit ratio:", l2hits/(l2hits+l2misses))
PY
ARC hit ratio: 0.8777331151094286
L2ARC hit ratio: 0.03076857030623538
Interpretation: a ~3% L2ARC hit ratio is a strong hint that the device is mostly acting as a write sink for evicted buffers.
There are edge cases, but in practice this usually means “stop feeding it junk.”
Task 9: Watch L2ARC feed rate and size (platform-dependent, still useful)
cr0x@server:~$ kstat -p zfs:0:arcstats:l2_size zfs:0:arcstats:l2_asize zfs:0:arcstats:l2_write_bytes zfs:0:arcstats:l2_read_bytes
zfs:0:arcstats:l2_size 81234165760
zfs:0:arcstats:l2_asize 79877812224
zfs:0:arcstats:l2_write_bytes 918230012928
zfs:0:arcstats:l2_read_bytes 9112346624
Interpretation: nearly a terabyte written, and only ~9 GB read back. That’s not “cache,” that’s a treadmill.
It also implies wear on the cache device for very little return.
Task 10: Identify datasets causing heavy reads (to decide who gets L2ARC)
cr0x@server:~$ sudo zpool iostat -r -v tank 1 3
capacity operations bandwidth
pool alloc free read write read write
-------------------------- ----- ----- ----- ----- ----- -----
tank 6.12T 5.88T 900 140 95.0M 16.0M
raidz2-0 6.12T 5.88T 900 140 95.0M 16.0M
-------------------------- ----- ----- ----- ----- ----- -----
Interpretation: depending on your OpenZFS build, you may not get per-dataset iostat here.
If you do have per-dataset tooling in your environment, use it; otherwise correlate by application logs and access patterns.
The operational point: pick winners and losers for secondarycache rather than enabling everything.
Task 11: Inspect recordsize and see whether you’re caching giant blocks you never reread
cr0x@server:~$ zfs get -o name,property,value recordsize tank/backups tank/vmstore
NAME PROPERTY VALUE SOURCE
tank/backups recordsize 1M local
tank/vmstore recordsize 128K local
Interpretation: a backup dataset with recordsize=1M is often large, sequential, and not reread.
Caching 1 MB blocks into L2ARC is rarely a win. This is a prime candidate for secondarycache=none.
Task 12: Make the change safely, then validate with before/after snapshots of counters
cr0x@server:~$ kstat -p zfs:0:arcstats:l2_hits zfs:0:arcstats:l2_misses zfs:0:arcstats:l2_write_bytes > /tmp/l2arc.before
cr0x@server:~$ sudo zfs set secondarycache=none tank/backups tank/logarchive
cr0x@server:~$ sleep 300
cr0x@server:~$ kstat -p zfs:0:arcstats:l2_hits zfs:0:arcstats:l2_misses zfs:0:arcstats:l2_write_bytes > /tmp/l2arc.after
cr0x@server:~$ diff -u /tmp/l2arc.before /tmp/l2arc.after
--- /tmp/l2arc.before
+++ /tmp/l2arc.after
@@
-zfs:0:arcstats:l2_write_bytes 918230012928
+zfs:0:arcstats:l2_write_bytes 918380112896
Interpretation: after five minutes, L2ARC write bytes barely moved. That’s what you want if those datasets were the bulk of your feed.
Now confirm that application latency improved and pool I/O decreased. Metrics win arguments.
Task 13: Check for memory pressure that could make L2ARC counterproductive
cr0x@server:~$ free -h
total used free shared buff/cache available
Mem: 128Gi 92Gi 4.1Gi 1.2Gi 32Gi 28Gi
Swap: 16Gi 3.4Gi 13Gi
Interpretation: swap usage on a ZFS server is not always evil, but sustained swap-in during peak I/O is a performance tax.
If you see swapping plus L2ARC overhead, reducing L2ARC admission (via secondarycache) is often a quick win.
Task 14: If you suspect the cache device is misbehaving, look at its error counters
cr0x@server:~$ sudo smartctl -a /dev/nvme0n1 | sed -n '1,120p'
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0] (local build)
=== START OF INFORMATION SECTION ===
Model Number: ExampleNVMe 1TB
Serial Number: XYZ123
Firmware Version: 1.02
...
Percentage Used: 23%
Data Units Written: 912,345,678
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Interpretation: if “Data Units Written” is skyrocketing in a system where L2ARC isn’t yielding hits, you are burning endurance for sport.
That’s another reason to set secondarycache=none for the wrong datasets.
Three corporate-world mini-stories
Mini-story #1: The incident caused by a wrong assumption
A mid-sized company ran a ZFS-backed NFS cluster for developer home directories and build artifacts.
They had a busy season—new product, lots of CI, lots of “just run it again”—and the storage team got the classic complaint:
builds were stalling on I/O, especially in the afternoon.
Someone proposed the obvious solution: “Add L2ARC.” They had spare enterprise SSDs, they had a maintenance window, and the change was easy.
Afterward, dashboards looked “improved” in the way dashboards do when everyone wants them to: the cache device showed activity, which felt like success.
Two days later, the real incident: clients started reporting intermittent hangs. Not constant slow performance—hangs.
The storage nodes weren’t crashing, but latency graphs looked like a heart monitor.
The helpdesk tickets were almost poetic: “Sometimes it’s fine, sometimes it freezes for 20 seconds.”
Root cause was a wrong assumption: they assumed the workload was “random rereads.” It wasn’t. The dominant traffic was CI artifact uploads and large sequential
reads of build outputs that were rarely reread on the same node. ZFS prefetch did what it does, ARC filled and evicted, and L2ARC feed started hammering the SSDs.
The cache device wasn’t serving reads; it was busy ingesting evictions. Meanwhile, the system paid the bookkeeping cost and the I/O scheduling cost.
The fix was embarrassingly simple: set secondarycache=none on the build artifact datasets, and secondarycache=metadata on home directories.
L2ARC stopped acting as a churn amplifier, latency stabilized, and the “random hang” tickets disappeared.
The lesson stuck: if you can’t describe the reuse window, you’re not tuning a cache—you’re gambling.
Mini-story #2: The optimization that backfired
Another shop ran a virtualized environment where a ZFS pool served VM disks via iSCSI.
They had good NVMe for L2ARC, modest RAM, and a lot of VMs. Somebody had read that “L2ARC is great for VM workloads,” which is often true.
They set secondarycache=all at the pool root, so every dataset inherited it.
Performance initially improved in a narrow test: boot storms were faster. That was the trap.
In production, the environment included nightly full-VM backups that read every block of every VM. Those backups were sequential and enormous.
Once the backup kicked off, L2ARC feed rate spiked, the cache device wrote continuously, and latency for normal VM I/O climbed.
The backfire had a specific flavor: the storage graphs showed the cache device “helping” (busy!), yet the VMs were slower.
Engineers tried to “fix” it by adding a second cache device and increasing cache size, which made the fill traffic worse and consumed more RAM for indexing.
At that point the cache wasn’t a cache; it was a background job with a very expensive SSD.
The turning point came from a boring measurement: they compared L2ARC read bytes to write bytes during the backup window.
Writes dwarfed reads. The cache device was absorbing data that would never be reread before eviction.
They changed policy: backups dataset set to secondarycache=none, VM disks dataset kept at all.
After that, boot storms stayed improved, backups stopped punishing everyone else, and the L2ARC device’s endurance stopped dropping like a countdown timer.
The moral: “L2ARC for VMs” is not wrong; “L2ARC for everything in the same pool” often is.
Mini-story #3: The boring but correct practice that saved the day
A financial services team (the kind that fears unpredictable latency more than they fear slow latency) ran a ZFS appliance for analytics.
Their primary workload was a database with a known working set, plus occasional large exports and re-index operations.
They had a standing rule: no caching changes without a before/after counter capture and a rollback plan.
When the DB team asked for “more cache,” storage engineering didn’t argue. They asked: “Which dataset, which access pattern, and what’s the reuse window?”
The DB dataset got secondarycache=all. The exports dataset got secondarycache=none. The shared staging area got metadata.
This wasn’t heroics; it was just refusing to treat all I/O as morally equivalent.
Months later, a platform patch required a reboot. After the reboot, performance complaints arrived—predictably—because caches were cold.
But the incident didn’t become a fire. They had baseline counters and an expectation for warm-up time.
More importantly, because the export dataset never fed L2ARC, the cache warmed on the DB’s real working set instead of drowning in export traffic.
They ended up looking lucky. They weren’t. They were boring on purpose.
The practice that saved the day was policy: explicit secondarycache decisions per dataset, validated with counters, revisited quarterly.
“No surprises” is a feature you can engineer.
Common mistakes: symptoms and fixes
Mistake 1: Enabling L2ARC globally and forgetting that backups exist
Symptoms: L2ARC device shows constant writes during backup windows; pool latency increases; L2ARC hit ratio remains low.
Fix: Set secondarycache=none on backup/replication/export datasets. Consider metadata for directories and catalogs.
Mistake 2: Interpreting “L2ARC is busy” as “L2ARC is working”
Symptoms: High write bandwidth to cache device; little read bandwidth; no meaningful improvement in application latency.
Fix: Compare l2_write_bytes to l2_read_bytes. If reads don’t follow writes, restrict admission with secondarycache.
Mistake 3: Using L2ARC to compensate for insufficient RAM without measuring memory pressure
Symptoms: Swapping, kswapd activity, jittery latency, ARC constantly shrinking; L2ARC still not producing hits.
Fix: Reduce L2ARC footprint (or remove) and prioritize RAM. If you must keep L2ARC, use secondarycache=metadata for many datasets.
Mistake 4: Expecting L2ARC to fix synchronous write latency
Symptoms: Commit latency high, database fsync slow, NFS sync writes slow, but read metrics look fine.
Fix: Look at SLOG, sync behavior, and write path tuning. L2ARC is not a write cache; secondarycache changes won’t help here.
Mistake 5: Caching “too large” blocks and destroying cache efficiency
Symptoms: L2ARC fills quickly; hit rate remains low; workload is mixed but dominated by large record reads.
Fix: Set secondarycache=none for datasets with large streaming records (often backups), or reduce what’s eligible via dataset separation.
Mistake 6: Treating L2ARC as mandatory once installed
Symptoms: Persistent performance regressions; teams resist disabling it because “we paid for the SSD.”
Fix: Operate for outcomes, not sunk cost. Disable admission for the wrong datasets; if necessary, remove the cache vdev and redeploy SSD elsewhere.
Checklists / step-by-step plan
Step-by-step: deciding whether a dataset should cache nothing
- Identify the dataset’s access pattern. If it’s sequential scans, assume
secondarycache=noneuntil proven otherwise. - Capture baseline counters. Record pool iostat, ARC hits/misses, L2ARC read/write bytes.
- Change only one dataset first. Start with the most obviously streaming dataset (backups/log archives).
- Apply
secondarycache=none. Do not change five knobs at once. - Wait for a representative window. At least one workload cycle (e.g., a full backup run or peak hour).
- Compare after metrics. L2ARC write bytes should drop; pool latency should improve or stay stable.
- Iterate. Move the policy boundary. Keep
allfor proven win datasets, usemetadatafor mixed file workloads, andnonefor streaming.
Operational checklist: “We have L2ARC, but it’s not helping”
- Confirm you’re not CPU-bound or network-bound.
- Check ARC hit ratio and memory pressure. Fix RAM starvation first.
- Measure L2ARC hits, misses, and read/write bytes.
- If L2ARC is write-heavy with low hits, stop feeding it with
secondarycache=noneon streaming datasets. - If L2ARC is hit-heavy but still slow, inspect cache device latency and health.
- If improvements are still marginal, consider that your pool is already fast enough or the working set lacks reuse.
FAQ
1) Does secondarycache=none disable ARC?
No. ARC continues as normal in RAM. secondarycache only controls admission to L2ARC (the secondary cache device).
2) If L2ARC is a read cache, why does it write so much?
Because it must be populated. ZFS fills L2ARC by writing evicted ARC buffers to the cache device. If your workload streams through lots of unique data,
L2ARC will write constantly while producing few hits.
3) Should I set secondarycache=metadata everywhere?
Not everywhere, but it’s a strong default for mixed file workloads where metadata reuse is high and data reuse is uncertain.
It’s also a safer choice when you suspect “cache pollution” from large streaming reads.
4) What’s the difference between primarycache and secondarycache?
primarycache governs what can enter ARC (RAM). secondarycache governs what can enter L2ARC (cache device).
Most systems should be very cautious changing primarycache; use secondarycache to manage L2ARC policy.
5) If L2ARC doesn’t persist across reboot, is it useless?
Not useless, but you must account for warm-up time. If your system reboots frequently or your performance SLOs can’t tolerate cold-cache periods,
L2ARC may be the wrong tool—or you must ensure that only high-reuse datasets feed it so it warms quickly on what matters.
6) Can L2ARC make performance worse even if it gets some hits?
Yes. If the fill and bookkeeping overhead causes CPU contention, memory pressure, or device latency spikes, you can end up slower overall.
The cure is selective admission: set secondarycache=none where the hit rate and reuse don’t justify the cost.
7) How do I know whether my workload has enough reuse for L2ARC?
Measure. A practical proxy is: L2ARC read bytes should be a meaningful fraction of L2ARC write bytes over a representative window,
and application latency should improve. If you mostly write to L2ARC and rarely read from it, the reuse window is not there.
8) Is adding more L2ARC capacity a good fix for low hit ratio?
Sometimes, but it’s usually the wrong first move. Low hit ratio is more often a policy problem (caching the wrong datasets) than a size problem.
Increase capacity only after you’ve excluded streaming datasets via secondarycache=none and confirmed real reuse.
9) Should I remove the L2ARC device entirely if I set many datasets to none?
If you end up with only a small set of datasets that benefit, keeping L2ARC can still be worthwhile.
But if your measurements show minimal hits and significant overhead, removing it is valid engineering. SSDs have better uses than being a disappointment.
Conclusion
L2ARC is not magic; it’s a trade. The dataset property secondarycache is how you control that trade with precision.
If you remember only one thing: a cache that serves little is not “harmless.” It burns resources—CPU, RAM, write bandwidth, and SSD endurance.
The most production-hardened L2ARC configuration I’ve seen is also the least dramatic: cache the datasets that demonstrably reread,
consider metadata-only where it fits, and set secondarycache=none for streaming workloads without apology. Caching nothing can be the fastest option.