ZFS has a reputation for being “magic,” usually spoken in the same tone people use for espresso machines. ARC hits are fast, checksums are comforting, and snapshots make you look competent under pressure. Then your workload grows, RAM stops being cheap (or at least stops being available), and somebody suggests: “Just add L2ARC on NVMe.”
Sometimes that’s exactly the right move. Other times, it’s a costlier way to get the same performance you’d get by simply giving ARC more memory, fixing recordsize, or stopping your app from reading the same 4 KB in a 1 MB pattern. L2ARC is not a general “make ZFS faster” button; it’s a very specific lever with sharp edges.
What L2ARC is (and what it isn’t)
ARC (Adaptive Replacement Cache) is ZFS’s primary cache, living in RAM. It caches data (and metadata) and is aggressively effective because RAM latency is tiny and ARC is tightly integrated with ZFS’s I/O pipeline.
L2ARC is a secondary cache that typically lives on a fast device like NVMe. It stores copies of data blocks that were previously in ARC and are deemed valuable enough to keep around when ARC needs space. L2ARC does not replace ARC; it extends it.
What L2ARC is not:
- Not a write cache. ZFS writes are handled by transaction groups, the ZIL, and (if present) a SLOG device. L2ARC helps reads.
- Not a guarantee. It is a cache with eviction and warming behavior, and it can be cold after boot.
- Not free. It consumes CPU and memory for headers/indexing, it generates additional reads/writes on the cache device, and it can steal bandwidth from your pool if mis-sized or misused.
Two sentences of humor, as promised, and we’ll keep it professional: Adding L2ARC to fix random read latency is like adding a second fridge to fix cooking—sometimes it helps, but only if you actually have food to store. Also, “cache” is the only word in computing that means “faster” and “more complicated” at the same time.
Interesting facts and historical context
- ARC isn’t just LRU. It blends recency and frequency using an adaptive algorithm, which matters when workloads mix scans and hot sets.
- L2ARC arrived early in ZFS’s life. It was designed when SSDs were smaller and endurance was a bigger worry, which is why its write behavior and tunables matter.
- Early L2ARC didn’t persist across reboot. Many deployments learned the hard way that a reboot could “erase” performance until cache warmed again.
- Modern OpenZFS can persist L2ARC. Persistent L2ARC reduces the “Monday morning is slow after patching” pattern—if you configure and support it correctly.
- Cache isn’t just data. ZFS metadata (dnode data, indirect blocks) often provides outsized performance wins when cached, especially for large directory trees and VM images.
- NVMe changed the economics. It made “fast enough to act like cache” much cheaper and more compact, but also made it easier to saturate PCIe lanes or thermal throttle devices under sustained writes.
- L2ARC writes can be relentless. L2ARC populates by writing data from ARC eviction streams; under pressure, it can create steady device wear and bandwidth use.
- Compression changes caching math. ARC/L2ARC store compressed blocks (when compression is enabled), effectively increasing cache “capacity” in a way many sizing spreadsheets forget.
Why ARC wins by default
ARC is where ZFS wants your hot reads to land. It’s faster than any NVMe, and it allows ZFS to make caching decisions with minimal overhead. It also caches metadata aggressively, which is often more important than caching bulk data.
In production, I’ve seen more performance wins from:
- Adding RAM (or preventing it from being stolen by ballooning / other services)
- Fixing recordsize (and volblocksize for zvols)
- Enabling compression (especially lz4) to cut I/O
- Fixing sync settings and using SLOG properly (for write latency issues)
- Separating metadata onto a special vdev when the workload is metadata-heavy
L2ARC should be considered when you have a known hot read working set that’s too large for RAM but still small enough to fit in a realistic NVMe cache device, and when the latency gap between NVMe and your pool is material.
When L2ARC on NVMe is worth it
1) You have a “warm set” that’s larger than RAM but smaller than NVMe
Classic example: a virtualization host with dozens of VMs where each VM’s “hot” blocks aren’t huge, but the total hot set is tens to hundreds of GB. ARC can’t hold it all, and the pool is on HDDs or saturated SSDs. L2ARC on a good NVMe can lift read IOPS and reduce tail latency.
2) Your pool is slow at random reads
If your main pool is HDD mirrors/RAIDZ, random reads cost you seeks. L2ARC can turn “random read penalty” into “NVMe read latency,” which is a real quality-of-life improvement for VM boots, package installs, and dependency-heavy CI builds.
3) You’re CPU-rich and latency-sensitive
L2ARC isn’t free: it adds CPU cycles for checksum, compression handling, and cache management. If the host has spare CPU and the workload punishes read tail latency (p95/p99 matters), L2ARC can be a good trade.
4) You can tolerate warm-up behavior or you have persistent L2ARC
If your fleet gets rebooted regularly (kernel upgrades, firmware, power events), a non-persistent L2ARC might make performance inconsistent. Persistent L2ARC can help, but it also adds operational considerations: device reliability, import times, and validation.
5) Your application does repeat reads, not just streaming
L2ARC loves reuse. If your workload is “read once” (backups, large sequential scans), L2ARC can become a fancy write generator that mostly caches yesterday’s news.
When ARC is enough (and L2ARC is a distraction)
1) Your pool is already NVMe/SSD and not the bottleneck
If your pool is fast SSD and your app is CPU-bound or lock-bound, L2ARC won’t fix it. You’ll just add another layer of moving parts.
2) You don’t have memory headroom
L2ARC consumes memory for headers/index structures. If you’re already tight on RAM, adding L2ARC can worsen things by shrinking effective ARC and pushing the system into reclaim pressure. The result can be worse latency and more disk I/O.
3) Your pain is writes, not reads
If the problem is sync write latency (databases, NFS with sync, VM flush storms), you’re in SLOG territory. If the problem is bulk write throughput, you’re in vdev layout, recordsize, and device bandwidth territory. L2ARC won’t do much beyond making dashboards look busier.
4) Your working set is “too big to cache” in practice
If the active set is multiple terabytes and your proposed L2ARC is 400 GB, it may not meaningfully change hit rates. L2ARC helps when it can keep a meaningful fraction of the re-read working set.
Picking the NVMe device: boring details that matter
For L2ARC, you want a device that can sustain mixed random reads and a steady stream of writes without falling off a cliff when its SLC cache exhausts. Consumer NVMe can look brilliant in benchmarks and then throttle or collapse under sustained writes—exactly the pattern L2ARC can generate when ARC is under churn.
Practical selection notes:
- Endurance (TBW/DWPD) matters. L2ARC can write constantly. If you’re adding it to “save money,” don’t choose a device that becomes a consumable.
- Power loss protection is nice to have. L2ARC is not a write journal, but sudden power loss can still create unpleasant surprises in device behavior, and persistent L2ARC benefits from predictable media integrity.
- Thermals are real. NVMe devices can throttle under sustained write. If your server has cramped airflow, L2ARC can become an accidental heater with a performance penalty.
- Don’t share the device with other heavy I/O. A “cache NVMe” that also hosts logs, containers, and a database will teach you about contention.
Sizing L2ARC: capacity, headroom, and expectations
Sizing L2ARC is less “bigger is better” and more “bigger is more metadata and more warm-up time.” L2ARC needs RAM for its bookkeeping. If you add a huge L2ARC to a modest-memory system, you can starve ARC and lose.
Rules of thumb that survive contact with production:
- Start small. A modest L2ARC (tens to a few hundred GB) can deliver most of the benefit without excessive overhead.
- Prefer more RAM over more L2ARC. If you can add RAM, do that first. ARC hits are cheaper and more predictable than L2ARC hits.
- Measure hit ratio and latency, not feelings. L2ARC can make graphs look “improved” while p99 latency stays ugly because you’re bottlenecked elsewhere.
- Expect warm-up. A cold cache means you’re back to pool performance until the workload replays enough reads.
Tuning knobs that actually move the needle
Most L2ARC tuning is about controlling churn and deciding what gets admitted. Default settings are often conservative, but “cranking it up” can hurt—especially on busy systems.
Key concepts:
- Feed rate: How quickly L2ARC is populated with evicted ARC buffers. Higher feed can warm faster but increases write load.
- Admission policy: Whether to cache only certain types of data (metadata vs data), and whether to prioritize frequently used blocks.
- Prefetch interaction: Caching prefetched data can be wasteful when workloads stream.
- Persisted L2ARC: Helpful for reboot-heavy environments, but it changes import behavior and operational expectations.
Three corporate-world mini-stories from the trenches
Mini-story #1: The incident caused by a wrong assumption
It was a mid-sized enterprise virtualization cluster. Storage was a ZFS pool on HDD mirrors (solid, if not glamorous). Performance complaints were escalating: VM boot storms in the morning, patch windows dragging, and “the storage feels slow” tickets that never included metrics.
A well-meaning engineer proposed L2ARC on an NVMe drive. The assumption: “NVMe cache = faster reads = problem solved.” They added a large cache device, and the first day looked better—some boots were quicker, and the helpdesk quieted down. Then the weekly maintenance reboot happened.
After reboot, performance cratered. Not just back to baseline—worse. The host thrashed: ARC shrank because the system was memory-constrained, L2ARC headers consumed more RAM than expected, and the cache was cold. The morning boot storm hit the slow pool, and the hypervisor queue depth spiked. Latency ballooned and a few VMs experienced I/O timeouts. Management got the kind of incident report that starts with “We recently made a change…” and ends with a new approval process.
The fix wasn’t heroic. They removed the oversized L2ARC, added RAM to restore ARC headroom, and enabled persistent L2ARC only after confirming kernel/module support and measuring import behavior. The lesson: L2ARC isn’t a “set it and forget it” accelerator; it has a relationship with RAM, and reboot behavior matters if your workload has predictable storms.
Mini-story #2: The optimization that backfired
A data engineering platform had a ZFS-backed object store for intermediate artifacts. Reads were “random enough,” and someone noticed the ARC hit rate wasn’t great. The team added a high-end consumer NVMe as L2ARC and decided to “make it really work” by increasing the L2ARC feed rate and caching more aggressively.
Within days, the NVMe started throwing media errors. SMART looked ugly. What happened wasn’t mysterious: the workload had huge churn, the cache device was being written constantly, and the drive’s sustained write behavior under heat was not what the spec sheet implied. The box wasn’t failing because NVMe is bad; it was failing because the workload turned the “cache” into a write-heavy treadmill.
Even before the errors, performance gains were inconsistent. Peak throughput sometimes improved, but tail latencies worsened during heavy ingest because the system spent time feeding L2ARC while also trying to serve reads. The optimization didn’t just fail to help—it created a new bottleneck and a new failure domain.
They replaced the device with an endurance-rated NVMe, dialed back the feed, and changed admission to avoid caching prefetch-heavy streams. The more important change: they recognized the workload was fundamentally streaming-heavy and switched the pipeline to reduce rereads. L2ARC became a modest assist, not the main strategy.
Mini-story #3: The boring but correct practice that saved the day
A finance-adjacent application ran on ZFS, with strict maintenance windows and minimal tolerance for surprise. The team wanted L2ARC for a large reference dataset queried repeatedly during business hours. They did the unfashionable thing: they staged it.
First they measured: ARC size, ARC hit ratio, cache misses, read latency distribution, and pool I/O. They established a baseline during the busiest hour and again during quiet time. Then they added a small L2ARC on a server-grade NVMe with good endurance. No tuning at first; just observe.
They also put operational guardrails in place: alerts on NVMe temperature and media errors, trend lines on L2ARC hit ratio, and a rollback procedure that included removing the cache device cleanly. They tested reboot behavior and confirmed that persistent L2ARC import time was acceptable.
When a later incident hit—an application deployment that accidentally doubled read amplification—the storage didn’t become the scapegoat. The metrics made it obvious: ARC miss rate spiked, L2ARC was helping but saturated, and the pool was still seeing too many random reads. Because they had clean instrumentation, they pushed the fix to the application and kept storage stable. Boring practices saved the day: baselines, staged changes, and alerts that triggered before users did.
Practical tasks (commands + interpretation)
Commands below assume a Linux system with OpenZFS installed. Adjust paths for your distribution. The intent here is operational: “run this, read that, decide next.”
Task 1: Identify pool layout and current cache devices
cr0x@server:~$ sudo zpool status -v
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
cache
nvme0n1 ONLINE 0 0 0
errors: No known data errorsInterpretation: Confirms whether L2ARC exists (“cache” section) and shows any errors. If you see read/write/cksum errors on the cache device, treat it like a failing component—not “just cache.”
Task 2: Confirm dataset properties that influence caching
cr0x@server:~$ sudo zfs get -o name,property,value -s local,received compression,recordsize,primarycache,secondarycache tank/datasets/vmstore
NAME PROPERTY VALUE
tank/datasets/vmstore compression lz4
tank/datasets/vmstore primarycache all
tank/datasets/vmstore secondarycache all
tank/datasets/vmstore recordsize 128KInterpretation: If secondarycache=none, L2ARC won’t help. If primarycache=metadata, you’re intentionally keeping ARC for metadata only—valid for some workloads, but it changes expectations.
Task 3: Check ARC size, target, and pressure
cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(size|c_max|c_min|memory_throttle_count)'
size 4 34359738368
c_min 4 8589934592
c_max 4 68719476736
memory_throttle_count 4 0Interpretation: ARC size is current usage; c_max is the ceiling. Non-zero memory_throttle_count suggests the ARC is being forced to back off due to memory pressure—often a sign L2ARC might hurt if you add more overhead.
Task 4: Measure ARC hit ratio vs misses (quick sanity)
cr0x@server:~$ awk '
/^hits /{h=$3}
/^misses /{m=$3}
END{
total=h+m;
if(total>0) printf("ARC hit%%: %.2f\n", (h/total)*100);
}' /proc/spl/kstat/zfs/arcstats
ARC hit%: 92.14Interpretation: High ARC hit% often means ARC is already doing great. If performance is bad anyway, your bottleneck may not be read caching.
Task 5: Check L2ARC effectiveness (hit ratio and size)
cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(l2_size|l2_hits|l2_misses|l2_read_bytes|l2_write_bytes)'
l2_hits 4 1843200
l2_misses 4 921600
l2_size 4 214748364800
l2_read_bytes 4 9876543210
l2_write_bytes 4 45678901234Interpretation: If l2_hits are tiny and l2_write_bytes huge, you might be writing a cache that rarely gets read. That’s an endurance and bandwidth tax with little payoff.
Task 6: Watch real-time ARC/L2ARC behavior (one-liner)
cr0x@server:~$ sudo arcstat 1
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c l2hits l2miss l2miss% l2read l2write
12:00:01 320 22 6 3 14 19 86 0 0 32G 64G 120 40 25 60M 280MInterpretation: You’re looking for whether misses correlate with latency pain, and whether L2ARC meaningfully reduces misses. If L2ARC is busy writing while you’re latency-bound, consider reducing feed/tuning admission.
Task 7: Confirm device health and wear (NVMe SMART)
cr0x@server:~$ sudo nvme smart-log /dev/nvme0n1
Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning : 0x00
temperature : 48 C
available_spare : 100%
percentage_used : 7%
data_units_read : 12,345,678
data_units_written : 98,765,432
media_errors : 0
num_err_log_entries : 0Interpretation: percentage_used climbing quickly after adding L2ARC is a clue you’re converting the NVMe into a write workload. Temperature approaching throttling ranges is also a red flag.
Task 8: Verify pool-level read latency and I/O mix
cr0x@server:~$ sudo zpool iostat -v tank 1 5
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 3.20T 6.80T 850 220 85.0M 12.0M
mirror-0 3.20T 6.80T 850 220 85.0M 12.0M
sda - - 430 110 42.5M 6.0M
sdb - - 420 110 42.5M 6.0M
cache - - 200 0 20.0M 0.0M
nvme0n1 - - 200 0 20.0M 0.0MInterpretation: Confirms whether reads are actually being served from cache device. Also shows if the pool is saturated on read ops.
Task 9: Add an L2ARC device safely
cr0x@server:~$ sudo zpool add tank cache /dev/nvme0n1Interpretation: This attaches the whole NVMe as cache. In many production setups you’ll want partitions and stable by-id naming; still, this is the core action.
Task 10: Remove an L2ARC device safely
cr0x@server:~$ sudo zpool remove tank nvme0n1Interpretation: Cache devices are removable. If you’re troubleshooting or you chose the wrong drive, removal is straightforward—don’t just yank the device and hope.
Task 11: Set dataset-level policies for what goes into L2ARC
cr0x@server:~$ sudo zfs set secondarycache=metadata tank/datasets/vmstore
cr0x@server:~$ sudo zfs get secondarycache tank/datasets/vmstore
NAME PROPERTY VALUE SOURCE
tank/datasets/vmstore secondarycache metadata localInterpretation: This can be a powerful control when data is huge and streaming but metadata is hot. If your L2ARC was being trashed by big sequential reads, this often calms it down.
Task 12: Check and adjust ARC limits (carefully)
cr0x@server:~$ sudo bash -c 'echo 68719476736 > /sys/module/zfs/parameters/zfs_arc_max'
cr0x@server:~$ cat /sys/module/zfs/parameters/zfs_arc_max
68719476736Interpretation: If your system has RAM but ARC is artificially capped, raising zfs_arc_max may outperform any L2ARC you could buy. Don’t set it so high that the kernel and applications starve.
Task 13: Check prefetch behavior impacts (high-level)
cr0x@server:~$ cat /sys/module/zfs/parameters/zfs_prefetch_disable
0Interpretation: Prefetch is good for sequential reads, but it can pollute caches under some patterns. Disabling prefetch globally is rarely the right first move; instead, consider dataset-level cache policies or tuning admission for L2ARC if your implementation supports it.
Task 14: Confirm compression and logical vs physical reads
cr0x@server:~$ sudo zfs get -o name,property,value compressratio tank/datasets/vmstore
NAME PROPERTY VALUE
tank/datasets/vmstore compressratio 1.78xInterpretation: Compression increases effective cache capacity. A 1.78x ratio means your ARC/L2ARC are storing less physical data for the same logical working set.
Fast diagnosis playbook
This is the “it’s slow and people are watching” sequence. You can run it in five to fifteen minutes and come out with a working hypothesis.
First: Is it reads or writes?
cr0x@server:~$ sudo zpool iostat -v tank 1 3
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 3.20T 6.80T 900 120 90.0M 8.0MInterpretation: If writes dominate and latency is the issue, look at sync behavior, SLOG, txg, and application fsync patterns. L2ARC is not the first responder.
Second: Are we cache-missing or device-bottlenecked?
cr0x@server:~$ awk '
/^hits /{h=$3}
/^misses /{m=$3}
/^l2_hits /{l2h=$3}
/^l2_misses /{l2m=$3}
END{
total=h+m;
l2total=l2h+l2m;
if(total>0) printf("ARC hit%%: %.2f\n", (h/total)*100);
if(l2total>0) printf("L2ARC hit%%: %.2f\n", (l2h/l2total)*100);
}' /proc/spl/kstat/zfs/arcstats
ARC hit%: 71.03
L2ARC hit%: 62.50Interpretation: ARC hit% low and L2ARC hit% decent suggests L2ARC is helping. ARC hit% low and L2ARC hit% low suggests either L2ARC is undersized, cold, poorly admitted, or the workload has low reuse.
Third: Is the NVMe cache itself the bottleneck?
cr0x@server:~$ sudo nvme smart-log /dev/nvme0n1 | egrep 'temperature|critical_warning|media_errors|percentage_used'
temperature : 72 C
critical_warning : 0x00
percentage_used : 31%
media_errors : 0Interpretation: High temperature can indicate throttling. Fast-growing wear suggests your cache policy is too write-heavy for the device class.
Fourth: Is memory pressure sabotaging ARC?
cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(size|c_max|memory_throttle_count)'
size 4 10737418240
c_max 4 17179869184
memory_throttle_count 4 921Interpretation: If memory_throttle_count is rising, ARC is being squeezed. Adding L2ARC in this state often makes things worse. Consider adding RAM, reducing ARC cap only if needed, and fixing the real memory consumer.
Fifth: Confirm you’re not chasing the wrong problem
cr0x@server:~$ sudo iostat -xz 1 3
avg-cpu: %user %nice %system %iowait %steal %idle
22.10 0.00 6.40 1.20 0.00 70.30
Device r/s w/s rkB/s wkB/s await %util
sda 55.0 12.0 6400.0 1100.0 18.2 92.0
nvme0n1 210.0 95.0 24000.0 33000.0 1.9 78.0Interpretation: If your disks are pegged and latency is high, cache might help—but if CPU is saturated, or the app is blocked on something else, you’re optimizing the wrong layer.
Common mistakes: symptoms and fixes
Mistake 1: Using L2ARC to fix sync write latency
Symptom: Users complain about commits/transactions being slow; metrics show write latency spikes; read metrics aren’t the issue.
Fix: Evaluate sync workload: consider a proper SLOG device, check sync settings (and whether you’re allowed to change them), and profile fsync patterns. L2ARC is irrelevant here.
Mistake 2: Oversizing L2ARC on a RAM-poor host
Symptom: After adding cache, ARC size shrinks, memory_throttle_count rises, and overall latency gets worse—especially after boot.
Fix: Reduce or remove L2ARC, add RAM, and confirm ARC max/min are reasonable. If you keep L2ARC, keep it modest and measure.
Mistake 3: Caching streaming reads and filling L2ARC with junk
Symptom: L2ARC write bytes climb quickly, L2ARC hit% remains low, NVMe wear increases, and performance gains are minimal.
Fix: Set secondarycache=metadata for streaming datasets, or otherwise restrict what gets admitted; avoid caching prefetch-heavy flows if possible.
Mistake 4: Sharing the L2ARC NVMe with other services
Symptom: Random latency spikes that correlate with container image pulls, log bursts, or database compaction on the same NVMe.
Fix: Dedicate the device (or at least isolate with partitions and I/O controls). A cache device should not be a general-purpose scratch disk in production.
Mistake 5: Expecting immediate improvement after adding L2ARC
Symptom: “We added cache and nothing changed.” Then, days later, it’s better—or worse—depending on workload churn.
Fix: Plan for warm-up. If the environment reboots often and consistency matters, evaluate persistent L2ARC and measure post-reboot performance explicitly.
Mistake 6: Ignoring NVMe thermals and endurance
Symptom: Throttling, sudden performance drops, SMART wear jumps, or media errors after “successful” deployment.
Fix: Use endurance-rated NVMe, ensure airflow, monitor temperature and wear, and tune L2ARC feed/admission to reduce write intensity.
Checklists / step-by-step plan
Step-by-step: deciding whether you need L2ARC
- Confirm the problem is read latency/IOPS. Use
zpool iostatand system latency tools; don’t guess. - Measure ARC effectiveness. If ARC hit% is already high, focus elsewhere.
- Estimate working set size. Even rough: “how much data is re-read during peak hour?” If it’s far beyond affordable NVMe, L2ARC won’t be a miracle.
- Check RAM headroom. If the host is memory-tight, plan RAM first.
- Consider dataset cache policy. If most reads are streaming, L2ARC may hurt unless restricted to metadata or hot subsets.
- Evaluate alternatives. Recordsize/volblocksize tuning, compression, special vdev for metadata, or simply faster pool vdevs may be better ROI.
Step-by-step: deploying L2ARC safely
- Choose the right NVMe. Endurance and sustained performance beat flashy benchmarks.
- Baseline metrics. Capture ARC/L2ARC stats, pool iostat, latency distributions during peak.
- Add a modest cache device first. Don’t start with “as large as possible.”
- Leave defaults initially. Observe for at least one workload cycle (a business day/week).
- Validate reboot behavior. If performance after reboot is unacceptable, evaluate persistent L2ARC or accept warm-up time.
- Set dataset-level secondarycache policy. Especially for known streaming datasets.
- Monitor NVMe health. Temperature, wear, media errors; alert before it becomes a surprise outage.
- Document rollback. Practice
zpool removeand confirm the system behaves as expected.
FAQ
1) Does L2ARC speed up writes?
No. L2ARC is for reads. If your write problem is sync latency, you’re looking for SLOG tuning and device choices. If it’s throughput, look at vdev layout and bandwidth.
2) Is it better to buy more RAM or add NVMe for L2ARC?
More RAM usually wins. ARC hits are faster and simpler than L2ARC hits, and RAM doesn’t introduce cache-device contention or endurance concerns. L2ARC is for when RAM can’t practically hold the working set.
3) How big should L2ARC be?
Big enough to hold a meaningful portion of the re-read working set, small enough to avoid excessive memory overhead and warm-up time. Start modest, measure, and scale only if hit rates and latency improve.
4) Why did performance get worse after adding L2ARC?
Common reasons: memory pressure reducing effective ARC, L2ARC being filled with streaming/prefetch data, cache device contention, or an NVMe that can’t sustain the write pattern and starts throttling.
5) Can I use a consumer NVMe for L2ARC?
You can, but you’re accepting risk: endurance limits, thermal throttling, and unpredictable sustained write behavior. For production, endurance-rated devices tend to be cheaper than the incident you’ll eventually write up.
6) Should I cache metadata only or everything?
Metadata-only caching is often the best “safe default” for mixed workloads or streaming-heavy datasets. Caching everything can help VM and database read sets, but it can also increase churn and wear if your access pattern is not reuse-heavy.
7) Does persistent L2ARC eliminate warm-up?
It reduces warm-up after reboot, but it doesn’t eliminate it in all cases. You still need to validate import behavior, device reliability, and that the persisted cache matches your current data reality after changes.
8) How do I know L2ARC is actually being used?
Look at l2_hits, l2_read_bytes, and real-time tools like arcstat. Also verify pool reads drop when L2ARC hits rise. If L2ARC is writing a lot but not reading much, it’s not doing useful work.
9) Is L2ARC the same as a special vdev?
No. A special vdev stores certain classes of data (often metadata and small blocks) as part of the pool, not as a cache. It can offer big, consistent wins for metadata-heavy workloads, but it changes redundancy and failure considerations. L2ARC is removable cache; special vdev is part of your storage.
Conclusion
L2ARC on NVMe can be a legitimately sharp tool: it reduces read latency and boosts read IOPS when your hot set is larger than RAM and your pool is slower than NVMe. But it’s not a substitute for ARC, and it’s not a universal performance fix.
In production, the winning pattern is consistent: measure first, add RAM when you can, use L2ARC when you have repeat reads and enough headroom, and keep it boring—dedicated devices, sensible sizing, and metrics that tell you when the cache is helping versus merely working hard.