ZFS arcstat: The Fastest Way to See if Cache Helps

What arcstat is (and what it isn’t)

arcstat is a tiny script that samples ZFS ARC statistics and prints them as a live dashboard. You run it, you watch it,
you stop guessing. It doesn’t “tune” anything. It doesn’t fix anything. It just tells you what the cache is doing right now:
hits, misses, sizes, and sometimes the shape of your workload.

If you’re trying to answer “is cache helping?” there’s a brutally honest test:
if your ARC hit rate is low and your disks are overloaded, cache isn’t saving you.
If your ARC hit rate is high and your disks are calm, ARC is doing paid work.

The trap: people treat ARC stats like a vanity metric. “Look at that 99% hit ratio!” Great—on what? Metadata? Tiny files? Prefetched junk?
Your job is to map the numbers to the real bottleneck: latency and throughput where the application actually waits.

ARC in plain ops terms

ARC (Adaptive Replacement Cache) is ZFS’s in-memory cache. It’s not “a page cache.” It competes with your OS page cache, your application
memory, and your sense of calm. It caches data blocks and metadata, and it’s aggressive because disk is slow and RAM is fast.

ARC is “adaptive” because it tries to balance two classes of cached items:
recently used and frequently used. In practice, this means ARC tries to survive workloads that would wreck simpler LRU caches:
scan-heavy reads, mixed random workloads, and metadata-heavy filesystem operations.

But ARC can’t break physics. If your working set doesn’t fit in memory, your hit rate will suffer. If your workload is mostly streaming reads,
caching can be neutral or even harmful. If you’re memory-constrained and ARC grows too large, you’ll trade disk waits for swap storms—
a bargain no one should accept.

One quote worth keeping near your change window:
Hope is not a strategy. — Gene Kranz

Interesting facts & history you can weaponize in meetings

• ARC wasn’t invented for ZFS. The ARC algorithm was published in 2003 and later adopted by ZFS because it adapts well under changing workloads.
• ZFS’s cache is block-based, not file-based. It caches blocks from the DMU layer, which is why “file cache” intuitions often fail.
• ARC caches metadata aggressively. For many environments (VM hosts, build servers, Git repos), metadata hits are the difference between “fine” and “why is ls slow?”
• L2ARC is not a write cache. It’s an extension of ARC onto fast devices, but it still needs ARC to index it, so it consumes RAM to use “more cache.”
• L2ARC used to be volatile across reboots. Early implementations essentially started cold; later improvements made persistent L2ARC possible in some implementations.
• ZFS has a separate intent log (ZIL/SLOG) path. ARC helps reads; synchronous write latency is often more about ZIL behavior than ARC.
• Prefetch can be a silent ARC bully. For sequential workloads, ZFS prefetch may fill ARC with data you’ll never reuse, pushing out valuable blocks.
• Compression changes the cache math. ARC stores compressed blocks (in many implementations), so compression can effectively increase cache capacity for compressible data.
• ARC hit ratio can look “good” while users suffer. Because the misses that matter might be on the critical path (e.g., small random reads) while hits are on cheap operations.

Joke #1: ARC is like office snacks—everyone loves it until it eats the budget and someone starts hiding granola bars in their desk.

Fast diagnosis playbook (first/second/third)

First: confirm the system is actually waiting on storage

Don’t start with ARC. Start with “what is the bottleneck.” If CPU is saturated, ARC won’t save you. If you’re swapping, ARC is part of the problem.
If disks are idle but latency is high, you might be blocked elsewhere (network, locks, single-threaded app behavior).

Second: use arcstat to classify the workload

Watch ARC hits/misses over time while the problem is happening. Look at read demand and miss rate, not just the “hit%” column.
A hit ratio without context is a KPI designed to make you feel better.

Third: decide what knob actually matters

Based on what you find:

• If ARC is too small for the working set, you need more RAM or a workload change (recordsize, primarycache, access pattern).
• If ARC is large but thrashing, you likely have streaming reads, too much prefetch, or a workload that defeats caching; look at application behavior and dataset properties.
• If ARC hit rate is fine but latency is bad, stop staring at ARC and look at sync writes, fragmentation, vdev layout, and queue depth.

How to read arcstat like you’re on call

arcstat outputs columns that vary slightly by implementation, but the core concepts stay stable:
cache size, target size, hits, misses, demand reads, prefetch reads, and often separate metadata stats.

The mental model that works in production:

• Hits save I/O. Misses create I/O.
• Demand reads are reads the application asked for. Prefetch reads are ZFS guessing the future.
• Demand misses hurt latency. Prefetch misses mostly hurt bandwidth and cache pollution.
• ARC size should be stable under steady load. Wild swings mean memory pressure or a workload that whips the cache.

Stop worshiping hit ratio; start correlating with pain

A hit ratio is a fraction. Fractions lie by omission. You can have a great hit ratio while your app is blocked on the few misses
that touch a hot table or a VM’s random read pattern. You can also have a mediocre hit ratio with perfectly acceptable latency
if the misses are sequential and your disks can stream them efficiently.

ARC size vs target: who’s winning the RAM fight

ARC usually has a “target” and a current “size.” If ARC can’t reach its target, something else is demanding memory. If ARC keeps shrinking,
the kernel is reclaiming pages—sometimes correctly, sometimes violently. The failure mode is predictable: memory pressure causes ARC eviction,
ARC eviction increases disk I/O, disk I/O increases latency, latency increases queueing, and now you’re troubleshooting the wrong layer.

Demand vs prefetch: your workload’s personality test

If demand hits are high, ARC is doing its job. If prefetch reads dominate and prefetch hits are low, you’re likely caching garbage.
That’s when you look at dataset primarycache, secondarycache, recordsize, and whether you should disable prefetch for specific workloads.
You don’t “turn off prefetch” globally because a single box had a bad day.

Joke #2: Turning on L2ARC to fix a workload is like buying a faster toaster because dinner service is slow.

Practical tasks: commands, output meaning, and decisions

These are real tasks I’ve used during incidents and performance work. Each one has three parts: the command, what the output means,
and the decision you make from it. Run them while the problem is happening. Historical averages are cute; live symptoms pay the bills.

Task 1: Run arcstat live and identify demand vs prefetch

cr0x@server:~$ arcstat 1
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
12:01:01   822    61     7    34    4    27    3     2    0   42.1G  48.0G
12:01:02   910    80     9    52    6    26    3     2    0   42.2G  48.0G

Meaning: read is total ARC read activity; miss/miss% is overall misses; dmis/dm% are demand misses; pmis/pm% are prefetch misses; arcsz is ARC size; c is ARC target.

Decision: If demand misses are high during user-visible latency, cache isn’t covering the working set. If prefetch misses dominate, you may be polluting ARC with read-ahead.

Task 2: Confirm ARC isn’t being forced to shrink (memory pressure)

cr0x@server:~$ arcstat -f time,arcsz,c,avail 1
    time   arcsz      c    avail
12:03:10  39.8G  48.0G    1.2G
12:03:11  38.9G  48.0G  900.0M
12:03:12  37.5G  48.0G  620.0M

Meaning: ARC is shrinking while available memory is collapsing. That’s the kernel asking for RAM back.

Decision: If avail is low and ARC is shrinking, prioritize memory: reduce ARC max, fix the memory hog, or add RAM. Don’t add L2ARC; it costs RAM too.

Task 3: Check ARC summary counters (Linux)

cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(hits|misses|demand_data_hits|demand_data_misses|prefetch_data_hits|prefetch_data_misses|size|c) '
hits                            4 231889442
misses                          4 18122344
demand_data_hits                4 168221991
demand_data_misses              4 12188511
prefetch_data_hits              4 23311233
prefetch_data_misses            4 4932100
size                            4 45188173824
c                               4 51539607552

Meaning: Raw counters since boot. Use them to compute ratios and confirm what arcstat shows.

Decision: If demand_data_misses climbs rapidly during incidents, fix working set fit (RAM, recordsize, caching policy). If only prefetch misses climb, address prefetch behavior.

Task 4: Compute demand hit ratio quickly (without lying to yourself)

cr0x@server:~$ python3 - <<'PY'
import re
d={}
for line in open("/proc/spl/kstat/zfs/arcstats"):
    m=re.match(r'^(\w+)\s+\d+\s+(\d+)$', line.strip())
    if m: d[m.group(1)]=int(m.group(2))
h=d.get("demand_data_hits",0); m=d.get("demand_data_misses",0)
print("demand_data_hit_ratio = {:.2f}%".format(100*h/(h+m) if (h+m) else 0))
PY
demand_data_hit_ratio = 93.25%

Meaning: Demand hit ratio only, excluding prefetch noise.

Decision: If this is high and you’re still slow, your bottleneck is probably not read caching (think sync writes, CPU, locks, network).

Task 5: Verify whether you’re swapping (ARC’s favorite way to ruin your day)

cr0x@server:~$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1  10488  51200  18000 7230000   12   45   620   130 8200 9100 10  5 70 15  0
 1  1  10520  48900  17920 7215000    0   32  1110   140 7600 8800  9  4 68 19  0

Meaning: Non-zero si/so indicates swap in/out; wa is I/O wait. If you’re swapping, ARC is competing with everything.

Decision: Stop performance tuning and fix memory pressure. Reduce ARC, stop the memory hog, or scale up. Swapping plus ZFS is a bad sitcom.

Task 6: Check ZFS pool I/O and latency (are disks the real culprit?)

cr0x@server:~$ zpool iostat -v 1 3
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
tank        9.12T  3.45T    640    210  82.1M  21.3M
  raidz2    9.12T  3.45T    640    210  82.1M  21.3M
    sda         -      -    110     36  13.8M   3.2M
    sdb         -      -    108     35  13.7M   3.1M
    sdc         -      -    107     34  13.6M   3.0M
    sdd         -      -    109     35  13.7M   3.1M

Meaning: If disks are pegged on read ops while ARC demand misses are high, you’re read I/O bound. If disks are calm but app is slow, ARC isn’t the main story.

Decision: If read ops saturate and ARC misses are high: add RAM, add vdevs, change workload shape, or reconsider storage tiering. If write ops dominate: look at sync writes and ZIL/SLOG.

Task 7: Check per-process I/O (catch the “one weird job”)

cr0x@server:~$ sudo iotop -oPa
Total DISK READ: 115.62 M/s | Total DISK WRITE: 22.10 M/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN  IO>    COMMAND
18234 be/4   backup     98.12 M/s   1.40 M/s  0.00 % 92.00 %  /usr/bin/rsync -a --delete /data /backup
 9121 be/4   postgres   12.40 M/s  10.10 M/s  0.00 %  8.00 %  postgres: checkpointer

Meaning: A sequential scan or backup job can blow out ARC with read-ahead and create misses that hurt interactive workloads.

Decision: If a batch job is dominating reads, schedule it, throttle it, or isolate it to a dataset with different caching policy.

Task 8: Inspect dataset caching policy (primarycache/secondarycache)

cr0x@server:~$ zfs get -o name,property,value primarycache,secondarycache tank/vmstore
NAME         PROPERTY        VALUE
tank/vmstore primarycache    all
tank/vmstore secondarycache  all

Meaning: primarycache controls what goes into ARC; secondarycache controls what goes into L2ARC.

Decision: For VM images with heavy streaming behavior, consider primarycache=metadata to protect ARC for metadata while letting guest reads hit disk predictably.

Task 9: Change primarycache for a dataset (surgical, reversible)

cr0x@server:~$ sudo zfs set primarycache=metadata tank/backup
cr0x@server:~$ zfs get -o name,property,value primarycache tank/backup
NAME        PROPERTY      VALUE
tank/backup primarycache  metadata

Meaning: You’re telling ZFS to cache metadata but not file data for that dataset.

Decision: Use this when a dataset is mostly sequential reads/writes and you want to stop it from evicting useful ARC contents. If latency improves for other workloads, keep it. If it hurts the dataset’s own performance, back it out.

Task 10: Check recordsize (workload fit matters more than people admit)

cr0x@server:~$ zfs get -o name,property,value recordsize tank/postgres
NAME          PROPERTY    VALUE
tank/postgres recordsize  128K

Meaning: Recordsize affects I/O amplification and cache efficiency. Databases often prefer smaller blocks (e.g., 16K) to avoid reading 128K to satisfy an 8K read.

Decision: If you see lots of small random reads and poor ARC effectiveness, consider adjusting recordsize for that dataset (with care; it affects new writes, not old ones).

Task 11: Observe prefetch behavior via ARC counters

cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(prefetch_data_hits|prefetch_data_misses|demand_data_hits|demand_data_misses)'
demand_data_hits                4 168331120
demand_data_misses              4 12194410
prefetch_data_hits              4 23311610
prefetch_data_misses            4 4981044

Meaning: If prefetch misses climb rapidly while demand misses also climb, prefetch may be evicting useful data. If prefetch misses climb while demand misses stay flat, it may just be wasted bandwidth.

Decision: If prefetch looks harmful, isolate the offending dataset and consider caching policy or workload scheduling. Don’t knee-jerk disable system-wide prefetch.

Task 12: Check ARC sizing parameters (Linux OpenZFS)

cr0x@server:~$ sudo cat /sys/module/zfs/parameters/zfs_arc_max
68719476736
cr0x@server:~$ sudo cat /sys/module/zfs/parameters/zfs_arc_min
4294967296

Meaning: ARC max/min in bytes. If max is too high relative to system RAM and workload, you can trigger memory pressure and swap.

Decision: If the host runs apps that need RAM (databases, JVMs, VM hypervisors), cap ARC intentionally. If the host is a storage appliance, allow ARC to use most RAM but still leave headroom for kernel and bursts.

Task 13: Temporarily cap ARC max (careful, but effective)

cr0x@server:~$ echo $((32*1024*1024*1024)) | sudo tee /sys/module/zfs/parameters/zfs_arc_max
34359738368
cr0x@server:~$ sudo cat /sys/module/zfs/parameters/zfs_arc_max
34359738368

Meaning: You reduced ARC’s maximum to 32 GiB. ARC will gradually adapt; it won’t instantly shrink to the new limit.

Decision: Use this when memory pressure is causing swap or OOM risk. After applying, monitor swapping and ARC miss behavior. If misses spike and disks saturate, you may have traded one problem for another; reconsider architecture (more RAM, split roles, add vdevs).

Task 14: Validate whether L2ARC is even being used

cr0x@server:~$ cat /proc/spl/kstat/zfs/arcstats | egrep '^(l2_hits|l2_misses|l2_size|l2_asize)'
l2_hits                         4 892331
l2_misses                       4 3211044
l2_size                         4 17179869184
l2_asize                        4 21474836480

Meaning: L2ARC hits/misses show whether your “cache SSD” is actually serving reads. l2_size is data stored; l2_asize is allocated.

Decision: If l2_hits barely moves during load, L2ARC isn’t helping. Common reasons: working set already fits in ARC, dataset isn’t eligible for secondarycache, or the workload is write-heavy or streaming.

Task 15: Correlate ARC misses with real latency at the block device

cr0x@server:~$ iostat -x 1 3
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.12    0.00    4.01   18.55    0.00   67.32

Device            r/s     w/s   rkB/s   wkB/s  aqu-sz  await  svctm  %util
sda             110.0    36.0  14120   3300     9.20   68.2    2.4   35.2
sdb             108.0    35.0  14080   3200     9.10   66.7    2.3   34.8

Meaning: await is the average time requests spend (queue + service). High await with high ARC demand misses is a classic “cache isn’t covering, disks are queued” picture.

Decision: If await climbs when demand misses climb, focus on read working set, vdev layout, or adding IOPS. If await is fine but app latency is high, investigate locks, CPU, network, or sync-write behavior.

Task 16: Verify sync write behavior (because not everything is ARC)

cr0x@server:~$ zfs get -o name,property,value sync logbias tank/postgres
NAME          PROPERTY  VALUE
tank/postgres sync      standard
tank/postgres logbias   latency

Meaning: If your issue is “writes are slow,” ARC isn’t your hero. sync and logbias influence synchronous write handling and ZIL behavior.

Decision: For write-latency incidents, check whether the workload is sync-heavy and whether you have an appropriate SLOG device. Don’t paper over it with cache myths.

Three corporate mini-stories (pain included)

Incident caused by a wrong assumption: “More ARC means faster databases”

A mid-sized company ran a PostgreSQL cluster on ZFS with a nice amount of RAM. The storage team assumed “ARC should use as much memory as possible.”
They raised zfs_arc_max and celebrated a larger cache number in dashboards. A week later, during month-end reporting, latency spiked hard.

The database team blamed storage. Storage blamed queries. Everyone blamed “network” out of habit. The graphs showed ARC hit ratio looking healthy.
That was the wrong metric to worship.

When we watched arcstat during the incident, demand misses weren’t catastrophic—but memory pressure was. Available memory dropped,
the kernel reclaimed aggressively, and swap activity started. The database had shared buffers sized for a reason, but ARC had muscled in and forced
the OS to make ugly choices. The “bigger ARC” decision didn’t create more cache hits; it created less stable memory behavior.

Fix was boring: cap ARC to leave headroom for the database and kernel, then tune database memory with intent. Demand misses rose slightly,
but latency improved because the system stopped swapping. The big lesson: ARC is not “free performance.” It’s a tenant in your RAM apartment,
and you still need to pay rent to other tenants.

An optimization that backfired: “Let’s add L2ARC SSDs everywhere”

Another shop standardized on “two NVMe drives for L2ARC” in every storage host because someone read that “SSD cache makes ZFS fast.”
They rolled it out as a hardware template. It looked modern. Procurement was happy. People love buying shiny things.

Then the weirdness: some hosts got slower under mixed load. Not “a little.” Noticeably. The batch processing cluster would start fine, then degrade.
The dashboards showed memory use creeping, and the CPU had extra overhead in the kernel. Storage latency didn’t always improve.

We sampled arcstat and checked L2ARC counters. L2ARC hits were low. ARC was already serving most of the hot set in RAM.
Meanwhile, the system paid overhead to feed and manage L2ARC, and it used RAM for L2ARC headers. Under pressure, ARC shrank,
and the remaining cache became less effective. Adding “more cache” reduced effective cache quality.

The fix was again boring: remove L2ARC from nodes where it didn’t help and reserve it for specific workloads with a proven miss profile.
We also stopped treating L2ARC as a default and started treating it as an optimization you justify with measurements.
The backfire wasn’t that L2ARC is “bad.” It was that cargo-cult caching is expensive.

A boring but correct practice that saved the day: “Measure during the incident, not after”

A large internal Git service started timing out during a release week. The symptom was “random slowness”: sometimes clones hung,
sometimes web UI stalled, sometimes everything looked fine. The on-call did something rare and correct: they gathered live evidence
while users were shouting.

They ran arcstat, zpool iostat, and iotop simultaneously in a tmux session and captured five minutes of output.
The pattern was clear. Demand misses spiked in sync with a specific maintenance job walking huge directory trees.
Disks queued up, metadata misses rose, and latency followed. Nothing magical—just a workload collision.

Because they had data, not vibes, the mitigation was clean: move the job window, isolate the dataset caching policy, and keep ARC
biased toward metadata for the repo dataset. The incident ended without hardware changes, without blame, and without a two-week “performance project.”

The practice wasn’t glamorous. It was simply: collect the right live stats early, then decide with evidence. This is how you avoid
the postmortem line “we were unable to reproduce.”

Common mistakes: symptom → root cause → fix

1) “ARC hit ratio is high, but reads are still slow”

Symptom: Great hit%, user latency still bad.

Root cause: Hits are mostly metadata or cheap reads; the misses on the critical path are few but painful (small random reads, hot DB pages). Or the bottleneck is writes/CPU/network.

Fix: Use demand-only stats and correlate with disk latency (iostat -x) and pool I/O (zpool iostat). If disks aren’t the limiter, pivot: sync writes, CPU saturation, app-level contention.

2) “ARC keeps shrinking and growing; performance is jittery”

Symptom: arcsz swings, latency oscillates, sometimes swap.

Root cause: Memory pressure. ARC is being reclaimed; the host is doing too much (VMs + database + ARC) without caps.

Fix: Cap ARC (zfs_arc_max), reduce competing memory usage, or split roles across hosts. Verify swap is zero-ish under normal load.

3) “Prefetch misses are huge; ARC feels useless”

Symptom: pmis dominates, demand hit rate drops during scans.

Root cause: Streaming workload (backups, scrubs, scans) and prefetch pulling data that won’t be reused; cache pollution.

Fix: Isolate that workload (schedule/throttle), set primarycache=metadata on scan-heavy datasets, and confirm improvement in demand misses for latency-sensitive datasets.

4) “We added L2ARC and nothing changed”

Symptom: Cache SSD installed, but l2_hits barely increases; latency unchanged.

Root cause: Working set already fits in ARC; dataset not eligible for secondarycache; or workload is write-heavy and not read-miss bound.

Fix: Prove need first: sustained demand misses plus disk read saturation. Then ensure secondarycache=all where appropriate and size RAM accordingly for L2ARC metadata overhead.

5) “After reducing ARC, system got slower”

Symptom: You cap ARC and read latency increases.

Root cause: The workload truly needs read cache; ARC was hiding slow disks. Your cap removed the bandaid.

Fix: Decide intentionally: add RAM, add vdevs/IOPS, or change workload. If the host must run apps and storage together, plan capacity so both can breathe.

6) “Random read workload on RAIDZ feels terrible”

Symptom: Low ARC hit, high read IOPS demand, queueing, poor latency.

Root cause: RAIDZ vdevs are not IOPS machines; small random reads punish them. ARC can’t fix a mismatch between workload and vdev layout.

Fix: Add mirrors for IOPS, add more vdevs, or move the workload to a pool designed for random I/O. Use ARC to help, not to compensate for architecture.

7) “We tuned recordsize and now cache looks worse”

Symptom: After changing recordsize, hit rate shifts unexpectedly; performance mixed.

Root cause: Recordsize affects new writes only; old data remains at old block sizes. You now have a mixed on-disk format and confusing stats.

Fix: Evaluate on data written after the change, or rewrite data (where appropriate). Avoid recordsize changes without a migration plan and clear workload target.

Checklists / step-by-step plan

Checklist A: “Is ARC helping right now?” (5 minutes)

1. Start arcstat 1 and watch demand misses and ARC size stability.
2. Run zpool iostat -v 1 to see whether vdevs are saturated.
3. Run iostat -x 1 to check await and queue depth.
4. If demand misses are low and disks are calm: stop blaming cache.
5. If demand misses are high and disks are queued: cache is not covering; plan RAM/vdev/workload changes.

Checklist B: “Is ARC causing trouble?” (memory pressure triage)

1. Check vmstat 1 for swapping (si/so).
2. Check arcstat -f arcsz,c,avail 1 for shrinking ARC and low available memory.
3. If swapping or avail is persistently low: cap ARC and/or reduce application memory pressure.
4. After changes, confirm swap stays quiet and demand misses didn’t explode.

Checklist C: “Should we add L2ARC?” (don’t buy hardware on vibes)

1. Prove sustained demand misses during real workload windows.
2. Prove disks are read-saturated (ops and latency) when those misses occur.
3. Confirm the workload has re-reads (not a one-pass stream).
4. Confirm you have RAM headroom for L2ARC overhead and indexing.
5. Deploy to one host, measure l2_hits growth and latency impact, then scale.

Checklist D: “Make dataset-level changes safely”

1. Identify the dataset causing pollution (from iotop + timing + arcstat behavior).
2. Adjust primarycache or recordsize only on that dataset, not globally.
3. Document baseline: five minutes of arcstat, zpool iostat, iostat -x.
4. Apply change, re-measure during the same workload window.
5. If worse, revert immediately. Keep your ego out of it.

FAQ

1) What is the single arcstat number I should watch?

Demand misses during the problem window. Not overall hit%. If demand misses rise and disk latency rises with them, cache isn’t covering.

2) Is a 90–95% ARC hit ratio “good”?

Maybe. It depends on what’s hitting. A high hit ratio with poor latency often means the misses are on the critical path, or the issue isn’t read caching at all.

3) Why are prefetch misses so high, and should I panic?

Prefetch misses can be high on sequential workloads. Panic only if prefetch activity correlates with eviction of useful ARC data and user-facing latency.

4) Does ARC cache writes?

ARC is primarily a read cache. Writes go through transaction groups and can be buffered in memory, but write latency complaints are often about sync writes and ZIL/SLOG, not ARC.

5) If I add RAM, will performance always improve?

If you’re read-miss bound and your working set is cacheable, yes—often dramatically. If you’re limited by sync writes, CPU, or network, RAM won’t save you.

6) When does L2ARC actually help?

When you have a read-heavy workload with a working set larger than RAM but with significant re-reads, and your disks are the bottleneck. L2ARC is not a universal accelerator.

7) Why did my system get slower after adding L2ARC?

Because L2ARC consumes RAM for metadata and adds work to manage cache. If ARC was already sufficient, you paid overhead without benefit and possibly increased memory pressure.

8) Should I set primarycache=metadata for VM images?

Often a good move when VMs do lots of streaming reads that churn cache. Test it on a subset first; some VM workloads benefit from caching data.

9) Does compression improve ARC effectiveness?

Frequently, yes, because cached blocks may be stored compressed, effectively fitting more logical data into the same RAM. But compression also costs CPU—measure both sides.

10) Why does arcstat look different across systems?

Different distributions and scripts print different columns and names. The underlying concepts—hits, misses, ARC size, demand vs prefetch—are what you should anchor to.

Conclusion: next steps that actually help

If you take one habit from this: run arcstat during the incident, not after. Pair it with zpool iostat and iostat -x.
In ten minutes you can usually answer the only question that matters: “Are we blocked on disk reads that cache could have served?”

Practical next steps:

1. Capture a five-minute baseline during normal load: arcstat + zpool iostat + iostat -x.
2. During the next slowdown, capture the same trio and compare demand misses and disk latency.
3. If memory pressure exists, cap ARC intentionally and eliminate swapping first.
4. If demand misses are the limiter, choose one: add RAM, redesign vdev layout for IOPS, or change dataset/workload caching behavior.
5. If ARC looks healthy, stop tuning cache and go find the real bottleneck (sync writes, CPU, network, locks).

Cache is not a performance strategy. It’s a performance multiplier—use it when the math works, ignore it when it doesn’t, and measure like you mean it.