Everything is slow, the graphs are red, and someone has already volunteered the usual fix: “Just add more CPU.” Another person says “It’s storage.” A third insists “It’s the network.” This is how teams burn days, money, and credibility: by solving the loudest theory instead of the actual limit.
The cure is boring and reliable: measure saturation, latency, throughput, and queues—at the same time—on the critical path. Then make a decision that matches the physics. Not the vibes.
What a bottleneck is (and what it isn’t)
A bottleneck is the resource that caps your system’s useful work for a given workload. It is not “the thing with the biggest graph.” It is not “the component you upgraded last quarter.” It is not “whatever is at 90%.”
Bottlenecks are workload-specific. “Storage is the bottleneck” is not a diagnosis. It’s a category. The diagnosis is: “Random 4k reads at QD=1 are limited by NVMe read latency; we’re hitting p99 of 3.2ms and queueing in the app thread pool.” That’s actionable. That’s a knife, not a foghorn.
Three definitions you should tattoo on your runbook
- Capacity: how much work a resource can do per unit time (e.g., requests/sec, MB/sec, IOPS).
- Saturation: how close you are to that capacity under a workload (and whether queues are forming).
- Tail latency: what your worst-affected users see. P95/P99 is where your pager lives.
Also: bottlenecks move. Fix CPU and you expose lock contention. Fix locks and you expose storage. Fix storage and you expose network. Congratulations: your system is now fast enough to find a new way to be slow.
One quote to keep you honest. Gene Kim’s “paraphrased idea”: Improving flow means finding constraints and elevating them; optimizing non-constraints just creates local speed and global pain.
— Gene Kim (paraphrased idea)
Joke #1 (short, relevant): A “quick performance fix” is like a “temporary firewall rule.” It will be there at your retirement party.
The four signals that matter: saturation, latency, throughput, errors
If you want fewer arguments and faster incident calls, standardize on a measurement vocabulary. I use four signals because they force clarity:
Saturation: “Are we at the limit?”
Saturation is about queues and contention. CPU run queues. Disk request queues. NIC transmit queues. Database connection pools. Kubernetes pod scheduling. If queues grow, you’re saturated or misconfigured.
Latency: “How long does one unit of work take?”
Measure end-to-end latency and component latency. Average latency is polite fiction; use percentiles. Tail latency is where bottlenecks show up first because queues amplify variance.
Throughput: “How much useful work do we complete?”
Throughput is not “bytes moved” if the business cares about “orders placed.” Ideally you measure both: user-level throughput and system throughput. A system can move a lot of bytes while completing fewer transactions. That’s how retries and thundering herds pay for your cloud provider’s new yacht.
Errors: “Are we failing fast or failing slow?”
When saturated, systems often fail slowly before they fail loudly. Timeouts, retries, checksum errors, TCP retransmits, IO errors, context deadline exceeded—those are bottleneck smoke. Treat errors as first-class performance signals.
What “the real limit” means
The real limit is the point where increasing offered load does not increase completed work, while latency and queueing explode. That’s a measurable knee. Your goal is to identify that knee for the workload that matters, not for a synthetic benchmark that flatters your purchase order.
Queues: where performance goes to die quietly
Most production bottlenecks are queueing problems disguised as “slow.” The machine isn’t necessarily slow; it’s waiting its turn behind other work. That’s why “resource utilization” can look fine while the system is miserable.
Little’s Law, but make it operational
Little’s Law: L = λW (average number in the system = arrival rate × time in system). You don’t need a math degree to use it. If latency (W) goes up and arrival rate (λ) stays similar, the number of things waiting (L) must be going up. That’s queueing. Find the queue.
Key queue types you’ll meet at 2 a.m.
- CPU run queue: threads ready to run but not scheduled (watch load and run queue length).
- Block I/O queue: requests waiting for disks/NVMe (watch avgqu-sz, await, device utilization).
- Mutex/lock queues: threads waiting on a lock (perf, eBPF, or app metrics).
- Connection pools: DB/HTTP pools that cap concurrency (watch pool wait time).
- Kernel network queues: drops in qdisc, ring buffer overruns (watch drops, retransmits).
- GC and allocator stalls: queueing inside runtimes (JVM, Go, Python).
Queues are not always bad. They become bad when they are uncontrolled, hidden, or coupled to retries. Retries are queue multipliers. A small latency bump becomes a self-inflicted DDoS.
Fast diagnosis playbook (first/second/third)
This is the “walk into a burning room without becoming the fire” sequence. It’s designed for on-call reality: you have partial observability, angry stakeholders, and exactly one shot at not making it worse.
First: confirm the symptom in user terms
- What is slow? A page load? An API endpoint? A batch job? A DB query?
- Is it latency, throughput, or both?
- Is it global or isolated (one AZ, one node pool, one tenant)?
- What changed recently (deploys, config, traffic shape, data growth)?
Second: find the queue closest to the pain
Start at the user-facing service and walk down the stack:
- App metrics: request latency percentiles, concurrency, timeouts, retry rate.
- Thread/worker pools: queue length, saturation, rejected work.
- DB: connection pool waits, slow queries, lock waits.
- OS: CPU run queue, iowait, memory pressure, context switches.
- Storage: device latency/queue depth, filesystem stalls, ZFS txg sync, dirty page writeback.
- Network: retransmits, drops, interface saturation, DNS latency.
Third: measure saturation and headroom, then stop guessing
Pick 2–3 candidate constraints and gather hard evidence within 10 minutes:
- CPU: run queue, steal time, throttling, per-core hotspots.
- Memory: major faults, swap, reclaim, OOM risk.
- Disk/NVMe: await, svctm (careful), avgqu-sz, util, discard/writeback stalls.
- Network: retransmits, drops, qdisc, NIC errors, RTT.
If you can’t explain the slowdown with one of those categories, look for coordination bottlenecks: locks, leader elections, centralized services, quotas, API rate limits, or a single hot shard.
Practical tasks: commands, outputs, decisions (12+)
These tasks are meant to be run on a Linux host in the hot path. Each one includes: a command, what the output means, and what decision you make. Run them during an incident, but also in calm times so you know what “normal” looks like.
Task 1: Identify whether CPU is actually the limit (run queue + iowait)
cr0x@server:~$ uptime
14:02:11 up 37 days, 3:19, 2 users, load average: 18.24, 17.90, 16.88
Meaning: Load average counts runnable tasks and uninterruptible sleep (often I/O wait). “18” on a 32-core box might be fine, or terrible, depending on what those tasks are doing.
Decision: Don’t declare CPU bottleneck from load alone. Follow with vmstat and per-core view.
cr0x@server:~$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
12 7 0 82124 51200 842120 0 0 120 9820 820 2400 22 8 41 29 0
15 8 0 81560 51200 841980 0 0 140 10120 900 2600 21 9 39 31 0
Meaning: r is runnable tasks; b is blocked (often on I/O). High wa with high b points to I/O stalls, not CPU shortage.
Decision: If b and wa are high, shift attention to storage/network I/O before scaling CPU.
Task 2: Check per-core saturation and steal/throttling (virtualization pain)
cr0x@server:~$ mpstat -P ALL 1 3
Linux 6.1.0 (server) 01/10/2026 _x86_64_ (32 CPU)
13:58:01 CPU %usr %sys %iowait %irq %soft %steal %idle
13:58:02 all 24.10 10.20 18.50 0.00 0.90 0.00 46.30
13:58:02 7 89.00 9.00 0.00 0.00 0.00 0.00 2.00
13:58:02 12 10.00 5.00 70.00 0.00 1.00 0.00 14.00
Meaning: One CPU pegged can indicate single-threaded bottleneck or IRQ imbalance; high %steal indicates hypervisor contention; high %iowait on specific CPUs can correlate with IRQ affinity and device queues.
Decision: If a few cores are pinned, investigate single-threaded hotspots, interrupt distribution, or affinity settings before adding more cores.
Task 3: Identify top waiters: CPU vs I/O vs locks (high-level triage)
cr0x@server:~$ top -b -n 1 | head -n 20
top - 13:59:41 up 37 days, 3:17, 2 users, load average: 18.24, 17.90, 16.88
Tasks: 412 total, 11 running, 401 sleeping, 0 stopped, 0 zombie
%Cpu(s): 22.4 us, 8.9 sy, 0.0 ni, 46.1 id, 22.1 wa, 0.0 hi, 0.5 si, 0.0 st
MiB Mem : 64000.0 total, 1100.2 free, 4200.1 used, 58700.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 59600.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23120 app 20 0 4120.5m 520.2m 152.0m D 35.0 0.8 12:20.11 java
Meaning: Process in state D is uninterruptible sleep, commonly waiting on I/O. High system-wide wa reinforces I/O as a candidate bottleneck.
Decision: Follow the “D” processes to see what they’re waiting on (block devices, NFS, etc.).
Task 4: Map a stuck process to its open files and the backing filesystem
cr0x@server:~$ lsof -p 23120 | head
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 23120 app cwd DIR 253,0 4096 2 /srv/app
java 23120 app 12u REG 253,0 1048576 98342 /srv/app/data/index.bin
Meaning: Shows which paths the process is actively using. You’re trying to connect “app is slow” to “this filesystem/device is slow.”
Decision: Identify the mount and device that backs the hot path; focus I/O tooling there.
cr0x@server:~$ findmnt -T /srv/app/data/index.bin
TARGET SOURCE FSTYPE OPTIONS
/srv /dev/nvme0n1p2 ext4 rw,relatime
Decision: Now you know which device to inspect (nvme0n1 here).
Task 5: Device-level latency and queueing (the truth serum)
cr0x@server:~$ iostat -x 1 3
Linux 6.1.0 (server) 01/10/2026 _x86_64_ (32 CPU)
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz %util
nvme0n1 820.0 2100.0 3280.0 26800.0 0.0 0.0 0.00 0.00 1.2 18.9 42.10 98.00
Meaning: High %util with high aqu-sz means the device is saturated and requests are queueing. Write await is high, so writes are waiting a long time.
Decision: Stop pretending CPU will fix it. Either reduce write pressure, change I/O pattern, add devices, or move workload. Also check if the latency is coming from the device or from filesystem flush behavior.
Task 6: NVMe-specific health and error counters
cr0x@server:~$ sudo nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : 41 C
available_spare : 100%
percentage_used : 7%
media_errors : 0
num_err_log_entries : 0
Meaning: If you see media errors or a high percentage used, performance can degrade, and your “bottleneck” becomes “failing drive.”
Decision: If errors show up, switch from performance mode to reliability mode: plan replacement and reduce load.
Task 7: Filesystem and kernel writeback pressure (dirty pages can be your queue)
cr0x@server:~$ cat /proc/meminfo | egrep 'Dirty|Writeback|MemFree|MemAvailable'
MemFree: 80124 kB
MemAvailable: 61045232 kB
Dirty: 1248204 kB
Writeback: 92160 kB
WritebackTmp: 0 kB
Meaning: Large and persistent Dirty can mean the system is buffering writes and later flushing them in painful bursts, creating latency spikes.
Decision: If dirty data grows without draining, investigate writeback throttling, filesystem journaling, and application flush patterns.
Task 8: Detect memory pressure (the sneaky bottleneck that impersonates CPU)
cr0x@server:~$ vmstat 1 5 | tail -n +3
6 0 0 10240 45000 1200000 0 0 0 2000 1100 3800 30 12 48 10 0
5 0 0 8120 45000 1180000 0 0 0 2200 1200 4200 29 11 44 16 0
Meaning: Watch for si/so (swap in/out) and consistently low free memory with high cs (context switches). Even without swap, reclaim storms can spike latency.
Decision: If reclaim is heavy (check sar -B or PSI), reduce memory footprint or increase RAM; don’t chase phantom CPU issues.
Task 9: Pressure Stall Information (PSI): prove time lost to contention
cr0x@server:~$ cat /proc/pressure/io
some avg10=12.34 avg60=10.21 avg300=8.90 total=98324212
full avg10=4.12 avg60=3.20 avg300=2.75 total=31244211
Meaning: PSI quantifies how much time tasks are stalled due to I/O pressure. full means no task could make progress due to I/O at times—this is real pain, not theoretical.
Decision: If PSI is high, prioritize reducing I/O contention (batching, caching, queue limits) over micro-optimizing CPU.
Task 10: Network: find retransmits and drops (latency that looks like “app slowness”)
cr0x@server:~$ ss -s
Total: 1252 (kernel 0)
TCP: 1023 (estab 812, closed 141, orphaned 0, timewait 141)
Transport Total IP IPv6
RAW 0 0 0
UDP 12 9 3
TCP 882 740 142
INET 894 749 145
FRAG 0 0 0
Meaning: High connection churn can indicate retries/timeouts. This output is just the appetizer.
cr0x@server:~$ netstat -s | egrep 'retransmit|segments retransmited|packet receive errors|dropped'
18432 segments retransmited
92 packet receive errors
1184 packets received, dropped
Meaning: Retransmits and drops are throughput killers and tail-latency factories.
Decision: If retransmits rise during slowness, treat the network path as a candidate bottleneck: check NIC stats, queues, and upstream devices.
Task 11: NIC-level counters (prove it’s not “the app”)
cr0x@server:~$ ip -s link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 0a:1b:2c:3d:4e:5f brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
9823412231 9123421 0 812 0 0
TX: bytes packets errors dropped carrier collsns
11233412211 10233421 0 221 0 0
Meaning: Drops at the host are unambiguous. They may be due to ring buffer overruns, qdisc issues, or just too much traffic.
Decision: If drops climb, reduce burstiness (pacing), increase buffers carefully, or fix upstream congestion; don’t just “add threads.”
Task 12: Check filesystem latency spikes from sync-heavy workloads (ext4 example)
cr0x@server:~$ sudo dmesg -T | egrep -i 'blocked for more than|I/O error|nvme|EXT4-fs' | tail -n 10
[Fri Jan 10 13:55:12 2026] INFO: task java:23120 blocked for more than 120 seconds.
[Fri Jan 10 13:55:12 2026] EXT4-fs (nvme0n1p2): Delayed block allocation failed for inode 98342 at logical offset 152343 with max blocks 4
Meaning: Kernel tells you processes are blocked and sometimes why. This can implicate allocation pressure, writeback, or device stalls.
Decision: Treat “blocked for more than” as a serious I/O stall. Stop rolling restarts; find the I/O cause.
Task 13: ZFS: spot transaction group sync pressure (common “why are writes spiky?” culprit)
cr0x@server:~$ sudo zpool iostat -v 1 3
capacity operations bandwidth
pool alloc free read write read write
tank 1.20T 2.40T 820 2400 3.2M 26.8M
mirror 1.20T 2.40T 820 2400 3.2M 26.8M
nvme0n1 - - 410 1200 1.6M 13.4M
nvme1n1 - - 410 1200 1.6M 13.4M
Meaning: Good for bandwidth and ops, but not latency by itself. If writes look fine but app latency is awful, you may be stuck on sync writes, slog contention, or txg sync bursts.
Decision: If you suspect sync behavior, check the workload and dataset properties next.
cr0x@server:~$ sudo zfs get sync,logbias,compression,recordsize tank/app
NAME PROPERTY VALUE SOURCE
tank/app sync standard local
tank/app logbias latency local
tank/app compression lz4 local
tank/app recordsize 128K local
Decision: If the app does many small sync writes (databases, journaling), validate SLOG design and whether sync semantics are required. Don’t flip sync=disabled in production unless you enjoy explaining data loss to legal.
Task 14: Benchmark safely with fio (don’t lie to yourself)
cr0x@server:~$ sudo fio --name=randread --filename=/srv/app/.fiotest --size=2G --rw=randread --bs=4k --iodepth=1 --numjobs=1 --direct=1 --time_based --runtime=30 --group_reporting
randread: (groupid=0, jobs=1): err= 0: pid=31022: Fri Jan 10 14:01:02 2026
read: IOPS=18.2k, BW=71.0MiB/s (74.4MB/s)(2131MiB/30001msec)
lat (usec): min=55, max=3280, avg=68.11, stdev=22.50
clat percentiles (usec):
| 1.00th=[ 58], 50.00th=[ 66], 95.00th=[ 82], 99.00th=[ 120]
Meaning: This measures 4k random read latency at QD=1, which often matches real application patterns better than “big sequential read.” Percentiles show tail behavior.
Decision: If your app is latency-sensitive and fio shows p99 climbing under load, you’ve identified a storage latency ceiling. Plan mitigation (cache, sharding, faster media, better batching) instead of tuning unrelated layers.
Task 15: Identify block layer queue depth and scheduler
cr0x@server:~$ cat /sys/block/nvme0n1/queue/nr_requests
1024
cr0x@server:~$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq
Meaning: nr_requests influences how deep queues can get. Scheduler choice can affect tail latency, fairness, and throughput.
Decision: If tail latency is the pain, consider a scheduler that behaves better under contention and reduce queueing where appropriate. Don’t cargo-cult; test with your workload.
Task 16: Spot throttling (containers and cgroups: the invisible bottleneck)
cr0x@server:~$ cat /sys/fs/cgroup/cpu.stat
usage_usec 92833423211
user_usec 81223311200
system_usec 11610112011
nr_periods 124112
nr_throttled 32110
throttled_usec 8823122210
Meaning: If nr_throttled and throttled_usec climb, your workload is being CPU-throttled by cgroups. It looks like “we need more CPU” but it’s actually “we set a limit.”
Decision: Adjust CPU limits/requests, reduce burstiness, or move the workload. Don’t upgrade hardware to compensate for your own quota.
Joke #2 (short, relevant): If your bottleneck is “the database,” congratulations—you have discovered gravity.
Interesting facts and a little history (so you stop repeating old mistakes)
- Amdahl’s Law (1967) formalized why speeding up one part of a system has diminishing returns if the rest remains serial. It’s the math behind “we doubled CPU and got 12% faster.”
- Little’s Law (published 1961) became one of the most practical tools for reasoning about latency and concurrency in systems long before we had modern observability.
- The “utilization vs latency” curve was widely taught in queueing theory decades ago; modern “p99 went vertical” incidents are that curve, not a mystery.
- IOwait confusion is ancient: Linux counts tasks in uninterruptible sleep in load average. People still page each other about “high load” that’s really disk wait.
- Ethernet has been faster than many storage stacks for years; in practice, software overhead (kernel paths, TLS, serialization) often bottlenecks before line rate.
- NVMe improved parallelism dramatically by offering multiple queues and low overhead compared to SATA/AHCI, but it also made it easier to hide latency behind deep queueing—until tail latency bites you.
- “The Free Lunch Is Over” (mid-2000s) wasn’t just about CPUs; it forced software to confront parallel bottlenecks: locks, cache contention, and coordination costs.
- Write caches and barriers have a long history of trading durability for speed. Many catastrophic “performance wins” were just undetected data-loss settings.
- Tail latency became mainstream in large-scale web services because averages didn’t explain user pain. The industry’s shift to p95/p99 is one of the few cultural improvements we can measure.
Three corporate mini-stories from the trenches
Mini-story 1: The incident caused by a wrong assumption
The symptom was classic: API latency doubled, then tripled, then the error budget started screaming. The on-call crew saw high load averages on the app nodes and declared “CPU is pegged.” They scaled out the deployment and watched… nothing improve. More pods, same pain.
The wrong assumption was subtle and common: they treated load average as “CPU usage.” In reality, the nodes had plenty of idle CPU. The load was inflated by threads stuck in uninterruptible sleep. They were waiting on disk.
Root cause lived on a different layer: a change in the application’s logging strategy. A “harmless” update switched from buffered logging to synchronous writes for “audit correctness.” The filesystems were on network-backed volumes with decent throughput but mediocre sync latency. Under peak traffic, every request did at least one sync write. Latency went vertical because the bottleneck wasn’t bandwidth; it was fsync latency and queueing.
The fix wasn’t “more CPU.” It was moving audit logs to an async pipeline with durable buffering, and isolating log I/O from request threads. They also added a hard budget on sync operations per request path. The postmortem included a new rule: if you touch fsync, you must attach p99 latency measurements from production-like storage.
Mini-story 2: The optimization that backfired
A platform team wanted faster batch processing. They noticed their object storage ingest job was “underutilizing the network” and decided to crank concurrency: more workers, bigger buffers, deeper queues. Throughput went up in staging. People high-fived. The change rolled out on a Friday afternoon because of course it did.
In production, the job did achieve higher throughput—briefly. Then the rest of the fleet started suffering. API tail latency spiked across unrelated services. Database connections piled up. Retries erupted. The incident channel became a festival of screenshots.
What happened was a perfect example of optimizing a non-constraint until it became a constraint for everyone else. The batch job wasn’t just using more network; it was consuming shared resources: top-of-rack buffers, host NIC queues, and most importantly the storage backend’s request budget. Latency-sensitive services were now competing with a background job that had no backpressure and no fairness controls.
The final fix was not “limit the batch job to one thread.” It was implementing explicit scheduling and isolation: cgroup network shaping, separate node pools, and request rate limiting against the backend. They also introduced SLO-aware concurrency: the job backs off when p99 latency of customer traffic worsens. It’s not fancy. It’s grown-up.
Mini-story 3: The boring but correct practice that saved the day
A company running a mixed workload—databases, queues, and a search cluster—had a habit that looked uncool in architecture reviews: every quarter, they ran controlled load tests in production off-peak and recorded baseline “knee points” for key services. Same scripts. Same measurement windows. Same dashboards. No heroics.
One Tuesday, customer-facing latency climbed. Not a total outage, just a slow bleed: p99 went from “fine” to “people complain.” The first responder pulled up the baseline report from two weeks ago and noticed something: the system hit the throughput knee at 20% lower traffic than before. That meant capacity had effectively shrunk.
Because they had baselines, they didn’t waste time debating. They compared iostat and PSI against the baseline and saw a new pattern: higher write latency and deeper queues on a specific subset of nodes. Those nodes shared the same firmware revision on their NVMe drives—recently updated as part of “routine maintenance.” The update changed power management behavior and introduced latency spikes under sustained writes.
They rolled back the firmware on the affected fleet, and the knee point returned. No new architecture. No blame Olympics. Just a calm reversal backed by evidence. The postmortem’s action item was delightfully dull: pin firmware versions and add a latency regression check to maintenance procedures.
Common mistakes: symptom → root cause → fix
1) High load average → “We need more CPU” → you scale and nothing changes
Symptom: Load average high, users complain, CPU looks “busy” at a glance.
Root cause: Tasks are stuck in I/O wait (uninterruptible sleep). Load includes them.
Fix: Confirm with vmstat (b, wa), iostat -x, PSI I/O. Solve storage latency or reduce sync I/O in the request path.
2) Disk %util at 100% → “The disk is maxed” → you buy faster disks and still see latency spikes
Symptom: %util pinned, await high, queue depth large.
Root cause: The disk might be fine; the workload pattern is pathological (tiny sync writes, fsync storms, journaling contention), or the queue is too deep and amplifying tail latency.
Fix: Characterize I/O size, sync frequency, and concurrency. Introduce batching, move fsync off critical path, tune scheduler/queue depth, and validate with fio using the app’s I/O pattern.
3) “Network is fine, we have bandwidth” → tail latency still awful
Symptom: Interface Mbps below line rate, yet timeouts and long p99.
Root cause: Drops/retransmits, bufferbloat, or microbursts. Throughput graphs lie when packets are being resent.
Fix: Check netstat -s, ip -s link, qdisc stats. Apply pacing, fair queueing, or reduce burst concurrency.
4) “We optimized cache hit rate” → memory pressure quietly kills performance
Symptom: Higher cache usage, but latency spikes and CPU system time increases.
Root cause: Cache growth triggers reclaim, page faults, or compaction stalls; the system spends time managing memory, not serving requests.
Fix: Measure PSI memory, major faults, and reclaim. Cap caches, size them intentionally, and keep headroom for the kernel and working set.
5) “More threads = more throughput” → throughput stagnates while latency explodes
Symptom: Concurrency increased, p99 gets worse, CPU not fully used.
Root cause: You saturated a shared downstream (DB, disk, lock, or pool). More concurrency just deepens queues.
Fix: Find the limiting resource, add backpressure, and reduce concurrency to the knee point. Use load shedding rather than infinite waiting.
6) “We turned on compression” → CPU is fine but throughput drops
Symptom: Lower disk writes, but request throughput declines and tail latency rises.
Root cause: Compression shifts the bottleneck to CPU cache, memory bandwidth, or single-threaded compression stages; also changes I/O pattern.
Fix: Benchmark with representative data. Watch per-core hotspots. Use faster algorithms, adjust record/block sizes, or compress only cold paths.
7) “We disabled fsync for speed” → everything is fast until it isn’t
Symptom: Huge performance “improvement” after a config change; later, corruption or lost transactions after a crash.
Root cause: Durability semantics were traded for latency without a business decision.
Fix: Restore correct durability, design proper buffering/logging, and document durability requirements. If you must relax durability, do it explicitly and visibly.
Checklists / step-by-step plan
Step-by-step: establish the real limit (calm time)
- Pick one workload that matters (endpoint, job, query) and define success (p95/p99, throughput, error rate).
- Trace the critical path: client → service → downstreams (DB, cache, storage, network).
- Instrument queues: worker pool depth, DB pool wait, kernel PSI, disk queue depth.
- Run a controlled load ramp: increase offered load gradually; hold steady at each step.
- Find the knee: where throughput stops scaling and tail latency accelerates.
- Record baselines: knee point, p99, saturation metrics, and top contributors.
- Set guardrails: concurrency limits, timeouts, retry budgets, and backpressure.
- Re-test after changes: deploys, kernel/firmware updates, instance type changes, storage migration.
Incident checklist: avoid thrash and guesswork
- Stop the bleeding: if latency is exploding, cap concurrency or enable load shedding. Protect the system first.
- Confirm scope: which services, which nodes, which tenants, which regions.
- Check error signals: timeouts, retries, TCP retransmits, I/O errors.
- Check queues: app queue length, DB pool wait, run queue, disk queue depth, PSI.
- Pick one bottleneck hypothesis and collect 2–3 measurements that could falsify it.
- Change one thing at a time unless you’re in emergency containment.
- Write down the timeline while it’s happening. Future-you is an unreliable witness.
Decision rules (opinionated, because you’re busy)
- If p99 latency rises while throughput stays flat, you’re queueing. Find the queue, don’t add more work.
- If errors rise (timeouts/retries), treat performance as reliability. Reduce load; fix constraint second.
- If device latency rises but bandwidth is modest, suspect sync writes, writeback, or firmware/power management.
- If CPU is low but latency is high, suspect waiting: I/O, locks, pools, network retransmits, GC.
- If one core is pinned, stop scaling horizontally and find the serial section or contention point.
FAQ
1) How do I tell if I’m CPU-bound or I/O-bound quickly?
Use vmstat 1 and iostat -x 1. High wa and blocked tasks (b) point to I/O waits. High r with low wa suggests CPU saturation or runnable contention.
2) Is high disk %util always bad?
No. It can mean the device is busy (fine) or saturated with deep queues (bad). Look at aqu-sz and await. A busy disk with low await is healthy; a busy disk with high await is a bottleneck.
3) Why does adding threads make latency worse even if CPU is idle?
Because you’re likely saturating a downstream resource (DB connections, disk queue, lock) and creating longer queues. Idle CPU doesn’t mean headroom; it can mean you’re blocked elsewhere.
4) What’s the difference between throughput bottlenecks and tail-latency bottlenecks?
Throughput bottlenecks cap completed work; tail-latency bottlenecks ruin user experience first and may happen well before max throughput. Tail issues are often caused by queueing, garbage collection, jitter, or contention spikes.
5) Can caching “fix” a bottleneck permanently?
Caching can move the bottleneck by reducing work, but it creates new failure modes: cache stampedes, memory pressure, and consistency complexity. Treat cache as an engineering system with limits and backpressure, not a magic blanket.
6) How do I measure the “knee point” safely in production?
Ramp load gradually during a low-risk window, isolate test traffic if possible, and cap concurrency. Watch p95/p99, errors, and saturation metrics. Stop when errors or tail latency start accelerating.
7) Why do my benchmarks look great but production is slow?
Your benchmark probably measures the wrong thing: sequential I/O instead of random, QD=32 instead of QD=1, hot cache vs cold cache, or it ignores sync semantics. Also, production has contention and noisy neighbors. Benchmark the workload you actually run.
8) When should I scale up vs optimize?
Scale up when you have clear saturation of a resource that scales linearly with capacity (CPU for parallel compute, bandwidth for streaming). Optimize when the bottleneck is coordination, tail latency, or contention; hardware rarely fixes those.
9) What’s the fastest way to avoid bottleneck hysteria in a team?
Agree on a small set of standard measurements and a triage order. Make people bring evidence: “iostat shows 20ms write await and 40-deep queue” beats “it feels like disk.” Write it into the incident template.
Next steps you can actually do
If you want fewer performance fire drills, stop treating bottlenecks like folklore. Treat them like constraints you can measure.
- Pick one critical user journey and instrument latency percentiles, errors, and concurrency.
- Add queue metrics (app queues, pools, PSI, device queues). Bottlenecks hide in waiting, not working.
- Capture a baseline knee point under controlled load. Keep it in the runbook. Update it after meaningful changes.
- During incidents, run the fast diagnosis playbook and collect falsifiable evidence before changes.
- Implement backpressure (concurrency caps, retry budgets, timeouts). A system with backpressure fails predictably instead of theatrically.
The real limit is never a mystery. It’s just hiding behind your assumptions, waiting for you to measure it.