AI in Everything: When Labels Got Sillier Than Features

When labels outrun features

“AI” used to mean you had a model, you trained it, you deployed it, and you spent the next year learning what drift feels like in your on-call rotation. Now “AI” can mean any of these:

• A hard-coded rules engine with a new marketing sentence.
• A vendor API call where you do not control model behavior, data retention, or release cadence.
• A locally hosted model with a GPU bill and a queueing problem.
• A feature that used to be deterministic and now has a confidence score and a shrug.

Labels matter because labels change decision-making. Once something is tagged “AI,” teams accept failure modes they would never accept elsewhere: non-determinism, partial correctness, sudden regressions due to upstream changes, and output that can’t be easily explained to auditors or customers.

And “AI in everything” tends to create a specific organizational pathology: you stop asking what the feature does and start asking what the label buys you. That’s how you end up with AI-enabled toasters. Or, more commonly, AI-enabled ticket routing that costs more than the humans it replaced and routes worse when it matters.

Here’s the operational stance that keeps you employed: treat “AI” as an implementation detail, not a product category. Demand the same rigor you demand from any critical dependency: explicit SLOs, testable acceptance criteria, rollback plans, cost ceilings, and security controls.

One quote to keep in your pocket, because it never stops being true: “Hope is not a strategy.” — Rick Page.

Why the label is dangerous

It’s dangerous because it invites hand-waving. The moment someone says “the model will figure it out,” you should hear: “we haven’t specified requirements, and we’re outsourcing our thinking to probability.” That might be fine for autocomplete. It’s not fine for a billing pipeline.

Most “AI features” fail for boring reasons:

• They are slower than the existing flow, so users abandon them.
• They are flaky under load, so support teams learn to disable them.
• They are expensive at scale, so finance becomes your outage trigger.
• They are unsafe in edge cases, so legal makes you wrap them in disclaimers until they’re useless.

Joke #1: We called it “AI-powered” until the first incident, when it became “temporarily rule-based for stability.”

What “real feature” means in production

A real feature has:

• Inputs you can enumerate and validate.
• Outputs you can score, bound, and explain.
• Performance you can predict under load.
• Fallbacks that preserve the user journey.
• Telemetry that tells you whether it’s helping or hurting.

“AI” doesn’t remove those needs. It multiplies them. You now have a stochastic component whose failure modes can be subtle: it returns something, but not something useful, and it does so confidently. That’s not a 500. That’s silent data corruption in human language.

Facts and historical context that matter in ops

People treat the current “AI everywhere” moment as unprecedented. It isn’t. The technology is new-ish; the hype cycle is classic. A few concrete facts help you argue with confidence in rooms that prefer slogans:

1. The term “artificial intelligence” was coined in 1956 at the Dartmouth workshop. The branding was ambitious from day one, long before production systems existed to judge it.
2. AI has had multiple “winters” (notably in the 1970s and late 1980s/early 1990s) when funding collapsed after inflated promises didn’t deliver. Hype is cyclical; operational debt is permanent.
3. Expert systems dominated enterprise AI in the 1980s: rules, knowledge bases, and brittle maintenance. Today’s “prompt engineering as business logic” is eerily similar—just with more tokens and fewer guarantees.
4. Backpropagation’s modern resurgence in 1986 (Rumelhart, Hinton, Williams) is a reminder that breakthroughs can sit around until compute and data make them practical. Practicality is an ops constraint, not a philosophical one.
5. The 2012 ImageNet moment (deep learning beating prior vision methods) was as much about GPUs and datasets as algorithms. Your “AI feature” is probably a supply chain story: chips, bandwidth, storage, and vendor uptime.
6. Transformers (2017) made large language models scalable, but also made inference cost and latency a first-class product constraint. If you can’t afford p99, you don’t have a feature.
7. Model performance is not monotonic with time. Deployed models can get worse as user behavior changes, language shifts, seasonality hits, or upstream data pipelines drift. Traditional software rarely rots this way.
8. “AI” regulation is accelerating in multiple jurisdictions, which means provenance, explainability, and audit trails are not “enterprise extras” anymore; they are table stakes for risk-managed environments.

A decision framework: what to ship, what to ban, what to measure

Start with the only question that matters: what problem are we solving?

Not “how do we add AI.” The problem statement must be legible without the letters A and I. Example:

• Good: “Reduce mean time to resolution by summarizing incident timelines from logs and tickets.”
• Bad: “Add an AI assistant to the NOC.”

If you can’t write success criteria as a test, you’re about to ship a demo. Demos are fine. They belong in sandboxes with hard kill switches.

Define your feature class: deterministic, probabilistic, or advisory

Most production incidents happen because teams deploy a probabilistic system as if it were deterministic. Decide what you are building:

• Deterministic feature: the same input yields the same output. You can cache. You can reason about it. Great.
• Probabilistic feature: outputs vary, confidence matters, and correctness is statistical. Needs scoring, monitoring, and guardrails.
• Advisory feature: the system suggests; a human or a deterministic gate decides. This is where most “AI in everything” should live until proven safe.

Make it measurable: ship with an “outcome metric” and a “harm metric”

Outcome metrics answer “does this help?” Harm metrics answer “what breaks when it’s wrong?” You need both.

• Outcome examples: deflection rate (support), time-to-complete (workflow), conversion uplift (commerce), time saved per ticket (ops).
• Harm examples: incorrect refunds, misrouted high-severity tickets, policy violations, PII leakage, customer churn triggered by nonsense responses.

Then add constraints:

• Latency budget: p50/p95/p99 targets.
• Error budget: availability and correctness thresholds.
• Cost budget: cost per request and monthly ceiling with alerts.

Architect for failure, not vibes

“AI in everything” often means “new network dependency in everything.” That’s how you get cascading failures. Treat the AI component like a flaky upstream:

• Timeouts with sane defaults (shorter than you think).
• Circuit breakers to shed load.
• Fallback mode that preserves core functionality.
• Bulkheads: isolate the AI feature so it can’t starve the rest of the system.
• Replayable requests for offline evaluation and post-incident analysis.

Joke #2: Our “AI roadmap” was so aggressive we had to add a new tier to the incident taxonomy: “Sev-2, but philosophical.”

Don’t confuse “clever” with “reliable”

Teams love clever. Users love reliable. Executives love quarterly numbers. Your job is to deliver all three by putting reliability ahead of novelty. The easiest way: require every AI feature to have a non-AI fallback that meets minimum UX. If the fallback feels humiliating, good—now you know what “minimum” really means.

Three corporate mini-stories from the trenches

Mini-story #1: The incident caused by a wrong assumption

At a mid-sized B2B SaaS company, support leadership wanted “AI ticket triage.” The pitch was clean: ingest incoming emails, classify intent, route to the right queue, auto-tag priority, reduce response time. The vendor demo looked great. The integration went live behind a feature flag. Everyone congratulated themselves for being modern.

The wrong assumption was subtle: the team assumed the model’s classification was “like a rules engine, but smarter.” In other words, stable. They didn’t build explicit drift monitoring or robust fallbacks because the feature wasn’t “critical path”… until it quietly became critical path. Support agents stopped manually triaging because the new system was faster, and managers started measuring them assuming routing was correct.

Then a product launch changed customer language. People started using new terms that were close to old ones but not identical. The model began routing “billing dispute” into “feature request.” The first symptom wasn’t a 500; it was a customer who got an upbeat product roadmap response to a refund request. The customer escalated. Then a few more did.

Operations got pulled in after the fact. Logs showed the system was “healthy.” Latency fine. Error rate fine. The incident was correctness rot. Once they sampled misroutes, the failure was obvious, but it took days because there was no “ground truth” feedback loop and no baseline comparison. The fix wasn’t glamorous: force agents to confirm routing (advisory mode), store labeled outcomes, retrain and evaluate weekly, and add drift alarms based on label distribution shifts.

The real lesson: probabilistic systems need explicit correctness telemetry. “No errors in the logs” is meaningless when the system is wrong politely.

Mini-story #2: The optimization that backfired

A retail platform added an LLM-based “product description improver.” It took vendor text and cleaned it up for consistency. The initial version ran asynchronously and wrote results to a cache. It was slow but safe: if the job failed, the old text stayed. Then marketing wanted it “real time” in the product editor so humans could watch the magic.

The optimization plan looked reasonable: cache prompts and reuse results; increase concurrency; reduce timeouts so the UI wouldn’t hang. They also added a second model as a fallback. They measured p50 and declared victory.

What they missed was p99 during peak editor usage. When concurrency spiked, the upstream API started rate limiting. The system hit retries. Retries amplified load. The fallback model kicked in and returned different styles. Users started saving drafts with inconsistent tone and, worse, occasional hallucinated specs. The system wasn’t “down,” it was “erratic.”

Costs spiked because the retry storm increased token usage. Finance noticed before engineering did. The backfire was perfect: worse UX, higher cost, and a support backlog because vendors complained about incorrect edits.

The fix was classic SRE discipline: cap concurrency, implement jittered exponential backoff with hard retry budgets, move generation back to async with explicit “draft ready” UX, and add a validation layer that checks outputs against structured product attributes (dimensions, materials) before accepting. They also started monitoring rate-limit responses as a first-class SLI.

Mini-story #3: The boring but correct practice that saved the day

A financial services team rolled out an internal “AI assistant” for analysts. It answered questions using a retrieval system over policies and prior reports. The initial risk review was tense: data leakage, hallucinations, compliance. The team did something unfashionable: they built a thorough evaluation harness before broad rollout.

They created a fixed test set of queries with expected citations. Every model or prompt change had to pass: answer must include citations; citations must map to approved documents; no PII in output; latency under a budget. They also implemented request logging with redaction and stored the retrieved document IDs alongside the response for auditability.

Two months later, the upstream model provider changed behavior. Not “downtime,” just different output formatting and a slightly different tendency to generalize. The harness caught it the same day because citation adherence slipped. The team froze rollout, pinned the model version where possible, and used the fallback response mode (“Here are the documents; no synthesized answer”) until they could re-validate.

No incident, no executive escalation, no compliance panic. Boring gates did what boring gates do: prevent exciting outages.

Fast diagnosis playbook: find the bottleneck in minutes

This is the “on-call at 2 a.m.” version. You don’t need a philosophy of AI. You need a culprit.

First: Is it the model, the network, or your system?

1. Check user-visible symptoms: is it slow, wrong, or failing? Slow and failing are easier than wrong.
2. Check dependency health: LLM API rate limits, timeouts, DNS, TLS, outbound egress, vendor status (if you have it).
3. Check your own resource saturation: CPU, memory, GPU utilization, queue depth, thread pools, connection pools.

Second: Identify which stage is dominant

Most AI features are a pipeline:

• Request parsing + auth
• Prompt construction or feature extraction
• Retrieval (vector search / database fetches)
• Inference (remote API or local model)
• Post-processing (validation, formatting, policy checks)
• Writeback/caching

Time each stage. If you can’t, add instrumentation before you add more “AI.”

Third: Stop the bleeding

• Enable fallback mode (no generation, retrieval-only, cached responses, or disable feature flag).
• Lower concurrency and enforce timeouts.
• Turn off retries that don’t have budgets.
• Rate limit at the edge; protect core services with bulkheads.

Fourth: Decide if you have a correctness incident

If users report “it’s giving bad answers,” treat it like data corruption: sample, reproduce, find a common pattern, and stop rollout. Correctness incidents don’t show up in CPU graphs.

Practical tasks with commands: evidence over vibes

These tasks are biased toward Linux servers running an AI-adjacent service (RAG pipeline, vector search, model gateway, or a locally hosted inference server). For each: command, what the output means, and what decision you make.

Task 1: Confirm basic service health and error rate (systemd + logs)

cr0x@server:~$ systemctl status ai-gateway --no-pager
● ai-gateway.service - AI Gateway
     Loaded: loaded (/etc/systemd/system/ai-gateway.service; enabled)
     Active: active (running) since Mon 2026-01-22 08:11:02 UTC; 3h 12min ago
   Main PID: 2147 (ai-gateway)
      Tasks: 38
     Memory: 612.3M
        CPU: 1h 44min

Meaning: The process is up; memory and CPU are visible. This does not prove correctness or upstream health.

Decision: If not active/running, restart and investigate crash loops. If running, move to latency and dependency checks.

cr0x@server:~$ journalctl -u ai-gateway -n 50 --no-pager
Jan 22 11:12:55 server ai-gateway[2147]: WARN upstream timeout model=vendor-llm request_id=9c2a...
Jan 22 11:12:55 server ai-gateway[2147]: WARN retrying attempt=2 backoff_ms=400
Jan 22 11:12:56 server ai-gateway[2147]: ERROR upstream rate_limited model=vendor-llm status=429

Meaning: You’re not “down.” You’re being throttled and timing out. Retries are actively making it worse.

Decision: Reduce concurrency, enforce retry budgets, consider switching to fallback mode.

Task 2: Measure p50/p95/p99 latency at the edge (nginx access logs)

cr0x@server:~$ awk '{print $NF}' /var/log/nginx/access.log | tail -n 2000 | sort -n | awk 'NR==1000{p50=$1} NR==1900{p95=$1} NR==1980{p99=$1} END{print "p50="p50,"p95="p95,"p99="p99}'
p50=0.842 p95=6.913 p99=18.402

Meaning: Median looks fine; tail latency is brutal. That’s usually upstream contention, rate limiting, or queueing.

Decision: Hunt tail causes: timeouts, saturation, connection pools, slow retrieval queries, or model queue depth.

Task 3: Verify DNS and TLS aren’t your “AI problem”

cr0x@server:~$ dig +time=2 +tries=1 api.vendor-llm.internal A | sed -n '1,12p'
; <<>> DiG 9.18.24 <<>> +time=2 +tries=1 api.vendor-llm.internal A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4812
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
api.vendor-llm.internal. 30 IN A 10.20.30.40

Meaning: DNS resolves quickly and correctly here.

Decision: If DNS is slow or failing, fix resolvers/caching before blaming the model. If fine, test connectivity and TLS.

cr0x@server:~$ openssl s_client -connect api.vendor-llm.internal:443 -servername api.vendor-llm.internal -brief < /dev/null
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = api.vendor-llm.internal
Verification: OK

Meaning: TLS handshake succeeds; cert verifies. If this fails intermittently, your “AI outage” might be PKI rotation or MTU trouble.

Decision: If handshake is slow/failing, investigate network path, proxies, or cert deployment.

Task 4: Check rate limits and upstream error mix (curl)

cr0x@server:~$ curl -s -D - -o /dev/null -m 10 https://api.vendor-llm.internal/v1/models
HTTP/2 200
date: Mon, 22 Jan 2026 11:18:02 GMT
x-ratelimit-limit-requests: 3000
x-ratelimit-remaining-requests: 12
x-ratelimit-reset-seconds: 23

Meaning: You’re near the cliff. Remaining requests is low; reset is soon.

Decision: Add client-side throttling and queueing. If you can’t, enable caching or degrade gracefully.

Task 5: See if your own queue is the bottleneck (Linux socket backlog + app metrics proxy)

cr0x@server:~$ ss -lntp | awk 'NR==1 || /:8080/'
State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 512    4096   0.0.0.0:8080      0.0.0.0:*     users:(("ai-gateway",pid=2147,fd=12))

Meaning: A high Recv-Q relative to expected traffic suggests the app isn’t accepting fast enough or is blocked.

Decision: If Recv-Q climbs during incidents, profile thread pools, GC pauses, or downstream waits.

Task 6: Identify CPU saturation and run queue pressure

cr0x@server:~$ uptime
 11:19:44 up  3:22,  2 users,  load average: 18.42, 16.90, 15.77

Meaning: Load averages are high. On a small core count, you’re CPU-saturated or blocked on I/O with many runnable tasks.

Decision: Check core count, then use vmstat/top to decide if it’s CPU or I/O wait.

cr0x@server:~$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0      0  48212  21032 611204    0    0    10    24 1520 4820 78 12 10  0  0
15  0      0  47680  21032 611420    0    0    12    16 1602 5011 81 11  8  0  0

Meaning: High r with low wa implies CPU pressure, not disk I/O wait.

Decision: Scale out, reduce concurrency, or optimize hot paths (prompt building, JSON parsing, crypto, compression).

Task 7: Find the expensive process and its threads (top)

cr0x@server:~$ top -b -n 1 | sed -n '1,20p'
top - 11:20:31 up  3:23,  2 users,  load average: 19.02, 17.10, 15.92
Tasks: 238 total,  15 running, 223 sleeping,   0 stopped,   0 zombie
%Cpu(s): 84.5 us, 11.2 sy,  0.0 ni,  4.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15928.0 total,    121.4 free,   1912.5 used,  13894.1 buff/cache
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2147 app      20   0 3248216 612340  48212 R  720.1   3.8  105:12.33 ai-gateway

Meaning: The gateway is burning CPU across many cores. That’s often serialization, tokenization, encryption, or retrieval scoring.

Decision: Profile; consider moving heavy work off the request path; implement caching; lower log verbosity.

Task 8: Check memory pressure and whether you’re swapping

cr0x@server:~$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15928        1897         118          42       13912       13640
Swap:              0           0           0

Meaning: Low “free” is not a problem by itself; “available” is healthy. No swap configured.

Decision: Memory likely not the bottleneck. If available collapses or OOM appears, tune caches or set limits.

Task 9: If you host models, check GPU utilization and memory headroom

cr0x@server:~$ nvidia-smi
Wed Jan 22 11:22:06 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4   |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|  0  NVIDIA A10G                    On   | 00000000:00:1E.0 Off |                    0 |
|  0%   61C    P0              142W / 150W|  22340MiB / 24564MiB |     98%      Default |
+-----------------------------------------+----------------------+----------------------+

Meaning: GPU is pegged and memory is near full. Tail latency will spike when you queue.

Decision: Reduce batch sizes, increase replicas, move to a smaller model, or push work to an external provider for peaks.

Task 10: Confirm disk latency and saturation (vector indexes love to hurt you)

cr0x@server:~$ iostat -xz 1 3
Linux 6.8.0-41-generic (server)  01/22/2026  _x86_64_  (16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.10    0.00    6.22    9.84    0.00   51.84

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   w_await aqu-sz  %util
nvme0n1        842.0  91320.0     12.0    1.4   14.82   108.4    112.0   1840.0    3.11   8.92   92.6

Meaning: High read load and high r_await with ~93% util suggests storage is a bottleneck (often retrieval index scans or compactions).

Decision: Optimize vector search (HNSW params, caching), increase RAM for hot index, move indexes to faster NVMe, or split shards.

Task 11: Check filesystem space and inode exhaustion (the silent killer)

cr0x@server:~$ df -h /var/lib/vector
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  900G  874G   26G  98% /var/lib/vector

Meaning: You’re at 98% usage. Many databases and index builders behave terribly near full disks.

Decision: Free space immediately; stop compactions; expand volume; enforce retention policies.

cr0x@server:~$ df -i /var/lib/vector
Filesystem        Inodes   IUsed     IFree IUse% Mounted on
/dev/nvme0n1p2  58982400 58611210   371190  99% /var/lib/vector

Meaning: Inodes are exhausted. You can have space but still fail creating files.

Decision: Reduce tiny-file churn (segment counts, log rotation), reformat with more inodes if needed, or consolidate storage layout.

Task 12: Validate that your cache is actually working (Redis example)

cr0x@server:~$ redis-cli INFO stats | egrep 'keyspace_hits|keyspace_misses|evicted_keys'
keyspace_hits:1829441
keyspace_misses:921332
evicted_keys:44120

Meaning: Misses are high and evictions exist. You’re thrashing the cache, likely increasing upstream LLM calls and cost.

Decision: Increase cache size, fix TTLs, normalize cache keys, and cache retrieval results (not just final answers).

Task 13: Detect retry storms via request logs (grep + counts)

cr0x@server:~$ grep -c "retrying attempt" /var/log/ai-gateway/app.log
18422

Meaning: Retrying is common; during an incident this number will jump sharply.

Decision: Enforce a per-request retry budget, add jitter, and avoid retries on 429 unless instructed by Retry-After.

Task 14: Check whether your token usage (cost proxy) is exploding

cr0x@server:~$ awk '/tokens_in=/{for(i=1;i<=NF;i++) if($i ~ /tokens_in=/){split($i,a,"="); in+=a[2]} if($i ~ /tokens_out=/){split($i,b,"="); out+=b[2]}} END{print "tokens_in="in, "tokens_out="out}' /var/log/ai-gateway/app.log
tokens_in=9284410 tokens_out=16422033

Meaning: Output tokens exceed input tokens dramatically. That may be fine for summarization; it’s terrible for “short answer” features.

Decision: Tighten max tokens, force concise formats, and introduce stop sequences or structured output.

Task 15: Validate vector DB query latency (PostgreSQL + pgvector example)

cr0x@server:~$ psql -d rag -c "EXPLAIN (ANALYZE, BUFFERS) SELECT id FROM docs ORDER BY embedding <-> '[0.1,0.2,0.3]' LIMIT 5;"
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..1.05 rows=5 width=8) (actual time=42.118..42.140 rows=5 loops=1)
   Buffers: shared hit=120 read=980
   ->  Index Scan using docs_embedding_hnsw on docs  (cost=0.42..1200.00 rows=10000 width=8) (actual time=42.116..42.136 rows=5 loops=1)
 Planning Time: 0.312 ms
 Execution Time: 42.202 ms

Meaning: 42 ms might be fine, but note the high disk reads. Under load, this becomes tail latency.

Decision: Increase shared buffers, warm the index, improve locality (fewer shards per node), or adjust index parameters to trade accuracy for speed.

Task 16: Confirm kernel/network path isn’t dropping packets (ss + retrans)

cr0x@server:~$ ss -ti dst api.vendor-llm.internal | sed -n '1,20p'
ESTAB 0 0 10.0.1.10:51244 10.20.30.40:https users:(("ai-gateway",pid=2147,fd=33))
	 cubic wscale:7,7 rto:204 rtt:32.1/4.0 ato:40 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:10 bytes_acked:128394 segs_out:402 segs_in:377 send 3.6Mbps lastsnd:12 lastrcv:10 lastack:10 pacing_rate 7.2Mbps retrans:4/210

Meaning: Retransmissions exist. If retrans spikes during incidents, your “model slowness” might be packet loss.

Decision: Investigate network congestion, MTU mismatch, firewall state tables, or NAT exhaustion.

Common mistakes: symptoms → root cause → fix

1) Symptom: “It’s up, but users say answers got worse”

Root cause: Drift in input distribution, retrieval corpus changes, or upstream model behavior change; no correctness monitoring.

Fix: Build a fixed evaluation set; track answer quality with human labels or proxy metrics; add drift detection on query topics and retrieved-doc distributions; add a safe fallback (retrieval-only).

2) Symptom: p50 is fine, p99 is awful

Root cause: Queueing under load, rate limits, GPU saturation, or slow retrieval I/O.

Fix: Cap concurrency; add backpressure; prioritize requests; precompute embeddings; move retrieval to memory; enforce timeouts and abandon late work.

3) Symptom: Costs spike but traffic didn’t

Root cause: Retry storms, prompt bloat, cache misses, or verbose responses.

Fix: Retry budgets; prompt size budgets; output token caps; normalize cache keys; cache intermediate retrieval results.

4) Symptom: Random 429s and timeouts, especially at peak

Root cause: Provider rate limiting or your own connection pool saturation.

Fix: Token bucket rate limiting client-side; respect Retry-After; implement queueing; multi-provider strategy with consistent behavior constraints.

5) Symptom: “Security says no” and you stall for months

Root cause: No data classification, unclear retention, and prompts containing sensitive content.

Fix: Redaction before sending; allowlists for retrieval; encryption; clear retention policies; log scrubbing; run a threat model including prompt injection and data exfiltration.

6) Symptom: Great demo, terrible adoption

Root cause: Feature doesn’t integrate into the workflow; latency too high; results not trustworthy; no undo.

Fix: Advisory mode; inline citations; fast “accept/edit” UX; show confidence and sources; keep the existing workflow intact.

7) Symptom: The system “hangs” under load and then recovers

Root cause: Head-of-line blocking, synchronized retries, or GC pauses from huge prompts/logging.

Fix: Separate queues by priority; jittered backoff; streaming responses where possible; limit payload sizes; reduce synchronous logging.

8) Symptom: Legal/compliance escalations after rollout

Root cause: Unbounded generation without policy checks; outputs that create commitments or regulated advice.

Fix: Output filters; constrained templates; explicit disclaimers where needed; route high-risk intents to humans; audit logs with citations.

Checklists / step-by-step plan

Step-by-step: ship an “AI feature” without turning on-call into performance art

1. Write a non-AI problem statement. If you can’t, stop. You’re about to build theater.
2. Choose feature class: deterministic, probabilistic, or advisory. Default to advisory.
3. Define SLOs: availability, latency (p95/p99), and a correctness proxy. Put them in writing.
4. Define budgets: max cost per request, monthly ceiling, and a “shutoff” threshold.
5. Design fallbacks: cached response, retrieval-only, rules-based, or “feature off.” Make sure core flows still work.
6. Add guardrails: input validation, max prompt size, max tokens, timeouts, circuit breakers, and rate limits.
7. Instrument by stage: prompt build, retrieval, inference, post-processing, writeback. You need a latency breakdown.
8. Build an evaluation harness: fixed test queries, expected citations, red team prompts, regression thresholds.
9. Run a staged rollout: internal users → small cohort → larger cohorts. Watch p99 and correctness.
10. Do a game day: simulate upstream 429s, slowdowns, bad outputs, and drift. Practice toggling fallbacks.
11. Operationalize change management: version prompts, pin models where possible, track corpus versions, and document release notes.
12. Establish ownership: who approves prompt/model changes, who owns the on-call runbook, who owns budget alarms.

Checklist: what to ban from production by default

• Unbounded retries to an upstream LLM provider.
• “Best effort” timeouts (anything above the user’s patience window).
• Shipping without a rollback or fallback mode.
• Logging raw prompts/responses containing customer data.
• Letting generated text directly trigger irreversible actions (refunds, deletions, account changes) without a deterministic gate.
• “One model to rule them all” without evaluation across use cases.

Checklist: signals you should alert on

• p99 latency by pipeline stage (retrieval vs inference vs post-process).
• 429 and 5xx rate from upstream providers.
• Cache eviction rate and hit ratio.
• Token usage per request (in/out), plus variance.
• Queue depth and time-in-queue.
• Correctness proxies: citation rate, validation failures, user “thumbs down,” escalation rate.
• Drift indicators: query topic distribution, embedding distance shifts, retrieval document distribution changes.

FAQ

1) Is “AI in everything” always bad?

No. It’s bad when the label replaces requirements. Put AI where probabilistic help is acceptable: drafting, summarizing, search, routing suggestions. Be conservative with actions.

2) What’s the biggest difference between classic software and AI features operationally?

Correctness becomes statistical and time-varying. Your system can be healthy and wrong. You need evaluation harnesses and drift monitoring, not just uptime checks.

3) Should we host models ourselves or use a provider?

Pick the failure mode you can operate. Providers reduce infra burden but add rate limits, opaque changes, and dependency risk. Self-hosting gives control but adds GPU scheduling, capacity planning, and security surface. Hybrid is common: self-host for predictable workloads, provider for burst.

4) What do we measure first if we have nothing today?

Stage latency breakdown, upstream error mix (429/5xx/timeouts), token usage per request, cache hit ratio, and a single correctness proxy (user feedback or validation pass rate).

5) How do we prevent “prompt changes” from becoming unreviewed production changes?

Version prompts like code. Require review. Run an evaluation suite in CI. Log prompt version per request. Treat prompt edits as releases.

6) What’s the simplest safe fallback for an LLM feature?

Retrieval-only: return the top documents/snippets with links/titles (internally) and let users read. It’s less magical and far more dependable.

7) How do we handle hallucinations without pretending they don’t exist?

Constrain outputs (structured formats), require citations, validate claims against known fields, and route high-impact cases to humans. Also: measure hallucination rate via audits.

8) Does caching LLM responses actually help?

Yes, if you normalize inputs and accept that some outputs can be reused. Cache retrieval results, embeddings, and deterministic post-processing too. Don’t cache sensitive responses without a clear policy.

9) What’s the most common cost trap?

Retries plus verbose outputs. You pay for the same request multiple times, and then you pay again for excessive tokens. Budget retries and cap generation length.

10) How do we explain AI reliability to executives who want the badge?

Translate to risk: tail latency, vendor dependency, correctness regressions, and compliance exposure. Then offer a staged rollout with measurable outcomes and kill switches.

Conclusion: next steps you can take this week

“AI in everything” isn’t a strategy. It’s a label. Your job is to turn labels into systems that behave under pressure.

1. Pick one AI feature and write its outcome metric, harm metric, latency budget, and cost budget on one page.
2. Add a fallback mode that preserves the workflow when the model is slow, wrong, or rate-limited.
3. Instrument stage latency so you can answer “where did the time go?” without guessing.
4. Build a tiny evaluation harness (20–50 representative queries) and run it before every prompt/model/corpus change.
5. Set alerts on 429s, retries, token usage, cache evictions, and p99 by stage. Not because alerts are fun—because surprises are expensive.

If you do those five things, the label can stay silly while the feature becomes real. That’s the only trade that works in production.