Drivers as Weapons: How Software Can “Change” Your GPU

December 26, 2025 • February 3, 2026 • Read: 21 min • Views: 0

Was this helpful?

You reboot a box and your “same” GPU is suddenly slower, hotter, less stable, or just… different.
The silicon didn’t change. The driver did. And the driver is the part with opinions: about power,
clocks, memory, scheduling, error handling, even which features exist today versus yesterday.

In production, a GPU driver update is not “just a patch.” It’s a behavioral rewrite of the device you think you own.
Treat it like a change to the hardware contract. Because that’s what it is.

What it means for software to “change” a GPU

When engineers say “the driver changed my GPU,” they don’t mean it rewired transistors.
They mean the driver changed the GPU’s observable behavior—the practical contract:
how fast it runs, how it schedules, how it uses memory, how it handles errors, and how it throttles under heat or power pressure.

A modern GPU is a negotiated settlement between hardware, firmware, kernel driver, user-space libraries, and sometimes a daemon
that keeps state across processes. Different driver versions can:

Change default power limits and boost behavior.
Change P-state transitions (idle vs compute clocks).
Change memory management and eviction behavior (especially under oversubscription).
Change scheduling fairness between contexts (MPS, MIG, time-slicing).
Enable or disable features (BAR sizing, SR-IOV, resizable BAR, peer-to-peer access).
Change error recovery: what becomes a recoverable fault vs a dead GPU requiring reset.
Change library kernels (cuDNN, cuBLAS) selection heuristics and autotuning outcomes.

The part that stings is this: you can do everything “right” in your code and still see a 20% swing because the driver changed a default.
The second part that stings: the driver has legitimate reasons. Newer drivers fix correctness issues, security problems, and race conditions.
But they also change behavior, and behavior is what production cares about.

A note on what “driver” includes

People say “driver” as shorthand, but you should mentally separate:

Kernel module (Linux: nvidia.ko and friends): owns the device, handles interrupts, memory mapping, reset logic.
User-space driver libraries (CUDA runtime, OpenCL ICD, GL/Vulkan stacks): translate API calls into GPU work.
Firmware (on the card): handles microcontroller tasks like power management and sometimes scheduling primitives.
Management stack (NVML, nvidia-smi, persistence daemon): sets clocks, power limits, compute mode, MIG partitions.

Any one of these can change between “version A” and “version B.” Sometimes you upgrade one and not the other. That’s when the fun begins.

Facts & history: how we got driver-shaped GPUs

Here are a few concrete context points that explain why drivers have so much leverage today. Keep these in your head; they help diagnose reality.

Early GPUs were fixed-function. In the late 1990s and early 2000s, GPUs mostly did predefined graphics pipelines. Drivers were important, but “changing the GPU” meant bug fixes and performance tweaks, not whole new compute behaviors.
Shader programmability moved complexity into drivers. Once GPUs became programmable (shaders), drivers started doing aggressive compilation and scheduling decisions. You got performance… and variability.
CUDA (2006) made GPUs general compute devices. That raised the stakes: correctness and determinism began competing with maximum throughput. Driver policy became part of application performance.
Boost clocks turned frequency into a software policy knob. Dynamic boosting means two “identical” cards can run different clocks depending on thermal, power, and driver heuristics.
ECC on some GPUs introduced a speed-vs-safety trade. Enabling ECC can reduce usable memory and sometimes performance; drivers control reporting, scrubbing behavior, and sometimes defaults.
MPS and then MIG changed sharing semantics. Multi-process service and multi-instance GPU are ways to share a device. They are also new ways to have noisy neighbors, context thrash, and confusing utilization stats.
Resizable BAR and large BAR mappings altered CPU↔GPU data paths. Depending on platform and driver, the CPU can map more VRAM directly, changing transfer behavior and some workloads’ performance.
Security became a driver feature, not just an OS feature. Isolation, memory protections, and mitigations live in the driver stack. Some mitigations cost performance; some fix scary things. Pick your poison carefully, then document it.

Drivers as weapons: the real attack surface

“Weapon” doesn’t always mean “malware.” In ops, “weapon” often means “a lever that can ruin your SLO silently.”
Drivers are exactly that lever because they sit below your application and above your hardware.

Weaponized by accident: policy changes that look like hardware drift

Driver updates often bring new defaults: different boost curves, different fan policies, new memory allocators, new scheduling.
You don’t notice until something that used to run at P0 now sits in P2 under “the same” load,
or memory copies regress because the IOMMU path changed.

The easiest way to lose a week is to assume “hardware is stable.” Hardware is stable. The contract is not.

Weaponized on purpose: pushing you into an ecosystem

Vendors are not charities. Drivers can steer you toward certain toolchains and away from others:
deprecations, feature gating, “supported” versions, compatibility matrices that quietly punish you for upgrading the wrong component.

This is not automatically evil. It’s just how commercial stacks evolve. But in production, you should treat it as an incentive gradient.
Decide whether you will follow the gradient or isolate yourself from it with pinned versions and controlled rollouts.

Weaponized by adversaries: supply chain and privilege

GPU drivers run with high privileges. They touch kernel space, DMA, memory mapping. If you’re thinking like an attacker,
the driver is a spectacular place to look.

Your response should not be panic. Your response should be: measured paranoia.
Signed packages, controlled repositories, secure boot where possible, and a rollback plan you can execute at 3 a.m.

How drivers change performance and reliability

1) Clocks: your “same” GPU at a different frequency

Clock behavior is policy. Drivers read temperature, power draw, workload type, and sometimes “application clocks” settings.
If your new driver changed boost aggressiveness or power limit interpretation, you’ll see a throughput change without any code change.

Watch for these clues: P-state stuck above idle, or never reaching P0 under load; clocks oscillating; power cap hits earlier.

2) Power limits and thermal targets

Power limit is not just “how much heat you make.” It’s how often the GPU can sustain boost.
Data centers love power capping. ML training hates surprise power capping. Your driver can enforce or interpret caps differently.

3) Memory management and oversubscription

On paper, your GPU has X GiB. In practice, allocations, fragmentation, and page migration behavior are driver-defined.
Unified memory, pinned memory, and UVA all add layers where “the driver decides.”

That’s why a driver update can turn a workload from “runs fine” into “random OOM at step 8000.”
The memory allocator changed. Or the eviction threshold did. Or the driver now reserves more memory for itself.

4) Scheduling and concurrency: MPS, MIG, and time-slicing

If you share GPUs, you are trusting the driver scheduler to be fair and performant.
That scheduler changes. Sometimes it gets better. Sometimes it changes the shape of latency spikes.

MPS can improve throughput for many small kernels. It can also smear faults across clients and confuse attribution.
MIG can give strong isolation, but you can still bottleneck on shared resources and driver-level overhead.

5) Error handling and recovery: the difference between “warning” and “dead node”

The driver decides what to do when the GPU misbehaves: reset the channel, reset the device, or give up.
This is where reliability lives.

In Linux, you’ll see NVRM Xid messages. Sometimes they’re benign and recoverable. Sometimes they predict imminent death.
Driver versions change what “recoverable” means.

6) Library kernel selection: “driver update” that isn’t only the driver

Many performance swings blamed on “the driver” are really user-space libraries changing their heuristics:
cuDNN chooses different convolution algorithms; cuBLAS changes GEMM selection; autotuning results vary with minor changes.

If you upgrade the driver and the CUDA toolkit in the same sprint, you’ve created a mystery novel. Separate them.

One reliability quote to keep you honest

Paraphrased idea from Werner Vogels: Everything fails, all the time; build systems that expect it and recover automatically.

Fast diagnosis playbook (first/second/third)

When a GPU workload slows down or gets flaky after a driver change, do not start by rewriting your model or blaming “PCIe.”
Start with a tight triage loop that isolates where the contract changed.

First: confirm what actually changed

Driver version, CUDA version, kernel version.
GPU firmware version if available.
Power limit, persistence mode, compute mode, MIG/MPS state.
ECC enabled/disabled.

Second: decide whether the bottleneck is clocks, memory, or scheduling

Clocks/power issue: P-state not reaching expected levels, power cap hits, temperature spikes, perf oscillates.
Memory issue: OOMs, fragmentation symptoms, paging/migration spikes, sudden drop in effective batch size.
Scheduling issue: multi-tenant regressions, weird utilization, tail latency spikes, kernels delayed behind other contexts.

Third: reproduce with a minimal test and pin variables

Run a stable microbenchmark or a small representative inference/train step with fixed seeds.
Use the same container image and same host driver across tests (or vice versa—control one side).
Collect: clocks, power, temp, utilization, memory usage, Xid logs.

If you can’t reproduce it in a minimal test, it’s probably a concurrency or contention problem, not “raw GPU speed.”

Practical tasks: commands, outputs, decisions (12+)

These are the tasks I actually run when someone says “the driver changed our GPUs.”
Each task includes: command, what the output means, and the decision you make.

Task 1: Identify the driver and GPU inventory

cr0x@server:~$ nvidia-smi
Tue Jan 13 10:22:41 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4  |
|-----------------------------------------+----------------------+----------------------|
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  | 00000000:3B:00.0 Off |                    0 |
|  36%   48C    P0             210W / 400W|  1234MiB / 40960MiB   |     12%      Default |
+-----------------------------------------+----------------------+----------------------+

Meaning: Confirms driver version, reported CUDA compatibility, persistence mode, power cap, perf state, and basic health counters.

Decision: If the driver version differs from your baseline, treat everything else as suspect until proven stable. Record it in the incident notes.

Task 2: Confirm kernel module version loaded

cr0x@server:~$ modinfo nvidia | head -n 8
filename:       /lib/modules/6.5.0-28-generic/updates/dkms/nvidia.ko
version:        550.54.14
license:        NVIDIA
description:    NVIDIA kernel module
author:         NVIDIA Corporation
srcversion:     8F6C9C5B6B61A2D4E3B1B19
depends:
retpoline:      Y

Meaning: Ensures the running kernel module matches the package version you think you installed.

Decision: If nvidia-smi and modinfo disagree, you have a partial upgrade or stale module. Fix before benchmarking.

Task 3: See recent driver errors (Xid) and GPU resets

cr0x@server:~$ sudo dmesg -T | egrep -i 'nvrm|xid|gpu has fallen off|reset' | tail -n 20
[Tue Jan 13 10:18:02 2026] NVRM: Xid (PCI:0000:3b:00): 31, pid=24811, name=python, Ch 0000002a
[Tue Jan 13 10:18:02 2026] NVRM: Xid (PCI:0000:3b:00): 31, pid=24811, name=python, MMU Fault: ENGINE GRAPHICS
[Tue Jan 13 10:18:04 2026] nvidia-modeset: WARNING: GPU:0: Corrected error detected

Meaning: Xid codes are driver-level fault reports. MMU faults can indicate buggy kernels, bad memory access, or driver regressions.

Decision: If Xids appear after the update and correlate with job failures, stop arguing about “performance” and start treating it as a reliability regression. Consider rollback or isolate to canary nodes.

Task 4: Check power limit and whether it was silently changed

cr0x@server:~$ nvidia-smi -q -d POWER | egrep -i 'Power Limit|Default Power Limit|Enforced Power Limit'
    Power Limit                    : 400.00 W
    Default Power Limit            : 400.00 W
    Enforced Power Limit           : 400.00 W

Meaning: Shows current and default power policy. A mismatch between “default” and “enforced” is a red flag.

Decision: If enforced limit dropped (common in shared racks), decide whether to raise it (if allowed) or retune batch sizes and expectations. Don’t benchmark with unknown caps.

Task 5: Observe real-time clocks and throttling reasons

cr0x@server:~$ nvidia-smi dmon -s pucvmt -d 1 -c 5
# gpu   pwr  u  c  v  m  t
# Idx     W  %  MHz  MHz  %  C
    0   212 12  1410 12150  3  49
    0   225 85  1410 12150 61  56
    0   238 92  1410 12150 72  59
    0   399 98  1170 12150 84  66
    0   401 99  1140 12150 86  67

Meaning: You can see clocks drop as power hits the cap. Utilization stays high, clocks fall: that’s power throttling.

Decision: If clocks collapse at power cap, either raise the cap (if policy allows), improve cooling/airflow, or accept lower throughput. Don’t blame “the model.”

Task 6: Confirm persistence mode (to avoid cold-start jitter)

cr0x@server:~$ nvidia-smi -q | egrep -i 'Persistence Mode' | head -n 1
    Persistence Mode               : Enabled

Meaning: Disabled persistence can cause repeated init costs and clock settling delays between jobs.

Decision: In shared clusters running short jobs, enable persistence mode on compute nodes to reduce variability—unless your org has a reason not to.

Task 7: Check ECC mode and error counts

cr0x@server:~$ nvidia-smi -q -d ECC | egrep -i 'ECC Mode|Volatile|Aggregate'
    ECC Mode
        Current                     : Enabled
        Pending                     : Enabled
    Volatile Uncorrectable ECC Errors : 0
    Aggregate Uncorrectable ECC Errors : 0

Meaning: ECC changes memory behavior and can surface hardware faults. Volatile errors are since boot; aggregate persists.

Decision: If aggregate uncorrectable errors increase, this is trending toward a hardware RMA conversation, not a tuning exercise. Quarantine the node.

Task 8: Verify MIG state and layout (is the GPU actually partitioned?)

cr0x@server:~$ nvidia-smi -L
GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-0a1b2c3d-4e5f-6789-abcd-0123456789ef)
  MIG 1g.5gb Device 0: (UUID: MIG-GPU-0a1b2c3d-4e5f-6789-abcd-0123456789ef/1/0)
  MIG 1g.5gb Device 1: (UUID: MIG-GPU-0a1b2c3d-4e5f-6789-abcd-0123456789ef/2/0)

Meaning: If MIG devices exist, your “GPU 0” is not a single full GPU from the job’s perspective.

Decision: If performance regressed after driver update and MIG is on, confirm your scheduler is assigning the expected MIG profile. Many “regressions” are actually smaller slices.

Task 9: Validate PCIe link width/speed (classic silent limiter)

cr0x@server:~$ sudo lspci -s 3b:00.0 -vv | egrep -i 'LnkCap|LnkSta'
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
LnkSta: Speed 8GT/s (downgraded), Width x8 (downgraded)

Meaning: The GPU is running at reduced PCIe speed/width. That can happen after firmware changes, BIOS settings, or marginal hardware.

Decision: If link is downgraded, your bottleneck may be host I/O. Fix the platform (reseat, BIOS, riser, ASPM settings) before blaming the driver.

Task 10: Check IOMMU status (can affect DMA performance and compatibility)

cr0x@server:~$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-6.5.0-28-generic root=/dev/nvme0n1p2 ro quiet splash intel_iommu=on iommu=pt

Meaning: Shows whether IOMMU is enabled and if pass-through mode is used. Some setups regress if IOMMU is fully translating.

Decision: If you see unexpected IOMMU settings after a kernel/driver upgrade, re-evaluate DMA performance and device isolation requirements. Don’t “optimize” this blindly.

Task 11: Confirm container sees the expected CUDA runtime

cr0x@server:~$ docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
Tue Jan 13 10:25:10 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4  |
+---------------------------------------------------------------------------------------+

Meaning: Confirms host driver is exposed into the container and that the container’s tooling agrees.

Decision: If the container can’t run nvidia-smi or reports mismatched components, fix your container runtime integration (NVIDIA Container Toolkit) before you chase “model bugs.”

Task 12: Detect library-level shifts (cuDNN / cuBLAS) that look like driver regressions

cr0x@server:~$ python -c "import torch; print(torch.__version__); print(torch.version.cuda); print(torch.backends.cudnn.version())"
2.2.1
12.1
8902

Meaning: Your framework’s CUDA/cuDNN versions matter as much as the driver. A driver upgrade may coincide with a container refresh.

Decision: If versions changed, you do not have a clean A/B. Re-run tests with only one variable changed (driver only, or container only).

Task 13: Check GPU process list and whether “utilization” is lying to you

cr0x@server:~$ nvidia-smi pmon -c 1
# gpu        pid  type    sm   mem   enc   dec   jpg   ofa   command
# Idx          #   C/G     %     %     %     %     %     %     name
    0      24811     C    92    18     0     0     0     0     python
    0      25102     C     5     2     0     0     0     0     python

Meaning: Multiple compute processes share the device. Your “slowdown” might be contention, not a driver regression.

Decision: If unexpected processes exist, fix scheduling/isolation first. Performance debugging without tenancy control is astrology with better graphs.

Task 14: Capture a short performance trace (high-level) for regression proof

cr0x@server:~$ nvidia-smi --query-gpu=timestamp,driver_version,pstate,clocks.sm,clocks.mem,power.draw,temperature.gpu,utilization.gpu,utilization.memory --format=csv -l 1 -c 5
timestamp, driver_version, pstate, clocks.sm [MHz], clocks.mem [MHz], power.draw [W], temperature.gpu, utilization.gpu [%], utilization.memory [%]
2026/01/13 10:26:31, 550.54.14, P0, 1410, 12150, 225.11, 56, 85, 61
2026/01/13 10:26:32, 550.54.14, P0, 1410, 12150, 238.42, 59, 92, 72
2026/01/13 10:26:33, 550.54.14, P0, 1170, 12150, 399.72, 66, 98, 84
2026/01/13 10:26:34, 550.54.14, P0, 1140, 12150, 401.03, 67, 99, 86
2026/01/13 10:26:35, 550.54.14, P0, 1140, 12150, 401.10, 67, 99, 86

Meaning: This is the simplest “receipt” to attach to an incident: clocks, power, temp, util in one CSV.

Decision: If you can show clocks dropping exactly when power caps, you stop debating and start negotiating power policy or thermal remediation.

Short joke #1: A GPU driver update is like a “minor change request” from finance—technically small, emotionally devastating.

Three corporate mini-stories from the trenches

Mini-story 1: The incident caused by a wrong assumption

A company ran a mixed GPU fleet for training and inference. The SRE team had a clean rule: upgrade drivers monthly, patch kernels quarterly.
They assumed the key compatibility boundary was “CUDA version inside the container.” If the container ran, they were happy.

One month, a driver update rolled out to half the inference nodes. Within hours, tail latency got weird: not catastrophic, just enough to miss an internal SLO.
Engineers saw GPU utilization hovering high. They assumed the model got heavier. It hadn’t.

The real change was power policy. The new driver interpreted the platform’s power limit controls differently. Under sustained load, the GPUs hit the cap and downclocked harder.
Same utilization, fewer effective cycles. The system looked “busy” while delivering less work—classic throughput collapse disguised as productivity.

The wrong assumption was that “driver updates don’t change power behavior.” They do. The fix was boring: record and enforce power limits explicitly as part of node provisioning,
and keep a baseline CSV trace (like the one above) from a known-good run. Once they did that, “performance regressions” became provable events, not vibe checks.

Mini-story 2: The optimization that backfired

Another org was chasing maximum throughput on a multi-tenant training cluster. They enabled a sharing feature to improve utilization:
more jobs per GPU, fewer idle gaps, better numbers on a dashboard.

It worked on day one. Then came the driver update. Suddenly, rare training runs started failing with memory-related errors and sometimes driver Xids.
The failures were not evenly distributed: some teams were cursed, others were fine, which made it politically delicious and technically awful.

The optimization had a hidden cost: the new driver version changed memory management behavior under oversubscription and increased the overhead of context switching
for the specific kernel mix they ran. More concurrency meant more fragmentation pressure and more time spent in driver paths, not on the SMs.

The “fix” wasn’t to abandon sharing forever. It was to apply it with constraints: enforce hard per-job memory headroom, cap concurrent contexts per GPU,
and gate the driver upgrade behind stress tests designed to trigger the fragmentation path. The lesson: utilization is not throughput, and throughput is not reliability.

Short joke #2: We tried to “optimize GPU sharing” and accidentally invented a random number generator with a fan.

Mini-story 3: The boring but correct practice that saved the day

A third company did something unfashionable: they kept a “golden node” per GPU class. Same BIOS settings, same kernel, same driver,
same container images. It ran a nightly suite: a microbenchmark, a representative inference batch, and a short training loop.

When a driver update was proposed, it first landed on the golden node and a small canary pool. The suite ran automatically and compared against a pinned baseline.
The team didn’t require perfection; they required explanation. If performance moved, they wanted a reason: power, clocks, memory, scheduling, or library changes.

One update showed a small throughput regression but also eliminated a class of Xid errors that previously caused rare node drains.
The business impact favored reliability. Because they had evidence, they could choose the tradeoff intentionally rather than by surprise.

The “boring practice” was version pinning plus controlled canaries plus regression proof artifacts. It didn’t make anyone famous.
It did keep the cluster out of the incident channel.

Common mistakes: symptom → root cause → fix

These are patterns I see repeatedly when teams treat GPU drivers like a harmless dependency.
Use this as a failure-mode lookup table.

1) Symptom: GPU utilization high, throughput lower than last week

Root cause: Power cap/thermal throttling changed; clocks are lower while SM utilization remains high.

Fix: Measure clocks/power with nvidia-smi --query-gpu and dmon. Confirm power limit. Improve cooling or adjust caps; retune batch sizes if caps are policy.

2) Symptom: Random OOM after driver update, same batch size

Root cause: Driver now reserves more VRAM; allocator behavior changed; fragmentation increased under concurrency.

Fix: Reduce peak VRAM (batch/sequence length), add headroom, reduce concurrent contexts, and compare memory usage baselines across versions.

3) Symptom: Training jobs hang, node needs reboot

Root cause: GPU channel wedged; driver can’t recover; reset path changed. Often correlated with Xid errors.

Fix: Collect dmesg Xids, isolate to driver version, test rollback. Consider enabling a health-check that drains node on Xid patterns before it hard-hangs.

4) Symptom: After update, some nodes are slow, others fine

Root cause: Mixed driver versions, mixed MIG layouts, or platform differences (PCIe link downgraded, BIOS settings).

Fix: Enforce configuration drift controls. Check driver versions, MIG state, and PCIe link status. Don’t compare apples to slightly broken apples.

5) Symptom: Container says CUDA version X, but runtime errors mention incompatible driver

Root cause: Host driver too old for container runtime expectations; or container runtime integration broken.

Fix: Align host driver with required CUDA compatibility. Validate with a simple docker run --gpus all ... nvidia-smi test before running real workloads.

6) Symptom: Latency spikes in multi-tenant inference after update

Root cause: Scheduler fairness changes; more aggressive time-slicing; MPS behavior differences; increased context switch overhead.

Fix: Reduce sharing, use MIG profiles for stronger isolation, or dedicate GPUs for latency-sensitive services. Measure tail latency under load, not average throughput in isolation.

7) Symptom: PCIe transfer performance dropped

Root cause: PCIe link width/speed downgraded; IOMMU mode changed; resizable BAR toggled by BIOS/driver combination.

Fix: Check lspci -vv LnkSta, confirm BIOS settings, validate IOMMU parameters. If needed, re-seat hardware or swap risers before tuning software.

Checklists / step-by-step plan

Step-by-step: safe GPU driver rollout in production

Inventory and classify nodes by GPU model, motherboard, BIOS version, kernel version, and workload type (training vs inference).
Pin a baseline: record driver version, power limits, ECC mode, MIG/MPS state, and a small benchmark result with clocks/power trace.
Upgrade only the driver first (not CUDA toolkit, not containers) on a golden node. Collect the same trace.
Run a stress suite that includes long-running kernels, memory pressure, and concurrency if you use it in production.
Canary a small pool with representative workloads. Watch tail latency and error rates, not just throughput.
Set explicit policies: power limits, persistence mode, compute mode, and MIG layout should be configured, not assumed.
Roll out gradually with a clear rollback condition: specific Xid patterns, a throughput regression threshold, or SLO miss.
Lock it down after rollout: ensure no one autoupdates drivers on random nodes. Drift is how you get “only Tuesdays are slow.”

Pre-flight checklist for “the GPU feels different” reports

Do we have a before/after driver version record?
Are clocks/power/temps comparable?
Did MIG/MPS state change?
Did ECC mode change?
Is PCIe link downgraded?
Are there Xids in dmesg?
Did container CUDA/cuDNN versions change at the same time?
Is there unexpected multi-tenancy on the GPU?

Operational guardrails I recommend (opinionated)

Never update driver + CUDA toolkit + framework in one change window if you care about root cause.
Keep a golden node per GPU class with an automated regression suite.
Make power limits explicit. Defaults are not contracts.
Require a rollback plan that includes kernel module rollback and node drain automation.
Measure tail latency for inference and steps/sec stability for training. Averages hide driver policy changes.

FAQ

1) Can a GPU driver really change performance without changing my code?

Yes. Drivers set clocks, power behavior, scheduling policy, memory allocation strategy, and kernel selection heuristics in libraries.
Your code can be identical while the effective hardware contract changes.

2) If `nvidia-smi` shows high utilization, doesn’t that prove the GPU is fully used?

No. Utilization is “busy time,” not “useful work done.” You can be 99% busy at lower clocks due to power cap and deliver less throughput.
Always look at clocks and power alongside utilization.

3) What’s the single fastest way to prove a power/clock regression?

Collect a short CSV trace of pstate, SM clocks, power draw, temperature, and utilization during the workload.
If clocks drop when power hits the cap, you have your culprit.

4) Are driver updates mostly about performance?

No. Many are about correctness, security, and stability across kernels and platforms. Performance changes are often side effects of policy shifts.
In production, stability fixes can be worth a small slowdown—if you choose it knowingly.

5) Is MIG always better for multi-tenancy?

It’s better isolation than simple time-slicing, but not magic. You can still bottleneck on shared resources and driver overhead.
MIG is great when you need predictable slices; it can be wasteful if your workloads are bursty and don’t fit the profiles well.

6) Why do some driver updates increase memory usage?

Drivers can reserve different amounts of VRAM for internal bookkeeping, page tables, or new features.
Also, library and allocator behavior may change. That “missing 1–2 GiB” is often real, not your imagination.

7) Should we pin drivers forever once things work?

No. Pinning forever is how you accumulate security risk and future incompatibility debt. Pin deliberately, upgrade deliberately.
A controlled cadence with canaries beats both chaos and stagnation.

8) What’s the difference between driver issues and hardware issues?

Hardware issues often show as persistent ECC error trends, repeatable faults across driver versions, or physical link downgrades.
Driver issues correlate tightly with version changes and often affect many nodes similarly. Use Xid logs and ECC counters to separate them.

9) Why does performance vary between “identical” GPUs?

Silicon variation, cooling differences, power delivery, and boost policies all matter. Drivers amplify these differences by changing boost and throttling heuristics.
In other words: identical part numbers are not identical runtime behavior.

10) What should I store in incident artifacts for a GPU regression?

At minimum: driver version, kernel version, GPU model, power limit settings, MIG/MPS/ECC state, a short nvidia-smi --query-gpu trace,
and relevant dmesg excerpts with Xids.

Conclusion: what to do next Monday

If you run GPUs in production, stop treating drivers like a background dependency and start treating them like firmware with opinions.
Drivers don’t just “enable the device.” They define its behavior under power, heat, contention, and failure.

Practical next steps:

Pick a baseline driver per GPU class and record power/ECC/MIG/MPS policies explicitly.
Build a tiny regression suite and run it on a golden node and canary pool before rollouts.
When you see regressions, capture clocks/power/util traces and Xid logs first. Argue later.
Separate driver upgrades from CUDA/toolkit/framework upgrades whenever you want a clean root cause.
Keep rollback paths rehearsed, not theoretical.

Hardware ages. Drivers evolve. Your job is to make those changes visible, testable, and reversible—so “the GPU changed” becomes a diagnosis, not a superstition.