NVIDIA vs AMD vs Intel: what competition needs to stay sane

What competition must deliver (and what it must stop doing)

“NVIDIA vs AMD vs Intel” is usually framed as performance, and performance is real. But in production, performance is the third question. First is can we run it reliably? Second is can we buy it predictably? Third is can we change our mind later?

1) Competition must keep software ecosystems honest

If one vendor’s proprietary stack becomes the de facto API for an industry, that vendor stops needing to care about your migration costs. The pressure valve is credible alternatives: not just “it compiles,” but “it ships.” Competition forces:

• Stable, documented kernels and drivers that don’t break userland on a point release.
• Tooling parity (profilers, debuggers, telemetry) so SREs can diagnose issues without a PhD in vendor lore.
• Better standards (compiler IRs, runtime APIs, container distribution patterns) because customers demand portability when they have choices.

2) Competition must make supply chains less ridiculous

Hardware roadmaps are now business roadmaps. If your model training run depends on a single SKU that only one supplier can deliver, you don’t have an infrastructure plan. You have a weather forecast.

3) Competition must push power efficiency as a first-class metric

In 2026, the “cost” of compute is dominated by power, cooling, rack density limits, and the humans keeping the thing alive. The best accelerator is the one that hits your latency/throughput target without forcing you to redesign the datacenter around it.

One quote, worth stapling to the procurement binder: “Hope is not a strategy.” — General Gordon R. Sullivan. In ops, it’s practically a monitoring philosophy.

Short joke #1: The GPU market is the only place where “available” is considered an advanced feature.

A few facts and history that still matter

Six to ten concrete facts won’t solve your architecture, but they will inoculate you against “this is brand new” thinking. Here are nine that still show up in today’s decision-making:

1. CUDA (introduced in 2006) turned GPUs into a mainstream general-purpose compute platform, not just graphics hardware. This created a gravity well of libraries and developer habits.
2. AMD bought ATI in 2006, inheriting GPU talent and product lines, then spent years reconciling graphics-first and compute-first priorities.
3. OpenCL (Khronos, 2009) promised cross-vendor GPU compute. It delivered portability in theory, and “portable performance pain” in practice for many workloads.
4. HBM (High Bandwidth Memory) became a defining advantage for data-parallel accelerators: it’s not just FLOPS, it’s feeding the cores without starving them.
5. PCIe has been the perennial bottleneck for GPU-to-CPU data movement; generation bumps help, but sloppy topology still kills performance.
6. NVLink pushed high-bandwidth GPU-to-GPU communication as a product differentiator, especially for training workloads that need fast all-reduce.
7. ROCm matured unevenly across GPU generations and Linux distros; the story has improved, but “supported matrix” is still a real operational constraint.
8. Intel’s oneAPI is a strategic attempt to unify programming across CPUs, GPUs, and accelerators—useful when it works, frustrating when the ecosystem around it doesn’t.
9. Inference changed the economics: training gets headlines, but steady-state inference at scale is where cost-per-token, power, and operational simplicity decide winners.

These facts aren’t trivia. They explain why NVIDIA’s software moat exists, why AMD’s opportunity keeps reappearing, and why Intel’s story is inseparable from the CPU platform and enterprise relationships.

How to think about NVIDIA vs AMD vs Intel without lying to yourself

Most comparisons are either fan wars or procurement theater. Here’s the production view: treat each vendor as a bundle of silicon + driver + runtime + libraries + tooling + support behavior + supply chain. The GPU is the least of it.

NVIDIA: the default, for reasons that are not purely technical

NVIDIA’s biggest advantage is not raw performance. It’s that CUDA became the center of gravity for ML tooling, plus a long track record of shipping developer experience that works “out of the box” more often than not. In production, that translates into:

• Faster time-to-first-success for teams without deep compiler/runtime expertise.
• Better ecosystem coverage: kernels, attention implementations, inference runtimes, quantization tooling, profilers.
• More predictable support in common Linux environments (still not perfect; nothing is).

The risk is also clear: the more your business depends on CUDA-only code paths, the less negotiating power you have on pricing, lead times, and roadmap constraints.

AMD: credible silicon, software maturity as the make-or-break

AMD’s core value proposition is straightforward: strong performance potential and a chance to break the monoculture. But in production, ROCm isn’t judged by slide decks; it’s judged by:

• Whether your exact distro/kernel combination is supported and stable.
• Whether your key frameworks and kernels hit expected performance without heroic tuning.
• Whether your team can debug and profile problems with the same speed they can on CUDA.

AMD can win big where customers are willing to standardize their environment, validate carefully, and demand vendor accountability. It loses where teams expect “install and pray.”

Intel: the dark horse that matters because enterprises exist

Intel’s position is unique: it already owns much of the CPU platform, the procurement relationships, and the “boring but critical” enterprise channels. The bet is that oneAPI and Intel GPUs can offer a workable acceleration path that fits into existing fleets.

In practice, Intel tends to be most interesting when:

• Your workloads already benefit from Intel CPUs and you want tighter integration.
• You care about long-term vendor stability and enterprise support models.
• You can accept that some corners of the ecosystem will be less mature than CUDA-first paths.

What sane competition looks like

It’s not “three equal vendors at all things.” That never happens. Sane competition is:

• At least two viable choices for your top workloads.
• Portable architectures where your business logic doesn’t marry one vendor API.
• Benchmarks you run yourself, tied to your data, your batch sizes, your latency SLOs.

Software stacks: CUDA, ROCm, oneAPI, and the real cost of “portable”

Portability is not a checkbox. Portability is an engineering budget. The right question isn’t “can it run on multiple vendors?” but “how expensive is it to keep it running on multiple vendors over time?”

The three layers you should separate

If you want optionality, design like you expect to switch. Separate:

• Framework layer: PyTorch, TensorFlow, JAX, XGBoost, Triton Inference Server-style serving stacks.
• Kernel/library layer: cuDNN/hipDNN analogs, BLAS, attention kernels, quantization, collective communication.
• Runtime/driver layer: CUDA runtime/driver, ROCm stack, Level Zero/oneAPI runtimes.

Your portability lives and dies in the middle layer. That’s where performance-critical kernels hide. You can write vendor-neutral model code and still be effectively locked in by the kernel implementations your framework picks at runtime.

What to do in production

• Prefer frameworks that already support multiple backends for your workload class, even if you deploy on one vendor today.
• Avoid custom CUDA extensions unless you must. If you must, isolate them behind a clean interface and keep a CPU fallback for correctness testing.
• Build a “portability CI lane”: run a weekly job on a non-primary vendor to keep the option alive.

Short joke #2: “We’ll just make it portable later” is the software version of “I’ll start backups after this deploy.”

Operational reality: drivers are part of your app

In GPU land, “the app” includes:

• Kernel version
• GPU driver version
• Firmware (GPU, NIC, sometimes motherboard)
• Container runtime and device plugin versions
• Framework build options

If you don’t pin and validate these like you pin and validate database schema changes, you’re choosing random outages.

Hardware realities: memory, interconnects, and why PCIe is always your suspect

When performance is bad, people blame “the GPU.” In practice, it’s often memory bandwidth, topology, or CPU-side starvation. Here are the knobs that actually move the needle.

HBM capacity and bandwidth: the silent constraint

For training and large-batch inference, HBM capacity determines whether you can keep activations and KV cache resident. HBM bandwidth determines whether your compute units spend their life waiting.

Decision rule: if you’re seeing low compute utilization but high memory controller pressure, you don’t need “more TFLOPS.” You need better memory behavior: fused kernels, better batch sizing, quantization, or a different SKU with more bandwidth.

GPU-to-GPU and GPU-to-CPU links: topology matters more than brand

Interconnect is where multi-GPU training lives or dies. If your all-reduce is slow, you can have the best GPU on paper and still lose to a cheaper box with better topology. You need to know:

• Is GPU-GPU traffic going over NVLink-like fabrics or bouncing over PCIe?
• Are GPUs split across CPU sockets with poor NUMA alignment?
• Is the NIC placed on the “wrong” root complex, forcing cross-socket hops?

Power and thermals: performance cliffs you don’t see in benchmarks

Thermal throttling is a quiet killer in dense racks. You can be “healthy” and still miss SLOs because clocks collapse under sustained load. Your monitoring should treat GPU clocks and power draw as first-class metrics, not trivia.

Procurement for SREs: the questions your vendor won’t ask you

Procurement conversations are usually dominated by price and peak throughput. In ops, you should care about: mean time to recover, change management, and fleet heterogeneity.

What you should demand before buying

• A supported software matrix: exact OS versions, kernel ranges, driver versions, and container runtime expectations.
• A firmware update story: how do you patch at scale, how often, and what breaks?
• RMA and sparing expectations: lead times, cross-ship, and what “failure” looks like in their logs.
• Telemetry compatibility: can you pull the metrics you need without proprietary agents that fight your environment?
• Clear interconnect topology docs for the server SKU you’re actually buying, not a reference design.

What you should build internally

• A golden image per vendor stack with pinned versions.
• A canary upgrade lane that runs representative training and inference workloads.
• A benchmark harness that measures not just throughput, but tail latency, warmup behavior, and failure recovery.

Fast diagnosis playbook: what to check first/second/third to find the bottleneck quickly

This is the “stop arguing, start measuring” sequence. It works across vendors because it’s based on physics and operating systems, not brand identity.

First: is the GPU actually being used?

• Check utilization, clocks, power draw, and memory usage.
• If utilization is low but the job is “slow,” you’re likely CPU-bound, I/O-bound, or blocked on synchronization.

Second: is data movement the real bottleneck?

• Look for high PCIe RX/TX with low compute.
• Check PCIe link width/speed (x16 vs x8; Gen5 vs Gen4) and NUMA locality.
• For multi-GPU, validate peer-to-peer and collective bandwidth.

Third: is the framework falling back to a slow path?

• Confirm the intended backend is active (CUDA/ROCm/oneAPI) and that key ops are using accelerated kernels.
• Watch logs for “fallback” warnings and unexpected device placement.

Fourth: is the node stable under sustained load?

• Check throttling (thermals/power), ECC errors, and driver resets.
• Check kernel logs for PCIe/AER errors and GPU Xid-style events.

Fifth: is the cluster scheduling lying to you?

• Validate device plugin health, cgroup constraints, MIG/partitioning configuration, and container runtime integration.
• Confirm you’re not oversubscribing CPU/memory for GPU jobs (common in Kubernetes).

Practical tasks: commands, what output means, and what decision you make

These are real, runnable checks. Use them during incidents, capacity planning, and vendor bake-offs. The point isn’t to collect trivia; it’s to make decisions.

Task 1: Identify GPUs and the driver in use

cr0x@server:~$ lspci -nn | egrep -i 'vga|3d|display'
01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2330] (rev a1)
21:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:74a1] (rev c1)

What it means: You see which vendor devices are present and their PCI IDs. Mixed fleets are normal; mixed drivers on one node are usually not.

Decision: If the device doesn’t match what you think you deployed, stop. Fix inventory and scheduling before tuning anything else.

Task 2: Check kernel and OS details (driver compatibility starts here)

cr0x@server:~$ uname -r
6.5.0-17-generic

What it means: Kernel version affects DKMS modules, IOMMU behavior, and vendor support matrices.

Decision: If you’re outside the vendor’s supported kernel range, you’re debugging a science project. Move to a supported kernel.

Task 3: NVIDIA quick health view (utilization, power, clocks)

cr0x@server:~$ nvidia-smi
Tue Jan 21 02:31:09 2026
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14    Driver Version: 550.54.14    CUDA Version: 12.4               |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|  0  NVIDIA H100 PCIe                 On | 00000000:01:00.0 Off |                    0 |
| N/A   61C    P0              280W / 350W|  62480MiB / 81559MiB |     92%      Default |
+-----------------------------------------+----------------------+----------------------+

What it means: High GPU-Util with high power and stable P0 state suggests the GPU is doing work. Low utilization with high memory usage often means memory-resident but compute-idle (data pipeline, sync, or kernel inefficiency).

Decision: If GPU-Util is low, shift focus to CPU/I/O/scheduling and framework fallbacks.

Task 4: AMD ROCm health view (if ROCm is installed)

cr0x@server:~$ rocm-smi
========================ROCm System Management Interface========================
GPU[0]          : GPU ID: 0x74a1
GPU[0]          : Temp (Sensor edge) (C): 58.0
GPU[0]          : Average Graphics Package Power (W): 240.0
GPU[0]          : VRAM Total (B): 68719476736
GPU[0]          : VRAM Used (B): 42949672960
GPU[0]          : GPU use (%): 88

What it means: Same logic as NVIDIA: utilization/power/temp tell you whether the accelerator is engaged and whether it’s thermally stressed.

Decision: If utilization is high but throughput is low, suspect memory bandwidth or kernel choice; go to profiler and framework logs.

Task 5: Intel GPU enumeration (Level Zero)

cr0x@server:~$ /usr/bin/zeinfo | head
Level Zero API version: 1.13
Driver version: 1.3.29735
Devices:
  Device 0: Intel(R) Data Center GPU

What it means: Confirms Level Zero runtime sees the device and a driver is loaded.

Decision: If the runtime doesn’t enumerate devices, fix driver/runtime installation before touching app configs.

Task 6: Check PCIe link speed/width (common silent failure)

cr0x@server:~$ sudo lspci -s 01:00.0 -vv | egrep -i 'LnkCap|LnkSta'
LnkCap: Port #0, Speed 32GT/s, Width x16
LnkSta: Speed 16GT/s (downgraded), Width x8 (downgraded)

What it means: The card can do PCIe Gen5 x16, but it’s currently running Gen4 x8. That’s a performance tax, sometimes a disaster.

Decision: Treat this as a hardware/BIOS/topology incident: reseat, check risers, BIOS settings, lane sharing, and motherboard layout.

Task 7: Check NUMA topology (GPU placed far from your CPU threads)

cr0x@server:~$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0-31
node 1 cpus: 32-63
node 0 size: 256000 MB
node 1 size: 256000 MB

What it means: Dual-socket system. If your GPU is attached to node 1 but your dataloader runs on node 0, you pay cross-socket latency and bandwidth costs.

Decision: Pin CPU threads and memory allocation to the GPU-local NUMA node for throughput and tail latency stability.

Task 8: Map PCI devices to NUMA nodes

cr0x@server:~$ cat /sys/bus/pci/devices/0000:01:00.0/numa_node
1

What it means: This GPU is local to NUMA node 1.

Decision: Run your input pipeline and CPU-side preprocessing on node 1 (or accept the hit knowingly).

Task 9: Check IOMMU settings (can hurt latency and peer-to-peer)

cr0x@server:~$ dmesg | egrep -i 'iommu|dmari' | head
[    0.112345] DMAR: IOMMU enabled
[    0.112900] DMAR: Intel(R) Virtualization Technology for Directed I/O

What it means: IOMMU is enabled. That’s often required for virtualization and isolation, but misconfigurations can break P2P or reduce performance.

Decision: If you need GPU peer-to-peer or maximum throughput, validate the vendor’s recommended IOMMU mode (enabled, passthrough, or specific kernel params) and test.

Task 10: Find GPU-related kernel errors (AER, resets, Xid-like events)

cr0x@server:~$ sudo journalctl -k -S -2h | egrep -i 'nvrm|xid|amdgpu|pcie|aer|gpu reset' | tail
Jan 21 01:44:10 server kernel: pcieport 0000:00:01.0: AER: Corrected error received: id=00e0
Jan 21 01:44:10 server kernel: amdgpu 0000:21:00.0: GPU reset begin!
Jan 21 01:44:14 server kernel: amdgpu 0000:21:00.0: GPU reset succeeded

What it means: Corrected PCIe errors and a GPU reset. Even “corrected” errors correlate with flaky risers, marginal power, or bad lanes.

Decision: If you see resets under load, stop blaming the model. Quarantine the node, run hardware diagnostics, and engage the vendor.

Task 11: Confirm container runtime sees GPUs (Kubernetes or standalone)

cr0x@server:~$ docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14    Driver Version: 550.54.14    CUDA Version: 12.4               |
+---------------------------------------------------------------------------------------+

What it means: The container can access the GPU and the driver stack is wired correctly through the runtime.

Decision: If this fails, fix container runtime/device plugin/permissions before touching application code.

Task 12: Check Kubernetes GPU allocation (are you actually scheduled on a GPU?)

cr0x@server:~$ kubectl describe pod infer-7c9b6d6c6f-2kqnx | egrep -i 'node:|nvidia.com/gpu|amd.com/gpu|intel.com/gpu|limits|requests'
Node:           gpu-node-12/10.10.12.34
Limits:
  nvidia.com/gpu:  1
Requests:
  nvidia.com/gpu:  1

What it means: The pod is scheduled with a GPU resource limit/request, and you can see which node.

Decision: If requests/limits are missing, you’re on CPU and your “GPU slowdown” is actually “no GPU.” Fix the manifest and admission controls.

Task 13: Check CPU throttling and steal time (GPU jobs can be CPU-starved)

cr0x@server:~$ mpstat -P ALL 1 3
Linux 6.5.0-17-generic (server)  01/21/2026  _x86_64_  (64 CPU)

02:32:01 AM  CPU   %usr  %sys  %iowait  %steal  %idle
02:32:02 AM  all   85.1   8.2     1.3     0.0    5.4
02:32:03 AM  all   88.0   9.1     0.7     0.0    2.2

What it means: CPU is heavily loaded; if your dataloader and networking run on CPU, GPU utilization can drop waiting for input.

Decision: Increase CPU allocation for GPU pods, tune dataloader workers, or move preprocessing off critical nodes.

Task 14: Check disk I/O saturation (training input pipeline)

cr0x@server:~$ iostat -xz 1 3
Device            r/s     rkB/s   await  %util
nvme0n1         120.0   98000.0   18.2   99.5

What it means: NVMe is pegged at ~100% utilization and high await. Your GPU is probably waiting for data.

Decision: Cache datasets locally, use better sharding, increase read-ahead, or move to higher-throughput storage. Do not buy more GPUs to fix a disk bottleneck.

Task 15: Check network health (distributed training all-reduce, remote storage)

cr0x@server:~$ ip -s link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    RX: bytes  packets  errors  dropped  missed  mcast
    9834212345  8123456  0       0        0       0
    TX: bytes  packets  errors  dropped  carrier collsns
    11233455667  9234567  0       0        0       0

What it means: Errors and drops are zero (good). If you see errors/drops climbing, distributed training will look like “GPU slowness” but is actually retransmits and congestion.

Decision: Fix NIC firmware, cabling, switch config, or move collectives to a dedicated fabric.

Task 16: Verify hugepages and memory pressure (page faults hurt)

cr0x@server:~$ grep -E 'HugePages_Total|HugePages_Free|MemAvailable' /proc/meminfo
MemAvailable:   182345672 kB
HugePages_Total:    4096
HugePages_Free:     3900

What it means: Hugepages are mostly free and memory is available; good baseline. If MemAvailable is low, CPU-side paging can stall data pipelines and increase latency.

Decision: Prevent memory overcommit on GPU nodes; set proper pod limits; isolate nodes for GPU workloads.

Three corporate mini-stories (how this fails in real companies)

Mini-story 1: The incident caused by a wrong assumption

The company had a shiny new inference cluster. Same model, same container image, same Kubernetes manifests. The rollout plan was clean: add nodes gradually, let the autoscaler do the rest. The dashboard looked fine—pods were running, GPUs were “allocated,” and the service was returning answers.

Then tail latency doubled. Nothing was obviously broken. GPU utilization was low, CPU utilization was high, and application logs were full of harmless-looking warnings that no one had ever read.

The wrong assumption: “If the pod requests a GPU, it will use the GPU.” In reality, the model runtime inside the container didn’t match the driver stack on the new nodes. The framework detected an incompatibility and quietly fell back to CPU for several key ops. Not all ops—just enough to keep the service “working,” slow enough to make SLOs cry.

The fix was boring: align the driver version with the container’s expected runtime, add a startup check that fails the pod if the backend isn’t active, and enforce node labeling so incompatible nodes can’t be selected. The postmortem was even more boring: “We should have tested the exact node image in canary.” Correct. Also painful.

Mini-story 2: The optimization that backfired

A training team wanted faster epoch times. They had a new multi-GPU box with a fast interconnect and decided to crank up data loader workers and increase prefetching. It worked in a small test: GPU utilization rose, and steps got faster.

In the full run, performance degraded over hours. Not immediately, which made it a great mystery and an even better time sink. GPUs started oscillating: bursts of high utilization followed by idle gaps. Eventually, nodes began throwing corrected PCIe errors, then occasional GPU resets.

The “optimization” increased CPU load and memory churn, which increased chassis temperatures and pushed the platform into marginal stability. The GPUs were fine; the server wasn’t. Under sustained load, a combination of thermals and borderline PCIe signal integrity caused resets. The cluster looked like a software issue because the job would restart and continue—just slower, noisier, and with a growing backlog.

The backfire was solved by reducing CPU-side pressure, improving airflow management, and setting power/clock limits to avoid thermal cliffs. The lesson was sharp: if a performance tweak changes power draw, you’ve also changed reliability. Treat it like a production change, not a notebook experiment.

Mini-story 3: The boring but correct practice that saved the day

An infrastructure team ran a mixed-vendor fleet: some nodes on one vendor for training, others on another vendor for inference experiments. They were constantly tempted to “standardize later,” but they did one thing consistently: they pinned a golden image per node class, with a validated driver/kernel/container set. No snowflakes, no manual hotfixes.

During a security patch cycle, a kernel update rolled through the general compute fleet. GPU nodes were excluded by policy. Someone complained—“why are GPU nodes special?”—and the answer was the same as always: because kernel/driver mismatches are not a fun way to spend weekends.

Two weeks later, a vendor driver update was released to address stability under a certain workload pattern. The team rolled it out to a small canary pool, ran a full suite of training and inference benchmarks, and watched logs for corrected PCIe errors and throttling. Only then did they roll it broadly.

The payoff arrived quietly: while other teams were firefighting mysterious regressions after the kernel update, this team had clean baselines and a controlled upgrade path. The incident never happened for them. That’s the whole point of boring correctness—it prevents stories from existing in the first place.

Common mistakes: symptom → root cause → fix

1) Symptom: GPU utilization is low, but the job is slow

Root cause: CPU-bound input pipeline, storage bottleneck, or framework fallback to CPU for key ops.

Fix: Check CPU (mpstat), disk (iostat), and backend activation logs. Fail fast if the backend isn’t active. Pin CPU/memory to the GPU’s NUMA node.

2) Symptom: Multi-GPU training is slower than single GPU scaling math

Root cause: Collective communication bottleneck (NIC placement, topology, poor interconnect utilization), or small batch sizes causing sync overhead dominance.

Fix: Verify topology (PCIe/NVLink-like links), ensure NIC locality, tune batch sizes and gradient accumulation, and isolate network for collectives.

3) Symptom: Great performance in a benchmark, mediocre in production

Root cause: Benchmark fits in cache/HBM and doesn’t model real I/O, request patterns, or tail latency. Thermal throttling under sustained load.

Fix: Benchmark with realistic sequences, concurrency, and durations. Monitor clocks and power under sustained load. Set stable power caps if needed.

4) Symptom: Random “device disappeared” / GPU resets

Root cause: PCIe/AER errors, marginal risers, power delivery issues, or driver/firmware bugs triggered under load.

Fix: Quarantine node. Check journalctl -k for AER/GPU reset logs. Validate PCIe link width/speed. Update firmware/driver in a canary, then roll out.

5) Symptom: Inference latency spikes every few minutes

Root cause: Background compaction, dataset reads, autoscaler churn, CPU garbage collection, or GPU clock changes due to power management.

Fix: Warm models, pre-allocate memory pools, pin clocks/power where appropriate, and remove noisy neighbors (CPU and IO isolation).

6) Symptom: Porting from CUDA to “portable” backend compiles but is much slower

Root cause: Missing optimized kernels (attention, layernorm, GEMM variants), different fusion behavior, or fallback to generic implementations.

Fix: Profile and identify hot ops, then choose libraries that provide optimized kernels for the target backend. Keep the model architecture flexible where possible.

7) Symptom: Kubernetes shows GPUs allocated, but inside container no GPU is visible

Root cause: Device plugin mismatch, container runtime misconfiguration, insufficient permissions, or missing vendor container hooks.

Fix: Validate with a minimal GPU container test, check device plugin logs, and standardize runtime configuration across node pools.

Checklists / step-by-step plan

Step-by-step: selecting a vendor (or mixing vendors) without regrets

1. Write down your actual workload mix. Training vs inference, batch sizes, precision modes, sequence lengths, memory footprint, and communication patterns.
2. Define two metrics per workload: one performance (throughput or time-to-train) and one business metric (cost per token, cost per epoch, power per request).
3. Pick three representative tests you can run on every candidate platform: (a) steady-state inference, (b) long training run, (c) failure recovery test (node drain + reschedule).
4. Build a golden image per platform with pinned kernel/driver/runtime versions and a reproducible build pipeline.
5. Validate topology (PCIe speed/width, NUMA locality, NIC placement) before running any benchmark. Otherwise you’re benchmarking your mistakes.
6. Run sustained tests (hours, not minutes). Watch clocks, temps, corrected PCIe errors, and resets.
7. Decide your portability stance explicitly: “CUDA-first but portable architecture,” or “multi-backend CI,” or “single vendor for now.” Pretending you’ll do all three is how budgets die.
8. Operationalize upgrades: canary lane, rollback plan, and automated health checks that validate the backend is active.
9. Negotiate support like an SRE: require clear escalation paths for driver issues, and define what logs/telemetry are needed to open a case.

Checklist: before you blame the GPU vendor

• PCIe link is not downgraded (speed and width match expectations).
• NUMA alignment is correct for CPU threads and memory allocations.
• No corrected PCIe errors in kernel logs under load.
• Storage pipeline isn’t saturated.
• Network errors/drops are zero for distributed jobs.
• Framework is using the intended backend and not silently falling back.
• Thermals and power are stable; no sustained throttling.

Checklist: how to keep competition alive inside your org

• Keep at least one non-primary vendor node pool, even if small.
• Run a weekly portability job: compile, run correctness, record perf deltas.
• Ban vendor-specific extensions unless there’s a measured ROI and an exit plan.
• Track “migration blockers” like technical debt: assign owners and budgets.

FAQ

1) Should we standardize on one vendor to reduce operational complexity?

Standardize your process, not your supplier. A single-vendor fleet is simpler until it isn’t—pricing, lead times, and roadmap surprises are operational complexity too. If you can afford it, keep a small second-vendor pool to preserve leverage and portability muscle.

2) Is CUDA lock-in always bad?

No. CUDA can be the fastest path to production and often the best-supported. Lock-in becomes bad when you build custom CUDA-only kernels everywhere and can’t negotiate pricing or pivot when supply dries up. Use CUDA, but architect for optionality.

3) What’s the number one hidden performance killer?

PCIe and topology issues. A downgraded link (Gen5 x16 running as Gen4 x8) or cross-socket traffic can erase the benefit of a more expensive GPU.

4) For inference, what matters more: compute or memory?

Often memory—capacity and bandwidth—because KV cache and activations dominate. Compute matters too, but many inference workloads are bottlenecked by moving data, not doing math.

5) Can we rely on “portable” APIs and expect near-equal performance?

Expect functional portability, not performance parity. Performance comes from tuned kernels, fusion strategies, and mature libraries. Portability is still worth it; just budget for backend-specific tuning.

6) How do we prevent silent CPU fallback?

Add explicit startup checks in your service/job that verify the backend is active and that a small tensor op runs on the device. If not, crash early and loud. “Working but slow” is the worst failure mode.

7) Do we need separate golden images per vendor?

Yes, if you want sanity. Treat the driver/runtime/kernel combo as part of the application. Pin it, test it, and roll it out like you roll out database migrations.

8) What’s a sane way to benchmark vendors?

Use your real models and real request patterns. Run sustained tests. Measure tail latency, stability (resets, corrected errors), and operational friction (tooling, upgrade ease). If you only measure peak throughput for five minutes, you’re benchmarking optimism.

9) Are power caps a hack or a best practice?

They’re a best practice when you’re hitting thermal or power-delivery cliffs. A slightly lower peak clock with stable sustained performance beats a flaky node that resets under load.

10) What’s the simplest way to keep multi-vendor optionality?

Keep your model code free of vendor-specific extensions, run a periodic job on a second backend, and avoid building your serving stack around one vendor’s proprietary features unless you have a measured business need.

Next steps you can actually execute

If you want competition to keep the market sane, you have to keep your own architecture sane. That means you can switch, or at least credibly threaten to. Here’s the practical path:

1. Instrument first: add GPU utilization, clocks, power, memory, PCIe errors, and backend selection signals to your monitoring. If you can’t see it, you can’t negotiate it.
2. Build a portability lane: a weekly CI run on a non-primary vendor node. It doesn’t need to be fast. It needs to be real.
3. Harden your fleet: golden images, pinned versions, canary rollouts, and automated health checks that fail fast on fallback paths.
4. Benchmark like an operator: sustained runs, real data, real concurrency, and a failure injection (node drain, restart, reschedule) to measure recovery behavior.
5. Procure like an SRE: topology docs, supported matrices, firmware strategy, and support escalation paths are not “nice to have.” They’re the difference between shipping and firefighting.

The market will keep doing what markets do: concentrate, extract rent, and call it “innovation.” Your job is to keep options alive—technically and commercially—so your production systems don’t become a monument to somebody else’s margins.