Docker “Too Many Requests” When Pulling Images: Fix Registry Throttling Like You Mean It

What “too many requests” actually means

“Too many requests” is not one thing. It’s a family of throttles that happen at different layers, and the fix depends on which layer is slapping you.
Most teams treat it like a Docker problem. It’s usually a systems design problem with a Docker-shaped symptom.

The common manifestations

• HTTP 429 from a registry endpoint: classic rate limiting. You are exceeding a per-IP, per-user, per-token, or per-organization quota.
• HTTP 403 with “denied: requested access” that only happens under load: sometimes a registry returns misleading auth errors when it’s enforcing limits.
• Kubernetes ImagePullBackOff / ErrImagePull with “toomanyrequests”: the kubelet is pulling on many nodes at once. The registry is saying “nope”.
• CI failures where parallel jobs pull the same base image repeatedly. The image is “cached” only in theory.
• Not throttling, but looks like it: DNS failures, MTU issues, corporate proxies, or TLS interception can produce retry storms that resemble rate limiting.

Where throttling can be applied

Throttling can happen at the registry, at a CDN fronting the registry, at your corporate egress proxy, or at your own NAT gateway.
You might even be throttled by yourself: conntrack tables, ephemeral port exhaustion, or a local mirror that’s under-provisioned.

A useful mental model: a “docker pull” is not one request. It’s a sequence of token fetches, manifest requests, and layer downloads—often many layers, sometimes for multiple architectures.
Multiply that by 200 CI jobs or 500 nodes, and you’ve built a denial-of-service generator with YAML.

One paraphrased idea from John Allspaw (operations/reliability): Reliability comes from designing for failure, not from hoping failures won’t happen.
Treat registry throttling the same way: a known failure mode you design around.

Joke #1: A pull storm is just your infrastructure’s way of saying it misses the days when outages had a single root cause.

Fast diagnosis playbook (first/second/third)

When production is burning, you don’t have time for interpretive dance with logs. Here’s the fastest path to the bottleneck.

First: confirm it’s real rate limiting (not networking)

1. On an affected node/runner, reproduce with a single pull (not your whole deployment).
2. Look for HTTP 429 and any RateLimit headers.
3. Check if failures correlate with NAT IPs (all nodes egress through one IP? You share a quota).

Second: identify the blast radius and the traffic pattern

1. Is it one image/tag or everything?
2. Is it many nodes simultaneously (cluster scale-up, rolling restart, node replacement)?
3. Is it CI parallelism (50 jobs start at once) or developer desktops (Monday morning)?

Third: pick the correct mitigation class

• Short-term mitigation: slow down pulls (stagger deploy), reuse nodes, pre-pull, increase backoff, reduce concurrency.
• Medium-term: authenticate pulls, pin digests, reduce image size/layers, stop rebuilding identical tags.
• Long-term: add a caching proxy/mirror, run a private registry, replicate critical images, and engineer your CI to reuse caches.

Interesting facts and historical context (the stuff that explains today’s pain)

• Docker Hub rate limiting tightened in 2020, and many “free” workflows that relied on anonymous pulls became brittle overnight.
• Container image distribution borrows from Git and package repos, but unlike apt/yum, images are chunky and pulled in parallel—great for speed, terrible for quotas.
• OCI image specs standardized the format, which improved portability but also made it easy for every tool to hammer the same registries the same way.
• Content-addressed layers mean identical layers are reused across tags and images—if you actually let caches persist. Ephemeral runners throw that advantage away.
• CDNs front most public registries; you might be rate-limited by edge policy even if the origin registry is fine.
• Kubernetes made pull storms normal: a single deployment can trigger hundreds of near-simultaneous pulls during node churn or autoscaling.
• Multi-arch images increased request counts: your client may fetch an index manifest, then architecture-specific manifests, then layers.
• NAT gateways concentrate identity: a thousand nodes behind one egress IP can look like one extremely impatient client.
• Registry auth uses bearer tokens: each pull can include token service requests; those endpoints can be throttled independently.

Practical tasks: commands, outputs, and the decision you make

These are the field tools. Each task includes a command, what you might see, and what you decide next. Run them on a node/runner that’s failing.
Don’t “fix” anything until you’ve done at least the first four.

Task 1: Reproduce a single pull with verbose-ish output

cr0x@server:~$ docker pull nginx:1.25
1.25: Pulling from library/nginx
no matching manifest for linux/amd64 in the manifest list entries

Meaning: This isn’t throttling. It’s an architecture mismatch (common on ARM runners or weird base images).
Decision: Fix the image/tag/platform selection before chasing rate limits.

cr0x@server:~$ docker pull redis:7
7: Pulling from library/redis
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading

Meaning: This is real rate limiting from the registry (classic Docker Hub wording).
Decision: Move immediately to authentication + caching/mirroring; slowing down may buy time but won’t fix the class of problem.

Task 2: Identify your egress IP (are you sharing quota behind NAT?)

cr0x@server:~$ curl -s https://ifconfig.me
203.0.113.42

Meaning: That’s the public IP the registry sees.
Decision: If many nodes/runners show the same IP, assume quota is effectively shared. Plan for a mirror or split egress.

Task 3: Check Docker daemon and container runtime logs for 429 and auth churn

cr0x@server:~$ sudo journalctl -u docker --since "30 min ago" | tail -n 30
Jan 03 10:41:22 node-7 dockerd[1432]: Error response from daemon: toomanyrequests: Rate exceeded
Jan 03 10:41:22 node-7 dockerd[1432]: Attempting next endpoint for pull after error: Get "https://registry-1.docker.io/v2/": too many requests

Meaning: The daemon is getting throttled at the registry endpoint.
Decision: Continue to measure pull concurrency and introduce caching/mirroring.

Task 4: Confirm registry identity and headers (429 vs proxy)

cr0x@server:~$ curl -I -s https://registry-1.docker.io/v2/ | head
HTTP/2 401
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"

Meaning: 401 here is normal; it proves you’re hitting the expected registry and auth flow.
Decision: If you see corporate proxy headers or an unexpected server, you may be throttled or blocked upstream by your proxy/CDN path.

Task 5: Inspect per-node image cache (are you re-pulling because nodes are fresh?)

cr0x@server:~$ docker images --digests | head
REPOSITORY   TAG     DIGEST                                                                    IMAGE ID       CREATED        SIZE
nginx        1.25    sha256:2f7f7d3f2c0a7a6e1f6b0c1a3bcbf5b0e6c2e0d2a3a2e9a0b1c2d3e4f5a6b7c   8c3a9d2f1b2c   2 weeks ago    192MB

Meaning: The digest indicates content addressability; if you pin this digest, you can be more deterministic.
Decision: If nodes don’t have the image, you either need pre-pulls, longer-lived nodes, or a mirror that makes cache hits local.

Task 6: Check Kubernetes events for pull storms and backoff

cr0x@server:~$ kubectl get events -A --sort-by='.lastTimestamp' | tail -n 12
default   8m12s   Warning   Failed     pod/api-7c8d9c6d9c-7lqjv     Failed to pull image "nginx:1.25": toomanyrequests: Rate exceeded
default   8m11s   Normal    BackOff    pod/api-7c8d9c6d9c-7lqjv     Back-off pulling image "nginx:1.25"

Meaning: Kubelet is repeatedly retrying. Retries increase request volume. Volume increases throttling. You see the loop.
Decision: Stop the loop: pause rollouts, reduce replica churn, and put a cache in front of the registry.

Task 7: Quantify how many nodes are pulling simultaneously

cr0x@server:~$ kubectl get pods -A -o wide | awk '$4=="ContainerCreating" || $4=="Pending"{print $1,$2,$4,$7}' | head
default api-7c8d9c6d9c-7lqjv ContainerCreating node-12
default api-7c8d9c6d9c-k3q2m ContainerCreating node-14
default api-7c8d9c6d9c-px9z2 ContainerCreating node-15

Meaning: This is a live pull storm: many pods blocked on image pulls at once across nodes.
Decision: Stagger rollout or scale in, then implement pre-pulling or a DaemonSet cache warm-up, plus mirror/caching.

Task 8: Validate imagePullPolicy isn’t sabotaging you

cr0x@server:~$ kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}{"\n"}'
Always

Meaning: Always guarantees a registry hit even if the image exists locally. That’s fine for “latest” habits; it’s awful under rate limiting.
Decision: If you use immutable tags or digests, change to IfNotPresent and pin images properly.

Task 9: Check if you’re using “latest” (a polite way to say “non-deterministic”)

cr0x@server:~$ kubectl get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
myorg/api:latest

Meaning: You can’t reason about caching when tags float. Every node might legitimately need a fresh pull at the same time.
Decision: Stop using :latest in production. Use a version tag and/or pin by digest.

Task 10: Confirm containerd is the actual runtime (and where to configure mirrors)

cr0x@server:~$ kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}{"\n"}'
containerd://1.7.13

Meaning: You need containerd mirror configuration, not Docker daemon configuration (even if you still say “docker pull” out of habit).
Decision: Configure registry mirrors in containerd and restart it carefully (drain node, restart, uncordon).

Task 11: Inspect containerd registry config for mirrors (common on Kubernetes nodes)

cr0x@server:~$ sudo grep -n "registry" -n /etc/containerd/config.toml | head -n 30
122:[plugins."io.containerd.grpc.v1.cri".registry]
123:  config_path = ""

Meaning: No per-registry mirror config is currently used (or it’s external via config_path).
Decision: Add a mirror endpoint for Docker Hub (or whichever registry) via proper containerd configuration.

Task 12: Validate DNS and TLS quickly (rate limit lookalikes)

cr0x@server:~$ getent hosts registry-1.docker.io
2600:1f18:2148:bc02:6d3a:9d22:6d91:9ef2 registry-1.docker.io
54.85.133.21 registry-1.docker.io

Meaning: DNS resolves. If this fails intermittently, kubelet retries can mimic throttling with a similar operational impact.
Decision: If DNS is flaky, fix DNS first. Otherwise proceed to registry quota/caching.

Task 13: Check for conntrack or ephemeral port pressure during pull storms (self-inflicted throttling)

cr0x@server:~$ sudo conntrack -S | head
entries  18756
searched 421903
found    13320
new      9321
invalid  12
ignore   0
delete   534
delete_list 24
insert   9321
insert_failed 0
drop     0
early_drop 0

Meaning: If insert_failed or drops climb during storms, you’re losing connections locally, causing retries and more load.
Decision: Tune conntrack, reduce concurrency, or fix node sizing. Don’t blame the registry until your own house is in order.

Task 14: See if your CI runners are caching anything at all

cr0x@server:~$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          3         1         1.2GB     1.1GB (90%)
Containers      1         0         12MB      12MB (100%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Meaning: Your runner is basically amnesiac. Build cache is zero; images are mostly reclaimable. That’s a recipe for repeated pulls.
Decision: Use persistent runners, shared cache (BuildKit), or pre-populated images. Or accept that you need a local proxy cache.

Task 15: Pin by digest and test a pull (reduces tag churn and surprises)

cr0x@server:~$ docker pull nginx@sha256:2f7f7d3f2c0a7a6e1f6b0c1a3bcbf5b0e6c2e0d2a3a2e9a0b1c2d3e4f5a6b7c
sha256:2f7f7d3f2c0a7a6e1f6b0c1a3bcbf5b0e6c2e0d2a3a2e9a0b1c2d3e4f5a6b7c: Pulling from library/nginx
Digest: sha256:2f7f7d3f2c0a7a6e1f6b0c1a3bcbf5b0e6c2e0d2a3a2e9a0b1c2d3e4f5a6b7c
Status: Image is up to date for nginx@sha256:2f7f7d3f2c0a7a6e1f6b0c1a3bcbf5b0e6c2e0d2a3a2e9a0b1c2d3e4f5a6b7c

Meaning: Digest pins make caching and rollouts predictable. If the image exists locally, the runtime can skip downloading layers.
Decision: For production, prefer digest pinning (or at least immutable version tags) and align pull policy accordingly.

Fixes that work (and why)

1) Authenticate pulls (yes, even for public images)

Anonymous pulls are treated like a public utility. Public utilities get metered. If your production depends on anonymous pulls, your production depends on “someone else’s generosity.”
That’s not a strategy; that’s a vibe.

Authentication can raise rate limits and improves attribution. It also makes it easier to reason about who is pulling what.
On Kubernetes, this often means imagePullSecrets. In CI, it means docker login with a token and making sure jobs don’t share a single throttled identity.

2) Stop pulling the same thing 500 times: add a caching mirror

A registry mirror/proxy cache is the grown-up solution. Your cluster should pull from something you control (or at least something closer),
which then pulls from the public registry once and serves many.

Options include:

• Docker Registry in proxy cache mode: simple, works, but you must operate it (storage, HA, backups).
• Harbor proxy cache: heavier, but good enterprise controls (projects, RBAC, replication).
• Cloud provider artifact registries with pull-through caching or replication patterns (varies by provider; check limits).

The mirror should sit close to your nodes (same region/VPC) to reduce latency and bandwidth. It should use fast storage and handle concurrent layer downloads.
And it should have enough disk. Nothing says “professional operations” like a cache that evicts the hot layers every hour.

3) Pin by digest, and make pull policy match reality

If you pin by digest and keep imagePullPolicy: IfNotPresent, you get the best of both: deterministic content and fewer registry hits.
If you keep floating tags and Always, you are choosing to hit the registry. That may be acceptable for a small dev cluster. It’s reckless at scale.

4) Pre-pull images deliberately (warm caches)

If a cluster is about to scale, or you know a rollout will touch every node, pre-pull the image once per node before you flip traffic.
The pattern is old-school, boring, and extremely effective.

Kubernetes approach: a DaemonSet that pulls the image (and maybe does nothing else) so the nodes cache it. Then deploy the real workload.
This spreads pulls over time and makes failures visible before the rollout.

5) Reduce concurrency where it hurts

You can absolutely throttle yourself less by being less enthusiastic. Reduce parallel jobs in CI that all pull the same base image.
Stagger deploy waves. Avoid cluster-wide restarts during business hours unless you enjoy attention.

6) Make images smaller and more layer-reusable

Rate limiting is about request counts, but bandwidth and time still matter. Smaller images mean faster pulls, fewer concurrent connections, fewer retries, and less time spent in the danger zone.
Also: fewer layers can reduce total requests, but don’t chase “one layer” at the cost of cacheability. Good layering is still a skill.

7) Control egress identity (split NAT, don’t concentrate all pulls)

If your entire fleet egresses through one NAT IP, you’ve made one IP responsible for the world’s worst moment: your scale-up event.
Consider multiple egress IPs, per-node public egress (with care), or private connectivity to your registry/mirror when possible.

Joke #2: NAT gateways are like office coffee machines—everyone shares them until Monday morning proves that was a mistake.

Kubernetes-specific failure modes (because kubelet never pulls “a little”)

Kubernetes turns “pull an image” into a distributed, concurrent activity. It’s efficient when the registry is tolerant and caches are warm.
It’s a disaster when you have node churn, cold nodes, and an external registry with strict quotas.

ImagePullBackOff is a multiplier

Backoff is meant to reduce load, but in large clusters it becomes a synchronization mechanism: many nodes fail, then many nodes retry around the same time,
especially after network hiccups or after a registry recovers. The result is a thundering herd.

Node churn creates cold caches

Autoscaling, spot instances, and aggressive node recycling are all fine—until you realize you’re constantly creating new machines with empty caches.
If your registry is throttled, cold caches are not a minor inconvenience; they are an outage trigger.

Pull policy and tag discipline matter more than you think

Kubernetes defaults imagePullPolicy to Always when you use :latest.
That is Kubernetes politely telling you not to use :latest if you care about stability. Listen to it.

containerd vs Docker: configure the right thing

Many teams still “fix Docker” on Kubernetes nodes that run containerd. The fix never lands because it’s applied to a service that isn’t in the path.
Identify the runtime first, then configure its registry mirror support properly.

CI/CD: why your runners are the worst pullers

CI systems are optimized for throughput and disposability. That’s great for security and reproducibility.
It’s also great for repeatedly pulling the same base image until the registry tells you to go away.

Ephemeral runners throw away the two biggest advantages you have

• Layer cache: content-addressed layers only help if you keep them.
• Build cache: BuildKit can avoid re-downloading and re-building, but not if every job starts from an empty disk.

Parallelism is not free

CI vendors and teams love parallelism. Registries do not. If you need 50 parallel jobs, give them a shared mirror in your network and authenticate.
Otherwise you’re just paying to discover rate limits faster.

Tag discipline reduces pointless pulls

Reusing the same tag for different content (“we overwrote dev again”) forces clients to re-check and re-pull.
Use unique tags per build and keep a moving “human” tag only as a pointer.

Three corporate mini-stories from the trenches

Incident: the wrong assumption (“public images are basically free”)

A mid-sized SaaS company ran a Kubernetes cluster that autoscaled aggressively. The team used mostly public base images—common ones, well-known ones—
and assumed the internet would handle it. They had an internal registry, but only for their own app images. Everything else came straight from the public registry.

A busy weekday morning, nodes began recycling faster than usual due to a separate kernel update rollout combined with spot instance churn.
New nodes joined with empty caches. Kubelet did what kubelet does: pulled everything. At once. Across many nodes.

Within minutes, pods piled up in ContainerCreating and then fell into ImagePullBackOff.
The on-call engineer saw “too many requests” and assumed it was a transient blip. They restarted a few nodes—creating even more cold caches and more pulls.
The graph of failed pulls looked like a staircase to regret.

The root cause wasn’t “Docker being flaky.” It was a wrong assumption: that public registry capacity and quotas are aligned with their scaling behavior.
The fix was simple but not quick: authenticate, introduce a proxy cache for the public registry, and pre-pull critical images during node provisioning.
After that, node churn was still annoying, but it stopped being an outage trigger.

Optimization that backfired: “let’s purge caches to save disk”

Another company prided itself on lean nodes. They ran a job on every node to prune images aggressively. Disk graphs were immaculate.
The monthly infra review slides looked gorgeous: “We reduced wasted storage by 40%.” Everyone nodded.

Then they introduced a canary strategy that rolled pods frequently across the fleet. Each rollout caused a wave of pulls.
But because the prune job had deleted most layers, every node behaved like it was brand new.

The registry throttled them intermittently. When throttling hit, pods failed readiness, the canary held, automation retried, and the whole pipeline extended its own suffering.
The really fun part: the incident was intermittent, and therefore perfectly calibrated to waste human time.

The “optimization” saved disk and spent reliability. The fix was to stop treating image cache as garbage.
They set sensible disk thresholds, kept a baseline of hot images pinned on nodes, and moved aggressive cleanup to non-peak windows.
Storage is cheaper than downtime, and also cheaper than people.

Boring but correct practice that saved the day: “pre-pull and pin”

A financial services team ran regulated workloads with strict change control. Their release process was slow, which annoyed developers,
but it had one habit that paid rent: every release candidate was pinned by digest and pre-pulled across the cluster before traffic shifted.

One evening, a public registry started throttling in their region due to an upstream event. Other teams panicked and rolled back.
This team barely noticed. Their nodes already had the required layers locally, and the deployment used IfNotPresent.

They still saw errors in background pulls for unrelated images, but production rollout was unaffected.
The on-call’s incident report was almost embarrassing: “No customer impact. Observed external throttling. Continued as planned.”
That’s the dream: the outside world can be on fire, and your system shrugs.

The lesson is not “be slow like finance.” The lesson is: the boring practices—pinning, pre-pulling, controlled rollouts—create slack.
Slack is what keeps external dependencies from becoming outages.

Common mistakes: symptom → root cause → fix

1) Symptom: “toomanyrequests” only during deploys

Root cause: pull storms from simultaneous rollouts, scale-ups, or node replacements.

Fix: stagger rollouts, pre-pull via DaemonSet, add a caching mirror, and stop using floating tags.

2) Symptom: works on laptops, fails in CI

Root cause: developer machines have warm caches; CI runners are ephemeral and start cold every time.

Fix: persistent runners or shared cache, authenticate pulls, introduce a proxy cache inside your network.

3) Symptom: random 403/401 errors under load

Root cause: token service throttling or misinterpreted auth failures caused by rate limiting.

Fix: authenticate properly, avoid token-sharing across massive concurrency, and inspect headers/logs to confirm 429 vs auth remembering.

4) Symptom: only one cluster/region is affected

Root cause: a specific egress IP is hot, or a regional CDN PoP is applying stricter policy.

Fix: split egress/NAT, deploy a regional cache/mirror, or replicate critical images into a closer registry.

5) Symptom: “we have a mirror” but pulls still hit Docker Hub

Root cause: mirror configured for Docker daemon but nodes use containerd; or mirror hostname not trusted; or only some nodes updated.

Fix: confirm runtime, configure mirror at the correct layer, roll it out with node draining, and test with a controlled pull.

6) Symptom: throttling got worse after “cleanup” changes

Root cause: aggressive pruning deleted hot layers; every rollout became a cold-start.

Fix: keep baseline images, tune garbage collection thresholds, and align cleanup with actual risk (disk pressure), not aesthetics.

7) Symptom: pull failures look like throttling but no 429

Root cause: DNS flaps, MTU/TLS interception, proxy resets, conntrack exhaustion, or local network saturation.

Fix: verify DNS/TLS, watch conntrack/ports, check proxy logs, and reduce concurrent pulls while you fix network fundamentals.

Checklists / step-by-step plan

Phase 0: stabilize production (today)

1. Pause the thundering herd: stop/slow rollouts; reduce replicas temporarily if safe.
2. Authenticate pulls for the affected systems immediately (CI and cluster nodes where possible).
3. Pin the release artifact (tag or digest) so retries aren’t pulling a moving target.
4. Reduce concurrency: CI job parallelism, rollout maxUnavailable/maxSurge, autoscaler aggressiveness.
5. Pick one test node and verify a clean pull path before reattempting the rollout.

Phase 1: stop depending on public registry behavior (this week)

1. Deploy a caching registry mirror near the cluster/runners.
2. Configure containerd/Docker to use the mirror (and confirm it’s actually used).
3. Pre-pull critical images on nodes (DaemonSet warm-up or node bootstrap).
4. Fix tag/pull policy: stop using :latest, use IfNotPresent with immutable tags/digests.
5. Make failures observable: alert on ImagePullBackOff rates and registry 429s in logs.

Phase 2: make it boring and resilient (this quarter)

1. Replicate dependencies: mirror/replicate third-party images you rely on into your own registry.
2. Adopt a base-image strategy: fewer base images, standardized, patched regularly, stored internally.
3. Capacity plan the cache: disk, IOPS, concurrency, and HA requirements; treat it like production.
4. Govern concurrency: cluster rollout policies, CI concurrency controls, and scale-up guardrails.
5. Run game days: simulate registry failures/throttles; ensure your system degrades gracefully.

FAQ

1) Is this only a Docker Hub problem?

No. Any registry can throttle: public registries, cloud registries, and your own registry behind a load balancer.
Docker Hub is just the most famous place to get a 429 and a life lesson.

2) Why does rate limiting hit us “randomly”?

Because your traffic is bursty. Deploys, autoscaling, and CI fan-out create spikes.
Quotas are often enforced per window, per IP, or per token; once you cross the line, everyone sharing that identity suffers.

3) If we authenticate, are we done?

Authentication helps, but it doesn’t eliminate the architectural issue. You can still exceed authenticated quotas, and you can still melt your NAT/proxy.
Use auth as table stakes, not as the entire plan.

4) What’s the single best long-term fix?

A caching mirror/proxy close to your compute. It converts “internet dependency” into “local dependency,” and local dependencies are at least your problem to solve.

5) Should we pin by digest everywhere?

For production deployments, yes when feasible. Digests make rollouts deterministic and align nicely with IfNotPresent.
For dev workflows, immutable version tags may be enough, but “latest” is still a trap.

6) We already use IfNotPresent. Why are we still pulling?

Because the image isn’t present on that node (new node, pruned cache, different architecture), or because the tag points to new content and the runtime checks anyway.
Verify local cache presence and stop reusing tags for different builds.

7) Can we just increase Kubernetes backoff and be fine?

Backoff reduces immediate pressure but doesn’t solve the underlying demand. Also, synchronized backoff across many nodes can create waves of retries.
Use backoff tuning only as a stabilizer while you implement caching and policy fixes.

8) What about air-gapped or regulated environments?

You’ll end up running your own registry and curating images internally. The upside: no external throttling.
The downside: you must own patching, scanning, and availability. It’s still the right move for many regulated shops.

9) How do we know our mirror is actually being used?

Check runtime config on the node, then observe mirror logs/metrics during a pull. Also compare DNS lookups and outbound connections:
if nodes still talk to the public registry directly, your mirror is ornamental.

10) Could the bottleneck be our storage?

Yes—especially for self-hosted caches. If your proxy cache sits on slow disk, it will serialize layer reads and make pulls slow,
causing more concurrent pull attempts and more retries. Fast storage and concurrency matter.

Conclusion: practical next steps

Registry throttling is not a freak accident. It’s the expected outcome of modern elastic infrastructure hitting a shared external service.
Your job isn’t to hope it won’t happen again. Your job is to make it irrelevant.

1. Today: confirm 429 vs network issues, stop the pull storm, authenticate, pin the release artifact.
2. This week: deploy a caching mirror near compute, configure the real runtime (containerd/Docker), and pre-pull critical images.
3. This quarter: internalize third-party dependencies, standardize base images, and govern concurrency so “autoscaling” doesn’t mean “auto-outage.”

Make image delivery boring. Boring is what you want at 3 a.m.