Docker Containers Filling Disk: tmp/log/cache Cleanup That Won’t Burn You

The mental model: where “container disk” actually lives

When someone says “the container filesystem is full,” what they usually mean is: a writable layer associated with a container,
stored on the host under Docker’s storage driver (commonly overlay2), ran out of space on the underlying host filesystem.
That’s the default for a container’s root filesystem changes. It’s not magic. It’s a directory tree and some mount tricks.

Containers aren’t VMs. They don’t come with their own block device unless you explicitly give them one via volumes, mounts, or CSI.
If you don’t set storage budgets, a container can happily eat the host disk. It’s polite like that.

Interesting facts & historical context (because outages have ancestors)

• Union filesystems predate Docker: overlay-style stacking came from the Linux world’s long attempt to make immutable bases with writable tops practical.
• Docker’s early storage drivers were a zoo: aufs, devicemapper, btrfs, overlay, overlay2—each with different failure modes and cleanup behavior.
• json-file logging became the “default trap”: it was simple and worked everywhere, so it quietly became many people’s disk leak.
• Log rotation is older than containers: syslog and logrotate exist because disk fills are a classic Unix rite of passage.
• Build caches exploded with modern CI: once multi-stage builds and layer caching got popular, build caches became production disk residents.
• Container runtimes didn’t invent ephemeral storage: tmpfs has been around forever; we just keep forgetting to use it for genuinely temporary data.
• Kubernetes elevated “ephemeral storage” into a schedulable resource: mostly because enough nodes died from log and emptyDir growth.
• OverlayFS semantics matter: deleting a file inside a container doesn’t always “free space” if something is still holding a handle (classic log file issue).

One quote worth taping to your monitor:
Hope is not a strategy. — commonly attributed in ops circles; paraphrased idea from reliability practice.
Disk management is not the place for vibes.

Joke #1: Disk is the only resource that starts infinite and then abruptly becomes zero at 03:00.

Fast diagnosis playbook

This is the sequence I use when someone pings “container disk full” and expects me to be a wizard.
It’s not wizardry. It’s a checklist with strong opinions.

First: confirm what’s actually full

1. Host filesystem? If the host partition backing Docker is full, everything is on fire, even if only one container is loud.
2. Docker storage area? /var/lib/docker (or your configured data-root) often sits on the wrong partition.
3. A volume? Volumes can fill independently and won’t show up as “container layer” usage.
4. Inode exhaustion? You can have plenty of bytes and zero inodes. Small files are a great way to ruin your weekend.

Second: find the category of growth

1. Logs (Docker logs, app logs, access logs, debug dumps).
2. Tmp (uploads, extracted archives, OCR temp files, image processing scratch space).
3. Caches (package caches, dependency caches, language runtime caches, browser caches).
4. Build artifacts (CI leftovers, build cache, layers).
5. Data accidentally written to the container writable layer instead of a volume.

Third: choose the least risky relief valve

1. Rotate / cap logs (preferably without restart).
2. Delete safe temp directories (with clear ownership and age rules).
3. Prune unused Docker artifacts (with careful targeting).
4. Move the hot path to a volume or tmpfs (actual fix).

The bottleneck isn’t “disk.” The bottleneck is the absence of a budget and an owner.
Your goal: identify the owner and enforce the budget.

Practical tasks: commands, outputs, and decisions (12+)

These are real tasks I do under pressure. Each includes: command, what the output means, and what decision you make next.
Run them on the host unless noted.

Task 1: Is the host filesystem actually full?

cr0x@server:~$ df -hT
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda2      ext4   100G   96G  2.0G  98% /
tmpfs          tmpfs  3.1G  4.0M  3.1G   1% /run
/dev/sdb1      ext4   500G  120G  355G  26% /data

Meaning: / is 98% full. If Docker’s data-root is on /, you’re close to a full stop.

Decision: Confirm where Docker stores data. If it’s on /, prioritize immediate cleanup and plan a migration to /data.

Task 2: Where is Docker storing its data?

cr0x@server:~$ docker info --format 'DockerRootDir={{.DockerRootDir}} Driver={{.Driver}} LoggingDriver={{.LoggingDriver}}'
DockerRootDir=/var/lib/docker Driver=overlay2 LoggingDriver=json-file

Meaning: Docker’s root is on /var/lib/docker (on /) using overlay2. Logs are json-file.

Decision: Expect overlay2 directories and json logs to be primary suspects. Don’t “just prune everything” yet.

Task 3: Quick view of Docker’s disk usage breakdown

cr0x@server:~$ docker system df -v
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          24        12        18.4GB    6.2GB (33%)
Containers      36        18        4.8GB     1.1GB (22%)
Local Volumes   19        15        220.5GB   12.0GB (5%)
Build Cache     112       0         41.7GB    41.7GB

Meaning: Volumes dominate (220GB). Build cache is large and fully reclaimable. Containers add ~5GB.

Decision: If you need fast relief, build cache is a low-risk target. But also audit volumes—someone is storing real data.

Task 4: Find the biggest directories under /var/lib/docker

cr0x@server:~$ sudo du -xhd1 /var/lib/docker | sort -h
12G	/var/lib/docker/containers
41G	/var/lib/docker/buildkit
67G	/var/lib/docker/overlay2
220G	/var/lib/docker/volumes
340G	/var/lib/docker

Meaning: Volumes and overlay2 are heavy; buildkit is substantial.

Decision: Identify which containers map to those volumes. Do not delete volume data unless you know the application owner and retention policy.

Task 5: Which containers are generating huge Docker logs?

cr0x@server:~$ sudo find /var/lib/docker/containers -name '*-json.log' -printf '%s %p\n' | sort -n | tail -5
187654321 /var/lib/docker/containers/4c9f.../4c9f...-json.log
322198765 /var/lib/docker/containers/8a21.../8a21...-json.log
988877766 /var/lib/docker/containers/1b77.../1b77...-json.log
2147483648 /var/lib/docker/containers/aa11.../aa11...-json.log
4123456789 /var/lib/docker/containers/f00d.../f00d...-json.log

Meaning: You have multi-GB json logs. That’s disk pressure with a name and an address.

Decision: Identify the container names for those IDs, then cap/rotate logs. Do not delete the files blindly if the daemon is still writing.

Task 6: Map a container ID to a name and image

cr0x@server:~$ docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Image}}' | grep f00d
f00dbeefcafe1234567890abcdef1234567890abcdef1234567890abcdef api-prod myorg/api:2026.01

Meaning: The API container is generating huge logs.

Decision: Treat logs as a symptom. You still need to fix the noisy logging level or error loop.

Task 7: Check container writable layer size (smoke test)

cr0x@server:~$ docker ps -q | head -5 | xargs -I{} docker container inspect --format '{{.Name}} rw={{.SizeRw}} rootfs={{.SizeRootFs}}' {}
/api-prod rw=2147483648 rootfs=0
/worker-prod rw=536870912 rootfs=0
/nginx-prod rw=104857600 rootfs=0
/cron-prod rw=0 rootfs=0
/metrics rw=0 rootfs=0

Meaning: SizeRw (writable layer) for api-prod is ~2GB. That’s usually logs, tmp, or accidental data writes.

Decision: Exec into that container to find where the space lives, or check overlay2 diff directory mapping.

Task 8: Inside the container, find top disk consumers

cr0x@server:~$ docker exec -it api-prod sh -lc 'df -h; du -xhd1 / | sort -h | tail -10'
Filesystem                Size      Used Available Use% Mounted on
overlay                  100G       98G      2.0G  98% /
tmpfs                     64M         0       64M   0% /dev
tmpfs                    3.1G         0      3.1G   0% /sys/fs/cgroup
1.2G	/var
1.5G	/usr
2.8G	/tmp
6.0G	/app
9.1G	/log

Meaning: /tmp and /log are big. That’s actionable.

Decision: Determine whether those directories should be tmpfs, volume-backed, or aggressively cleaned.

Task 9: Find large individual files (often core dumps, trace dumps, or stuck uploads)

cr0x@server:~$ docker exec -it api-prod sh -lc 'find /tmp /log -type f -size +200M -printf "%s %p\n" | sort -n | tail -10'
2147483648 /log/app.log
536870912 /tmp/upload-20260103-0130.tmp
402653184 /tmp/image-cache.bin

Meaning: A single log is 2GB; a temp upload file is 512MB. This isn’t subtle.

Decision: Cap logs (rotate). For temp uploads, ensure cleanup on success/failure and consider tmpfs or a dedicated volume with quotas.

Task 10: Check if “deleted files” are still taking space due to open handles

cr0x@server:~$ sudo lsof +L1 | grep '/var/lib/docker/overlay2' | head
node    24781 root  21w  REG  8,2 2147483648  0 /var/lib/docker/overlay2/7a2.../diff/log/app.log (deleted)

Meaning: The file is deleted but still open. Space will not be freed until the process restarts or closes the file descriptor.

Decision: Restart the container (or reload the process) after making sure it will come back cleanly. Also fix log rotation so this doesn’t repeat.

Task 11: Prune build cache safely (usually low-risk)

cr0x@server:~$ docker builder prune --filter 'until=168h'
WARNING! This will remove all dangling build cache.
Are you sure you want to continue? [y/N] y
Deleted build cache objects:
r1m2n3...
freed: 38.6GB

Meaning: You reclaimed ~39GB without touching running containers.

Decision: If this was your top consumer, schedule this prune in CI runners or on builder nodes with a retention policy.

Task 12: Prune unused images (with eyes open)

cr0x@server:~$ docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:1a2b...
Total reclaimed space: 5.9GB

Meaning: Old images were taking space; you reclaimed ~6GB.

Decision: On prod nodes, pruning images can slow redeploys and break rollbacks if you’re not careful. Prefer running this on CI or nodes with controlled rollout strategy.

Task 13: Identify large volumes and map them to containers

cr0x@server:~$ docker volume ls -q | xargs -I{} sh -lc 'p=$(docker volume inspect -f "{{.Mountpoint}}" {}); s=$(sudo du -sh "$p" 2>/dev/null | awk "{print \$1}"); echo "$s {} $p"' | sort -h | tail -5
18G pgdata /var/lib/docker/volumes/pgdata/_data
22G redisdata /var/lib/docker/volumes/redisdata/_data
41G uploads /var/lib/docker/volumes/uploads/_data
55G elastic /var/lib/docker/volumes/elastic/_data
62G app-cache /var/lib/docker/volumes/app-cache/_data

Meaning: There’s a 62GB “app-cache” volume. That might be legitimate, or it might be unbounded cache growth.

Decision: Inspect which containers mount it, then set eviction/retention. Cache is not data; treat it as disposable with limits.

Task 14: Which containers mount that volume?

cr0x@server:~$ docker ps --format '{{.Names}}' | xargs -I{} sh -lc 'docker inspect -f "{{.Name}} {{range .Mounts}}{{.Name}}:{{.Destination}} {{end}}" {}' | grep app-cache
/api-prod app-cache:/app/cache
/worker-prod app-cache:/app/cache

Meaning: Two services share the cache volume. Shared caches are convenient and also excellent at becoming landfill.

Decision: Implement cache size limits in the application and schedule cleanup. If the cache is rebuildable, consider per-pod caches or tmpfs.

Task 15: Check inode exhaustion (the “lots of tiny files” version of full disk)

cr0x@server:~$ df -hi
Filesystem    Inodes IUsed IFree IUse% Mounted on
/dev/sda2       6.4M  6.4M   12K  100% /

Meaning: You ran out of inodes, not bytes. Common causes: unbounded small log files, cache directories with millions of entries, or temp file storms.

Decision: Identify the directory with the most files and clean that. Consider filesystem tuning or moving DockerRootDir to a filesystem with more inodes.

Task 16: Find directories with crazy file counts

cr0x@server:~$ sudo sh -lc 'for d in /var/lib/docker /var/log /tmp; do echo "## $d"; find "$d" -xdev -type f 2>/dev/null | wc -l; done'
## /var/lib/docker
1892345
## /var/log
45678
## /tmp
1203

Meaning: Most files are under Docker’s area. Overlay2 and container logs can explode inode usage.

Decision: Narrow down within /var/lib/docker, starting with overlay2 and containers. If it’s build cache, prune it.

/tmp and temporary files: cleanup strategies that don’t delete the wrong thing

Temporary space is where good intentions go to ferment. In containers, /tmp is often on the writable layer, which means:
it competes with everything else for the same host disk and it survives longer than you think (container restarts don’t necessarily clean it).

Strategy: decide what “temporary” means for your workload

• Truly ephemeral scratch (image processing, unzip, sort, small buffering): use tmpfs with a size cap.
• Large temp for uploads (multi-GB): use a dedicated volume or host path with quotas; don’t pretend tmpfs will save you.
• Work queues and spill files: make them explicit and observable; set TTL and max size.

Mount /tmp as tmpfs (good default for many web apps)

In Docker run:

cr0x@server:~$ docker run --rm -it --tmpfs /tmp:rw,noexec,nosuid,size=256m alpine sh -lc 'df -h /tmp'
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   256.0M         0    256.0M   0% /tmp

Meaning: /tmp is now RAM-backed and capped at 256MB.

Decision: If your app uses /tmp for small scratch files, this prevents disk leaks and makes failures loud (ENOMEM-like behavior) instead of silently filling the host.

For Compose:

cr0x@server:~$ cat docker-compose.yml
services:
  api:
    image: myorg/api:2026.01
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=256m

Cleanup rule: don’t “rm -rf /tmp” in production like a cartoon villain

The safe approach is age-based deletion in a directory your app owns, not global deletion.
Make your app write to /tmp/myapp, then clean only that subtree.

cr0x@server:~$ docker exec -it api-prod sh -lc 'mkdir -p /tmp/myapp; find /tmp/myapp -type f -mmin +120 -delete; echo "cleanup done"; du -sh /tmp/myapp'
cleanup done
12M	/tmp/myapp

Meaning: Two-hour-old files removed; directory is now 12MB.

Decision: If this helps repeatedly, your app is leaking temp files on failure paths. Fix the code, but keep the broom.

Joke #2: Nothing is more permanent than a temporary directory with no owner.

Logs: Docker json-file, journald, and app logs

Logs are the #1 reason “containers filled the disk” shows up on incident timelines.
Not because logging is bad—because logging without limits is a slow-motion denial-of-service attack against your own disk.

Docker json-file: cap it or it will capsize you

Default Docker logging (json-file) writes per-container logs under /var/lib/docker/containers/<id>/<id>-json.log.
If you don’t set rotation, they grow forever. Forever is longer than your disk.

Set log rotation in /etc/docker/daemon.json

cr0x@server:~$ sudo cat /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}

Meaning: Each container’s Docker-managed log is capped at ~250MB (5 files × 50MB).

Decision: Apply this on every host. Then restart Docker in a maintenance window. If you can’t restart, plan a rollout; don’t ignore it.

What about journald?

If Docker logs to journald, growth moves to the system journal. Good news: journald supports size limits centrally.
Bad news: your disk still fills, just with different-shaped files.

cr0x@server:~$ sudo journalctl --disk-usage
Archived and active journals take up 7.8G in the file system.

Meaning: Journals are 7.8GB. Not catastrophic, but not free.

Decision: Set retention (SystemMaxUse, SystemMaxFileSize) if this is creeping upward.

App logs inside the container: decide where they belong

You generally want apps to log to stdout/stderr and let the platform handle log shipping. When apps write to /log
inside the container filesystem, you’ve created a second log system with different retention rules and a stronger tendency to fill disks quietly.

If you must write files (compliance, batch workloads, legacy apps), put them on a volume with explicit retention and rotation.
And test rotation: log rotation is a change that can break things.

Truncate safely when the file is too big (emergency only)

cr0x@server:~$ sudo sh -lc ': > /var/lib/docker/containers/f00dbeefcafe*/f00dbeefcafe*-json.log; ls -lh /var/lib/docker/containers/f00dbeefcafe*/f00dbeefcafe*-json.log'
-rw-r----- 1 root root 0 Jan  3 02:11 /var/lib/docker/containers/f00dbeefcafe123.../f00dbeefcafe123...-json.log

Meaning: You truncated the file in place. Docker can continue writing without needing to reopen a new inode.

Decision: Do this only as a stopgap during an incident. Your real fix is log rotation + reducing log volume at the source.

Caches: package managers, language runtimes, and build artifacts

Cache is a performance feature that turns into a storage bug when nobody owns it. Containers amplify this because the same cache patterns that are fine on a dev laptop
become unbounded growth on a node with dozens of services.

Common cache hotspots

• OS package managers: apt, apk, yum caches left in images or created at runtime.
• Language runtimes: pip cache, npm cache, Maven/Gradle, Ruby gems, Go build cache.
• Headless browser caches: Chromium cache directories can balloon.
• Application-level caches: thumbnail caches, compiled templates, query caches, “temporary” exports.

Inside a running container: find cache directories

cr0x@server:~$ docker exec -it worker-prod sh -lc 'du -xhd1 /root /var/cache /app | sort -h | tail -15'
12M	/var/cache
420M	/root
2.3G	/app

Meaning: Something under /app is large; root’s home is 420MB (often runtime caches).

Decision: Inspect /app subtree for cache directories and decide: should it be a volume, should it have a size cap, or should it be deleted on start?

Prevent caches from landing in the writable layer

If the cache is disposable and local, mount a volume dedicated to cache. That way it doesn’t bloat the container layer and is easier to clean.
If you need hard limits, consider filesystem quotas on that volume path at the host level (or use storage classes in Kubernetes).

Also: don’t build images that carry caches. If you’re using Dockerfiles, clean package caches in the same layer you install packages,
otherwise the layer still contains the old data. This is not a moral judgment; it’s how layered filesystems work.

Docker artifacts: images, layers, overlay2, build cache

Docker creates and retains a lot of stuff: images, layers, stopped containers, networks, volumes, build cache.
The retention rules are intentionally conservative because deleting the wrong thing breaks workflows. Your job is to tune those rules for your environment.

Overlay2 growth: the hidden cost of “writing to root”

Overlay2 stores per-container writable changes in directories under /var/lib/docker/overlay2.
When an application writes to /var/lib/postgresql or /uploads without a volume, it writes into overlay2.
It works until the node dies of success.

Find which containers have large writable layers

cr0x@server:~$ docker ps -q | xargs -I{} docker inspect --format '{{printf "%-25s %-12s\n" .Name .SizeRw}}' {} | sort -k2 -n | tail
/api-prod                 2147483648
/worker-prod              536870912
/nginx-prod               104857600

Meaning: These containers are writing significant data to the writable layer.

Decision: For each, decide if the writes should go to stdout, tmpfs, or a named volume. “Keep it in the container” is not a storage strategy.

Pruning: useful, dangerous, and sometimes both

docker system prune is the chainsaw. It’s handy when the forest is on fire, but you can also amputate your ability to roll back or rebuild quickly.
In production, prefer targeted prunes:

• Build cache prune on builder nodes and CI runners.
• Image prune with awareness of deployment strategy and rollback needs.
• Container prune only if you accumulate stopped containers for no reason.
• Volume prune only when you have strong guarantees you’re not deleting live data.

cr0x@server:~$ docker container prune
WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
3d2c...
Total reclaimed space: 1.0GB

Meaning: Stopped containers were consuming space.

Decision: If you regularly see many stopped containers, fix your deployment process or your crash loops. Cleanup is not the root cause fix.

Volumes and bind mounts: the “not actually in the container” disk

Volumes are where durable data should live. They’re also where caches go to become “accidentally durable.”
The failure mode is predictable: the application silently grows a directory, the node’s disk fills, and everyone blames Docker.

Audit volumes like you audit databases

Volume usage is operationally significant. Put alerts on it. Put owners on it. If it’s a cache, put eviction and a maximum size on it.
If it’s durable data, put backups and retention on it.

Bind mounts: convenient and sharp

Bind mounts can bypass Docker’s managed storage and write directly to the host filesystem. That can be good (separating data from DockerRootDir)
or terrible (writing into / on a small root partition).

cr0x@server:~$ docker inspect api-prod --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}{{"\n"}}{{end}}'
volume /var/lib/docker/volumes/app-cache/_data -> /app/cache
bind /data/uploads -> /uploads

Meaning: Uploads are bind-mounted to /data/uploads. Great—if /data is large and monitored.

Decision: Confirm the host path has quotas/alerts. Bind mounts are effectively “write to the host,” so treat them as such.

Kubernetes notes: ephemeral storage and DiskPressure

In Kubernetes, disk issues usually present as DiskPressure, pod evictions, and nodes going NotReady.
The underlying causes are the same: logs, emptyDir, container writable layers, image cache, and volumes.
Kubernetes just adds a scheduler and a louder failure mode.

Key operational point: request and limit ephemeral storage

If you don’t set ephemeral storage requests/limits, the kubelet can’t make smart scheduling decisions and will resort to evicting pods when the node is already hurting.
That’s not capacity planning; that’s panic management.

emptyDir is not a trash can without a lid

emptyDir without a sizeLimit (and without monitoring) is a blank check.
If the workload needs scratch space, set a budget. If it needs durable storage, use a PVC.

cr0x@server:~$ kubectl describe node worker-3 | sed -n '/Conditions:/,/Addresses:/p'
Conditions:
  Type             Status  LastHeartbeatTime                 Reason              Message
  DiskPressure     True    Fri, 03 Jan 2026 02:19:54 +0000   KubeletHasDiskPressure   kubelet has disk pressure

Meaning: The node is under disk pressure and will start evicting.

Decision: Immediately identify the top consumers (container logs, images, emptyDir, writable layers) and relieve pressure. Then add limits and alerts.

Three corporate mini-stories (realistic, painful, useful)

1) Incident caused by a wrong assumption: “Containers are ephemeral, so they can’t fill disk”

A mid-size company ran a payments-adjacent API on a handful of Docker hosts. The app was stateless, so everyone assumed storage was “not a thing.”
Deployments were blue/green-ish: new containers came up, old ones got stopped, and nobody paid attention to the stopped ones because “they’re not running.”

Over a few months, the root filesystem on the hosts quietly climbed from comfortable to precarious. The actual culprit was boring:
Docker json logs on a few chatty containers, plus dozens of stopped containers retained for “debugging later.”
The logs were never rotated. The stopped containers had writable layers full of temp files created during request spikes.

The outage happened during a routine deploy. Docker tried to pull a new image layer. The pull failed with no space left on device.
The orchestration logic kept trying, creating more partial downloads and more error logs. Now the disk wasn’t just full; it was actively being attacked by retries.

Recovery was an incident commander’s favorite kind of comedy: the fastest fix was to stop the retry loop and prune build cache and old images.
But the team initially tried to delete log files inside containers, not realizing Docker was writing elsewhere. They reclaimed almost nothing and burned time.
Once they rotated Docker logs and pruned stopped containers, the node recovered.

The root cause wasn’t “Docker.” It was the assumption that ephemeral compute implies ephemeral storage usage.
Containers are ephemeral; the files they create on the host are very committed to the relationship.

2) Optimization that backfired: “Let’s cache everything on a shared volume”

Another organization had expensive requests: PDF generation plus image rendering. Someone made a sensible suggestion:
cache intermediate assets and rendered outputs so repeated requests are faster. They implemented it as a shared Docker volume mounted at /app/cache
by both the API and worker containers.

Performance improved immediately. Latency got better, CPU dropped, and the team congratulated themselves for “scaling without scaling.”
The cache directory became a success story in the weekly metrics review. Nobody asked how big it could get or how it would be cleaned.

A month later, the node started flapping. Not CPU, not memory—disk. The cache volume had grown steadily and then sharply,
because a new feature increased the variety of cache keys and reduced hit rate. The cache wasn’t acting like a cache anymore; it was acting like a second database,
except with no compaction, no TTL, and no backups.

The failure mode was sneaky: once the volume filled, writes failed. The app interpreted failed cache writes as “cache miss,”
so it recomputed the expensive work. That increased load. The system slowed down, error rates rose, and the cache kept thrashing.
It was a performance optimization that transformed into a reliability bug.

The fix was not heroic: enforce a maximum cache size, evict by LRU/age, and treat the cache volume like a resource with a budget.
They also made the cache per-worker in some cases to avoid a shared hot spot. Performance stayed decent, and disk stopped being the silent killer.

3) Boring but correct practice that saved the day: hard log caps and a weekly artifact budget review

A large internal platform team ran hundreds of containers per cluster. They were allergic to disk incidents because disk incidents are noisy, political,
and usually happen when leadership is watching a demo.

Their approach was aggressively dull. Every node had Docker log rotation configured by default. Every workload had a documented policy:
if you log to stdout, you get centralized shipping; if you log to files, you must rotate and you must store them on a dedicated volume.
They had a weekly “artifact budget” review: top volumes, top images, top build caches, and top log producers. Not a blame session—just a hygiene sweep.

During one incident, a service entered a tight error loop and started emitting verbose stack traces. In another environment, that would have filled disks
and taken down unrelated services. Here, the Docker log cap kept the damage contained. Observability didn’t disappear either: logs still flowed,
just with rotation and bounded retention at the node level.

The team still had to fix the bug. But they didn’t have to recover a fleet from disk exhaustion while fixing it.
That’s the real benefit of boring guardrails: they buy you time to do the real work.

Common mistakes (symptom → root cause → fix)

1) “Disk is full but I deleted files and nothing changed”

Symptom: You remove large log files, but df still shows little free space.

Root cause: The process still has the file open (deleted-but-open inode). Classic with log files.

Fix: Use lsof +L1 to find open deleted files, then restart the container/process or rotate logs properly.

2) “I ran docker system prune and disk filled again a day later”

Symptom: Temporary relief followed by recurring disk exhaustion.

Root cause: You treated symptoms (artifacts) while the workload continues to generate unbounded logs/tmp/caches.

Fix: Add log rotation caps; mount tmp as tmpfs with a size; implement cache eviction; move data writes to volumes with retention.

3) “Container says disk full but host has space”

Symptom: Host /data has hundreds of GB free, but container overlay shows 98% used.

Root cause: DockerRootDir is on / (small), while other partitions are large.

Fix: Move Docker’s data-root to the larger filesystem (planned migration). Short term: prune build cache/images and cap logs.

4) “We set log rotation but logs still blow up”

Symptom: json logs keep growing beyond expected.

Root cause: Rotation config applied only to new containers; or different logging driver in use; or app writes logs to files instead.

Fix: Verify docker info logging driver and per-container logging options. Redeploy containers after daemon config changes.

5) “Inodes hit 100% with plenty of free GB”

Symptom: df -hi shows 100% inode usage.

Root cause: Millions of small files: cache directories, extracted archives, runaway temp file creation.

Fix: Identify file-count hotspots; clean aggressively; redesign cache storage (fewer files) or use a filesystem with more inodes.

6) “Volume keeps growing but it’s ‘just cache’”

Symptom: A cache volume grows unbounded.

Root cause: Cache has no TTL/eviction; keys become effectively unique; workload changed.

Fix: Implement eviction and maximum size in app logic; consider per-instance caches; add volume alerts and dashboards.

7) “Cleanup job deleted the wrong files”

Symptom: App fails after a cleanup cron.

Root cause: Cleanup ran broad deletes (e.g., /tmp or /var) without app-owned directories and age rules.

Fix: Constrain cleanup to app-owned paths; use age-based deletion; write to dedicated directories; test in staging with realistic workloads.

Checklists / step-by-step plan

Step-by-step: emergency relief (keep services alive)

1. Confirm what’s full: df -hT and df -hi. Decide if it’s bytes or inodes.
2. Check Docker breakdown: docker system df -v to see if it’s volumes, build cache, images, or containers.
3. Stop the bleeding: if logs are huge, truncate in place or restart the offender after capturing enough diagnostics.
4. Reclaim low-risk space: prune build cache; prune stopped containers; prune unused images if your rollout strategy allows it.
5. Validate recovery: verify free space and that services aren’t in retry loops creating more disk churn.

Step-by-step: durable fix (make it boring)

1. Set Docker log caps globally (json-file max-size/max-file) and redeploy containers.
2. Move DockerRootDir off small root partitions. Put it on dedicated storage sized for artifacts and logs.
3. Stop writing durable data to writable layers: enforce volumes for anything persistent.
4. Use tmpfs for real temp: mount /tmp as tmpfs with size caps where appropriate.
5. Put cache policies in code: TTL + max size + eviction. “We’ll clean it later” is not a policy.
6. Add alerts: host filesystem usage, DockerRootDir usage, per-volume usage, inode usage, and log rate anomalies.
7. Operationalize cleanup: scheduled builder cache prunes on CI/builder nodes, with retention windows and change control.

What to avoid (because you’ll be tempted)

• Avoid deleting random directories under /var/lib/docker while Docker is running. That’s a great way to corrupt state.
• Avoid volume prune in production unless you have explicit confirmation the volumes are unused.
• Avoid “rm -rf /tmp/*” if multiple processes share /tmp. Own your temp directories.
• Avoid treating prune as maintenance for a system that is actively generating unbounded writes.

FAQ

1) Why does deleting a log file not free space?

Because a process can keep writing to an open file descriptor even after the path is deleted. The disk space is freed only when the handle closes.
Use lsof +L1 and restart/reload the process.

2) Should I switch Docker from json-file to journald?

Journald can centralize retention and integrates with system tools, but it’s not an automatic win. If you choose journald, set journald size limits.
If you keep json-file, set max-size and max-file. Pick one and manage it like you mean it.

3) Is docker system prune safe in production?

“Safe” depends on your rollout/rollback expectations. It can delete unused images and build cache, which may slow redeploys or remove rollback images.
Prefer targeted prunes (builder cache, stopped containers) and do aggressive pruning on CI nodes, not critical prod nodes.

4) What’s the best way to keep /tmp from filling?

If the temp usage is small and truly temporary, mount /tmp as tmpfs with a size limit. If temp usage can be large, use a dedicated volume with monitoring and cleanup.
In all cases, have the app write to an owned subdirectory and clean by age.

5) How do I know if the container writable layer is the problem?

Look at docker inspect SizeRw for containers, and inside the container run du -xhd1 /.
If big directories are in locations that should be volumes or tmpfs, you’ve found your problem.

6) Why do build caches get so big?

Modern builds generate lots of intermediate layers and cache objects. BuildKit is fast because it stores work; it’s also hungry.
Prune build cache on a schedule, especially on shared builders and CI runners.

7) What about inode exhaustion—how do I prevent it?

Prevent the creation of millions of tiny files (cache designs that shard into too many files, temp storms).
Monitor inode usage with df -hi. Put file-count-heavy caches on filesystems designed for it or redesign the cache layout.

8) Can I enforce per-container disk limits in plain Docker?

Docker doesn’t offer a simple universal “disk limit” like CPU/memory. You can approximate with tmpfs size limits, volumes on quota-enforced filesystems,
and operational guardrails (log caps, cache limits). In Kubernetes, use ephemeral storage requests/limits.

9) Should application logs go to stdout or files?

Stdout/stderr is usually the right move for containers: centralized collection, rotation handled by the platform, fewer moving parts.
File logs are acceptable for specific requirements, but then you own rotation, retention, and disk budgeting explicitly.

Next steps you can ship this week

1. Set Docker log rotation globally (json-file max-size/max-file) and redeploy the loudest services first.
2. Instrument disk and inode alerts for the filesystem hosting DockerRootDir and for your top volumes.
3. Identify the top 3 disk consumers using docker system df -v and du under DockerRootDir, then assign owners.
4. Convert “temp” to tmpfs where safe with explicit size caps. Make failure loud, not contagious.
5. Put cache eviction in code and prove it works under load. If you can’t explain your cache’s maximum size, you don’t have a cache.
6. Schedule build cache pruning on builder/CI nodes with a retention window that matches your development cadence.

The goal isn’t heroic cleanup scripts. The goal is predictable storage behavior. Containers are fast to create and easy to kill.
Your disk is neither.