Debian 13: Journald ate your disk — cap logs without losing what matters

Fast diagnosis playbook

This is the “pager is going off, disk is 98%, and the app is throwing weird errors” sequence. You want signal in under five minutes. Do these in order.

First: confirm where the disk went

• Check the filesystem that’s full and whether it’s the journal path (usually /var or root).
• Measure /var/log/journal and /run/log/journal.
• Check journald’s own view of usage.

Second: identify whether it’s a log storm or retention

• Look for rate limit messages, repeated unit failures, or a chatty service.
• Check ingestion rate: “last 5 minutes” counts by unit.
• Verify whether logs are persistent and how big the caps are configured.

Third: stop the bleeding safely

• Vacuum by size or time (not random deletion), and make sure you don’t delete the only copy of the logs you need.
• Apply caps in journald.conf and restart journald.
• Fix the chatty service. Otherwise you’re just sweeping water with a broom.

Joke #1: Logs are like glitter—fine in craft time, catastrophic in production, and you’ll find them in places they never should have been.

What journald is actually doing to your disk

systemd-journald collects logs from the kernel, services, and anything writing to stdout/stderr under systemd. It stores them in a binary journal format. That buys you indexing, structured fields, and fast filtering. It also buys you the ability to fill disk at the speed of “somebody deployed debug logging on a hot code path.”

Persistent vs volatile: the quiet fork in the road

journald can store logs in memory-backed runtime storage (/run/log/journal) or on disk (/var/log/journal). On many servers you want persistent logs, because reboots happen—sometimes on purpose. But persistent logs mean you must set caps and retention, because disk is finite and logs are ambitious.

Why “disk full” from logs breaks unrelated stuff

When the filesystem containing /var fills up, journald isn’t the only thing that fails writes. You can see package installs failing, SSH sessions failing to allocate ptys, databases refusing to create temp files, and system services failing because they can’t write state. The first problem is “logs grew.” The second problem is “everything assumes writeability.”

Journald’s storage controls are percent-based by default

journald can enforce maximum usage in absolute terms (SystemMaxUse=) and can keep free space buffers (SystemKeepFree=). Defaults can be surprisingly permissive for small volumes, especially on VMs with tight root disks.

Rotation isn’t the problem; retention is

People think “rotate logs” solves this. journald already rotates internally. The disk blowups happen when retention is effectively “keep everything until a limit you didn’t realize was huge,” or when the log rate is so high that even a sane cap gets hit too quickly—causing churn and loss of the newest or oldest messages depending on your settings and pressure.

Interesting facts & context (so you stop fighting the wrong war)

• Fact 1: journald stores logs in a binary format with indexed fields, which is why journalctl _SYSTEMD_UNIT=foo.service is fast compared to grepping flat files.
• Fact 2: Persistent journal storage requires /var/log/journal to exist; if it doesn’t, journald typically stays in runtime mode under /run.
• Fact 3: The journal format supports integrity features (like sealing) intended to detect tampering, but those features don’t replace centralized logging.
• Fact 4: syslog predates journald by decades and grew up around append-only text files and remote forwarding; journald grew up assuming metadata-rich, unit-aware systems.
• Fact 5: Linux kernels can emit floods of messages if a driver or filesystem goes unhealthy; the kernel ring buffer and journald can become unwilling accomplices.
• Fact 6: On busy systems, logging itself becomes a performance feature—or a performance bug—because every log line is IO, CPU, and contention somewhere.
• Fact 7: Container platforms popularized “log to stdout,” which funnels directly into journald on systemd-based hosts unless you route it elsewhere.
• Fact 8: Percent-based caps (like “use up to 10% of filesystem”) behave very differently on a 20GB root volume versus a 2TB log volume, and defaults don’t know your intent.

One quote, because it’s still the best guiding constraint in operations: Hope is not a strategy. — paraphrased idea often attributed to leaders in engineering and ops.

Practical tasks: commands, output meaning, decisions

These are not lab exercises. These are the things you do when the disk is full, or about to be. Each task includes: a command, what the output tells you, and the decision you make next.

Task 1: Identify which filesystem is actually full

cr0x@server:~$ df -hT
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/vda1      ext4    30G   29G  200M 100% /
tmpfs          tmpfs  3.1G  1.2M  3.1G   1% /run

What it means: Root is full; journald on persistent storage likely lives under /var which is on / here. Your writes are failing globally.

Decision: Move immediately to measuring journal directories and vacuuming. Also plan a long-term cap, or this will reoccur.

Task 2: Measure journal directories by size

cr0x@server:~$ sudo du -sh /var/log/journal /run/log/journal 2>/dev/null
18G	/var/log/journal
24M	/run/log/journal

What it means: Persistent journal is the hog. Runtime journal is small.

Decision: Vacuum persistent journal; then enforce caps in configuration.

Task 3: Ask journald how much space it thinks it uses

cr0x@server:~$ sudo journalctl --disk-usage
Archived and active journals take up 18.1G in the file system.

What it means: This confirms journald’s accounting matches what you saw in du. If it doesn’t, you may be dealing with open-but-deleted files or other log paths.

Decision: If this number is large and growing fast, you need both retention limits and rate/volume control at the source.

Task 4: See how journald is configured (effective values)

cr0x@server:~$ sudo systemd-analyze cat-config systemd/journald.conf
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=1month
RateLimitIntervalSec=30s
RateLimitBurst=10000

# /usr/lib/systemd/journald.conf
[Journal]
#Storage=auto
#SystemMaxUse=
#SystemKeepFree=

What it means: You can see what’s set in /etc versus defaults. If you don’t have explicit caps, you’re trusting defaults to match your disk layout. That’s a bet you usually lose.

Decision: Put explicit limits in /etc/systemd/journald.conf (or a drop-in) tuned to your disk and compliance needs.

Task 5: Confirm journald is using persistent storage and the right path exists

cr0x@server:~$ sudo ls -ld /var/log/journal
drwxr-sr-x+ 3 root systemd-journal 4096 Dec 28 08:11 /var/log/journal

What it means: Directory exists with the expected group. Good. If it didn’t exist, journald might be in runtime mode and you’d lose logs on reboot.

Decision: If you require persistence, ensure /var/log/journal exists and the system is in the systemd-journal group where needed.

Task 6: Find the noisiest unit in the last 5 minutes

cr0x@server:~$ sudo journalctl --since "5 min ago" --output short-unix | awk '{print $6}' | sort | uniq -c | sort -nr | head
  8421 myapp.service:
  2103 kernel:
   611 sshd[1023]:
   222 systemd[1]:

What it means: myapp.service is spamming. The kernel is also chatty but secondary.

Decision: Fix the application logging level or bug first; then adjust journald retention as a safety net, not as the primary control.

Task 7: Inspect the spam content (don’t guess)

cr0x@server:~$ sudo journalctl -u myapp.service --since "15 min ago" -n 20
Dec 28 08:34:01 server myapp[2211]: DEBUG cache miss key=user:1234
Dec 28 08:34:01 server myapp[2211]: DEBUG cache miss key=user:1235
Dec 28 08:34:01 server myapp[2211]: DEBUG cache miss key=user:1236

What it means: This is debug-level noise in production. It’s not “just logs”; it’s a symptom of a change or misconfig.

Decision: Roll back the config, hotfix the logging level, or add sampling. Don’t “vacuum harder.”

Task 8: Check whether journald is rate-limiting (and losing messages)

cr0x@server:~$ sudo journalctl -u systemd-journald --since "1 hour ago" | tail -n 20
Dec 28 08:22:12 server systemd-journald[342]: Suppressed 15432 messages from /system.slice/myapp.service
Dec 28 08:22:42 server systemd-journald[342]: Suppressed 18201 messages from /system.slice/myapp.service

What it means: journald is already dropping logs due to rate limits. Your disk problem and your observability problem are happening together.

Decision: Treat this as an incident: reduce log volume at the source. Consider a separate pipeline for high-volume logs (metrics, traces, or sampled logs).

Task 9: Vacuum safely by size (fastest “make room” move)

cr0x@server:~$ sudo journalctl --vacuum-size=2G
Vacuuming done, freed 16.1G of archived journals from /var/log/journal.

What it means: This enforces an immediate cap on existing stored data. It deletes older archived files until usage is about 2G.

Decision: If the system is on fire, do this now. Then implement configuration so you don’t repeat it manually.

Task 10: Vacuum by time (useful for compliance windows)

cr0x@server:~$ sudo journalctl --vacuum-time=14d
Vacuuming done, freed 6.3G of archived journals from /var/log/journal.

What it means: Keeps only the last 14 days (approximate), removing older entries.

Decision: Use this when your retention policy is time-based. Combine with size caps so a log storm doesn’t erase your whole 14-day window in two hours.

Task 11: Verify post-vacuum disk health

cr0x@server:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   12G   17G  42% /

What it means: You’re back from the cliff. Now you can fix root cause without the filesystem screaming.

Decision: Immediately set journald caps and fix the chatty unit. “We’ll do it later” is how you meet the same problem next week.

Task 12: Configure caps (drop-in), then restart journald

cr0x@server:~$ sudo install -d -m 0755 /etc/systemd/journald.conf.d
cr0x@server:~$ sudo tee /etc/systemd/journald.conf.d/00-limits.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
RuntimeMaxUse=256M
RuntimeKeepFree=64M
MaxRetentionSec=1month
MaxFileSec=1day
RateLimitIntervalSec=30s
RateLimitBurst=5000
EOF
cr0x@server:~$ sudo systemctl restart systemd-journald

What it means: You’ve set explicit ceilings and a “keep free” buffer to protect the rest of the system. You also set rotation cadence (MaxFileSec) and rate limits.

Decision: Tune these numbers to your disk size and incident response needs. If your root disk is 20–40GB, a 2GB journal is usually sane. On dedicated log volumes, you can go higher.

Task 13: Confirm effective settings after changes

cr0x@server:~$ sudo systemd-analyze cat-config systemd/journald.conf | sed -n '1,80p'
# /etc/systemd/journald.conf.d/00-limits.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
SystemKeepFree=1G
RuntimeMaxUse=256M
RuntimeKeepFree=64M
MaxRetentionSec=1month
MaxFileSec=1day
RateLimitIntervalSec=30s
RateLimitBurst=5000

What it means: You’re not trusting your memory. You’re reading the actual configuration layering that systemd will apply.

Decision: If you see multiple conflicting drop-ins, consolidate them. Configuration sprawl is how limits silently stop working.

Task 14: Identify deleted-but-still-open files (the “df says full, du says not” trap)

cr0x@server:~$ sudo lsof +L1 | head
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
systemd-j 342 root   12w  REG  253,0  1048576     0 1234 /var/log/journal/.../system@000000.journal (deleted)

What it means: Space can remain “used” until the process closes the file handle, even if the file is deleted.

Decision: Restart the offending process (often journald itself) after vacuuming or cleanup if you see many deleted journal files held open.

Task 15: Verify journald is healthy and not stuck thrashing

cr0x@server:~$ systemctl status systemd-journald --no-pager
● systemd-journald.service - Journal Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Sun 2025-12-28 08:40:12 UTC; 2min ago
       Docs: man:systemd-journald.service(8)
           man:journald.conf(5)

What it means: journald is running. This doesn’t guarantee it’s not dropping messages, but it clears the basic “service is dead” failure mode.

Decision: If journald is crash-looping, disk corruption or permission issues may be in play. Proceed to file integrity checks and permission verification.

Task 16: Watch disk usage trend (a cheap early warning)

cr0x@server:~$ sudo watch -n 2 'journalctl --disk-usage; df -h / | tail -n 1'
Archived and active journals take up 1.9G in the file system.
 /dev/vda1        30G   12G   17G  42% /

What it means: You can see whether journal usage is stable or climbing rapidly.

Decision: If it climbs fast even after caps, you’re seeing churn (constant deletion and rewrite) which can hurt performance and still lose important logs. Fix the log source.

Capping logs without regret: retention, rate limits, and separation

Pick a retention policy like you pick backups: based on recovery, not vibes

Log retention is an engineering requirement wearing a compliance hat. You want enough logs to answer: “What changed?”, “What failed first?”, and “Who/what touched this system?” If you can’t answer those, you don’t have observability—you have a scrolling anxiety feed.

On a typical Debian VM with a 20–50GB root disk, a journal cap in the 1–4GB range is sane. If you need more, mount a separate filesystem for /var/log or at least for /var/log/journal. Keeping persistent journals on your root filesystem is fine until it isn’t—and it’s always “until it isn’t.”

Use both time and size limits

Time-based retention alone fails during log storms: you can burn through “14 days” in an afternoon. Size-based retention alone fails during quiet periods: you might keep 90 days unintentionally, which sounds nice until someone asks why sensitive debug logs are still present.

A pragmatic combination:

• SystemMaxUse= sets an upper bound.
• SystemKeepFree= protects the rest of the filesystem from journal growth.
• MaxRetentionSec= expresses policy intent.
• MaxFileSec= controls rotation granularity, which can make vacuum operations less “all or nothing.”

Rate limits are not a substitute for fixing noisy services

Rate limiting in journald protects the system from meltdown. It also hides evidence when you most need it. If you see suppression messages, treat them as a failure signal.

The correct pattern is:

1. Reduce log volume at the source (log level, sampling, fix loops).
2. Cap journald usage so the node stays alive.
3. Forward the logs you truly need to a central system with its own retention and access controls.

Separate “local forensic buffer” from “enterprise retention”

Local journals are great for immediate triage: they survive reboots, they have metadata, and they’re right there. But a single node’s disk is not your compliance archive and not your single source of truth.

If you need durable retention, forward logs out of the node. Whether you use rsyslog, journald’s forwarding options, or an agent, the principle stays the same: local disk is the shock absorber, not the long-term warehouse.

Be intentional about what “matters”

What matters is not “everything.” What matters is:

• Auth logs (login attempts, sudo, sshd, PAM failures).
• Privilege changes and service restarts.
• Kernel and storage errors (I/O timeouts, filesystem errors).
• Deployment markers and config changes.
• Application errors and request failures, but ideally not every debug breadcrumb.

When you cap logs, you’re deciding what you can afford to lose. Make that decision consciously, not by letting the disk decide for you.

Three corporate mini-stories (pain, in triplicate)

Mini-story 1: The incident caused by a wrong assumption

A mid-sized company migrated a fleet of Debian servers from “classic syslog plus rotated text files” to systemd-centric logging. The SRE team liked how quickly they could filter by unit and how journald captured stdout from services that previously vanished into the void.

Someone made a clean, confident assumption: “journald has sane defaults; it won’t fill the disk.” Nobody verified what “sane” meant on their smallest production nodes. Those nodes had modest root volumes because “they only run sidecar-ish services.”

One Monday, a service started failing to talk to an internal dependency. The retry loop logged at INFO on every attempt, with full payload dumps because a debug flag had been left on after a test. The service didn’t go down. It stayed up and screamed.

Within hours, root filesystems filled. Suddenly, unrelated services failed: package updates, metrics agents, even SSH on a few boxes. The team chased network ghosts because the first symptoms were “can’t deploy” and “health checks failing.” It took too long to notice the journal size.

The fix was boring: cap the journal explicitly and add a “keep free” buffer. The root cause fix was also boring: remove payload dumps from default logs and move retries to structured metrics with sampling. The lesson was sharp: defaults are not policy, and “won’t fill disk” is not a property—it’s a configuration.

Mini-story 2: The optimization that backfired

A different org decided to be clever. They were running many services on a single node class and wanted to reduce disk usage and IO. So they set aggressive journald caps: tiny SystemMaxUse, short retention, and low rate limits. The idea was to “protect the node” and “force teams to use the centralized logging platform.”

It worked. Disk usage stayed low. The node stayed alive under log storms. Everyone congratulated themselves and moved on.

Then a production incident hit: a slow-burn bug that occurred once every few hours and left a faint trail in logs. During the incident, the system also produced a burst of unrelated noisy logs from a flapping dependency. journald’s small cap meant the relevant error logs were evicted quickly; the aggressive rate limiting suppressed bursts; and the central log pipeline—under maintenance—had gaps.

They ended up with exactly the wrong outcome: the system survived, but the evidence did not. The postmortem was painful because it wasn’t “we didn’t log.” It was “we logged, but we engineered the system to forget at the worst possible time.”

They rolled back to a larger local buffer on each node and treated the central pipeline as durable storage, not a single point of failure. The optimization that saved disk had quietly increased mean time to diagnosis.

Mini-story 3: The boring but correct practice that saved the day

A regulated company with a reputation for being “slow” had a practice that engineers teased: every node class had an explicit journald policy in configuration management. It specified caps, retention, and a minimum free-space buffer. Every change was code reviewed. Every node also forwarded critical logs off-host.

During a vendor driver regression, the kernel started emitting storage warnings on some hosts. Those warnings triggered a cascade: a user-space service also began retrying and logging. The system experienced a log storm, exactly the kind that makes disks vanish.

But the nodes didn’t fall over. journald hit its caps, kept the free-space buffer intact, and continued to accept enough messages to show the timeline. Because logs were forwarded, the team could analyze the sequence even on nodes that rebooted.

The incident was still real—hardware and driver issues always are. But it stayed inside the boundary of “degraded performance” instead of becoming “unbootable hosts with missing evidence.” Boring practice doesn’t win applause. It wins uptime.

Common mistakes: symptom → root cause → fix

1) Symptom: Disk is full; /var/log/journal is huge

Root cause: No explicit SystemMaxUse / SystemKeepFree, or caps are too high for the root filesystem size.

Fix: Vacuum immediately, then set explicit caps in a drop-in and restart journald. Consider a dedicated log filesystem.

2) Symptom: Disk is full, but du doesn’t show where space went

Root cause: Deleted files still held open (common after manual deletions), or growth in another mount.

Fix: Use lsof +L1 to find open deleted files; restart the offending process. Avoid manual deletion; use journalctl --vacuum-*.

3) Symptom: Missing logs during the incident

Root cause: Rate limiting suppressed messages, journal caps too small causing rapid eviction, or logs were only volatile and lost on reboot.

Fix: Reduce log volume at the source; tune RateLimit*; ensure Storage=persistent and directory exists; forward critical logs off-host.

4) Symptom: Journald CPU usage spikes; system feels sluggish during storms

Root cause: High log volume causes contention and IO; too-frequent rotations; constant vacuum churn under tight caps.

Fix: Fix the noisy unit, increase caps modestly to reduce churn, and keep rotation reasonable (MaxFileSec). Consider moving journals to faster storage or separate volume.

5) Symptom: After reboot, journal history is gone

Root cause: Journald is in runtime mode (/run) because /var/log/journal doesn’t exist or storage is set to auto without persistent directory.

Fix: Create /var/log/journal, set Storage=persistent, restart journald.

6) Symptom: You capped journald, but disk still fills

Root cause: Another log pipeline is writing elsewhere (rsyslog files, application logs), or journal caps are not being applied due to config layering.

Fix: Check systemd-analyze cat-config for journald, verify rsyslog targets, and use du -x to identify other growth under /var.

Joke #2: If you ever want to feel powerful, set SystemKeepFree correctly and watch your future self stop sending angry messages to your past self.

Checklists / step-by-step plan

When the disk is already full (incident mode)

1. Run df -hT and confirm which mount is full.
2. Measure /var/log/journal and confirm with journalctl --disk-usage.
3. Identify top talkers in the last 5–15 minutes (journalctl --since ... pipeline).
4. Vacuum by size to restore headroom (journalctl --vacuum-size=...).
5. Restart journald if you suspect deleted-but-open files.
6. Reduce log volume at the source: log level, rollback, or disable debug flags.
7. Set explicit caps and keep-free buffer using a drop-in; restart journald.

Hardening plan for the next 90 days (boring, correct, effective)

1. Set policy: Define retention intent (e.g., 30 days on-host buffer, 180 days centralized for critical streams).
2. Right-size storage: Decide whether /var/log/journal belongs on root. If not, create a dedicated volume or at least separate /var mount.
3. Implement caps: Use SystemMaxUse + SystemKeepFree, and a time cap (MaxRetentionSec).
4. Make storms visible: Monitor for journald suppression messages and for rapid increases in journalctl --disk-usage.
5. Fix log hygiene: Enforce logging guidelines: no payload dumps at INFO, no tight-loop logging without backoff, sample repetitive errors.
6. Forward what matters: Ensure authentication, privilege, kernel, and deployment logs are copied off-host.
7. Test it: In staging, simulate a log storm and verify you don’t lose the key “first failure” evidence.

Configuration guidance: numbers that usually work

• Small root disks (20–40GB): SystemMaxUse=1G–3G, SystemKeepFree=1G, MaxRetentionSec=2week–1month.
• Dedicated log volume (100GB+): SystemMaxUse=10G–50G, SystemKeepFree=5G–20G, plus central forwarding.
• High-churn nodes: Prefer fixing noisy emitters over raising caps; churn can become a performance and wear problem on SSDs.

FAQ

1) Is journald “bad” compared to rsyslog?

No. journald is excellent at local, structured logging with unit context. rsyslog is excellent at text logs and flexible forwarding. In production, they’re often complementary: journald locally, forwarding centrally.

2) If I set SystemMaxUse, will journald stop logging when it hits the limit?

It will keep operating by rotating and deleting older entries to stay within limits. That means the system survives, but older evidence gets evicted. That’s why you should forward critical logs off-host.

3) What’s the difference between SystemMaxUse and SystemKeepFree?

SystemMaxUse caps the journal’s maximum footprint. SystemKeepFree enforces a minimum free space on the filesystem. If you care about node stability, SystemKeepFree is the guardrail that prevents “logs killed the OS.”

4) Should I delete files under /var/log/journal manually?

Avoid it. Use journalctl --vacuum-size or --vacuum-time. Manual deletion increases the odds of open deleted files, inconsistent state, and you skipping the policy decision entirely.

5) Why do I see “Suppressed N messages” and should I panic?

You should pay attention. It means you’re losing log lines. Sometimes that’s acceptable during a storm, but often it’s a symptom of a broken service or misconfigured log level. Fix the source.

6) I need logs for audits. Can I still cap journald?

Yes, and you should. Treat on-host journald as a limited buffer, then forward audit-relevant logs to an immutable or access-controlled central store. Auditability and “infinite local retention” are not the same thing.

7) What about containers—does this get worse?

Often, yes. Containers tend to log to stdout, and if you run many chatty containers on a host, journald becomes the sink for all of it. Your caps need to reflect that, and your platform should enforce log levels and sampling.

8) How do I know if my caps are too small?

Two signs: (1) you can’t reconstruct timelines because relevant messages are evicted quickly, and (2) journald is constantly vacuuming/rotating under load (churn). Increase caps modestly and reduce noisy logging at the source.

9) Does moving /var/log/journal to a separate filesystem actually help?

Yes. It isolates failure domains. If logs fill the log volume, they won’t usually prevent the OS from writing critical state on root. You still need caps, but your blast radius shrinks.

10) What’s the safest “emergency cleanup” if I have no time?

journalctl --vacuum-size=..., then restart journald if needed, then fix the noisy unit. Don’t start deleting random files under /var while tired.

Conclusion: next steps you can actually do today

When journald eats your disk, it’s rarely because journald is “out of control.” It’s because you didn’t give it boundaries, and because some service decided the best way to express distress was to print a novel every millisecond.

Do this next:

1. Set explicit caps with SystemMaxUse and SystemKeepFree using a drop-in, not tribal memory.
2. Combine time and size retention so storms don’t erase your investigative window.
3. Find and fix the top talkers—debug logs in production are a tax you pay with disk and sleep.
4. Forward what matters off-host so the node can die without taking the evidence with it.
5. Monitor suppression messages; they’re your early warning that observability is failing.

If you remember one thing: capping logs is not about deleting history. It’s about choosing which history you can afford to keep—on purpose.