Restart Stuck Services the Safe Way (Without Rebooting)

Principles: what “safe restart” actually means

A “safe restart” is not “service comes back.” It’s “service comes back and we didn’t corrupt data, wedge the host, or trigger a cascading failure.” That sounds obvious until you’re staring at a stuck unit at 2 AM and your hands start typing reboot by muscle memory.

1) Don’t restart until you understand what’s stuck

When a service won’t stop, it’s usually not the service’s “stubbornness.” It’s waiting on something: disk I/O, a network mount, a kernel lock, a dependency, a shutdown hook, a child process that never exits, a PID file that lies, or systemd waiting for a notification that won’t come.

If you restart blindly, you often convert a single stuck process into a multi-process crime scene. Especially if you’ve got database clients, worker queues, or storage paths involved.

2) Treat state as a first-class citizen

Stateless web services are forgiving. Stateful services are not. If the stuck service owns data (database, queue, filesystem export, cache with persistence), stopping it violently can cause long recovery times, replay storms, or corruption. Your job is to trade immediate availability against recovery and integrity, explicitly.

3) Understand what systemd is doing, not what you hope it’s doing

Systemd isn’t “a fancy init script.” It’s a process supervisor with dependency graphs, watchdogs, cgroups, and timeouts. When you ask it to stop something, it uses specific signals, in a specific order, to a specific set of processes. If it’s stuck, the reason is usually discoverable—if you look at the unit properties, cgroup membership, and journal.

4) Escalate force in steps, and announce what you’re doing

You should have an escalation ladder. Start with graceful signals and correctness checks. Only then move to SIGKILL, cgroup termination, or resetting failed states. And when you do, communicate. A “quick restart” that drops in-flight work is rarely quick for the humans downstream.

5) A reboot is not a fix; it’s an amputation

Rebooting works because it clears state, resets drivers, unwedges kernel resources, and restarts everything. That’s also why it’s dangerous: you lose the chance to diagnose, you disrupt unrelated services, and you can trigger long fsck / RAID rebuild / journal replay sequences.

One quote to keep handy, because it’s the mindset you want during incident pressure:

“Hope is not a strategy.” — Vince Lombardi

Yes, it’s from sports. Operations is also a contact sport; the opponent is entropy.

Fast diagnosis playbook (first/second/third)

This is the “I have five minutes before the incident channel turns into interpretive dance” version. The goal is to locate the bottleneck class quickly: service logic, dependencies, system resources, or kernel/storage.

First: verify what is actually stuck

1. Check unit state and recent logs: Is it “activating,” “deactivating,” “failed,” or “running but unhealthy”?
2. Confirm the main PID and cgroup processes: Are there child processes that outlived the main process?
3. See what systemd is waiting for: timeouts, notify, watchdog, ExecStop hooks, or dependency ordering.

Second: determine whether it’s waiting on I/O, locks, or network

1. Look for D-state processes: uninterruptible sleep is a huge red flag; killing won’t help.
2. Check disk pressure: high iowait, saturated device, stuck multipath, full filesystem.
3. Check network mounts and name resolution: NFS hangs and DNS timeouts cause “stop” scripts to freeze.

Third: choose the lowest-risk intervention

1. If it’s a logic hang: restart gracefully, drain traffic, rotate the process.
2. If it’s a dependency hang: fix the dependency first (storage, DNS, mount), then restart.
3. If it’s kernel/storage wedged: don’t spam restarts. Decide between isolating the host, failing over, or rebooting with intent.

Small joke #1: If you restart a service five times and it “randomly” works, congratulations—you’ve invented a load test for your luck.

Interesting facts & historical context

• Unix signals were designed for cooperative shutdown. SIGTERM is a request, not a command. Some daemons treat it politely; some ignore it; some take it as a personal challenge.
• “Stuck in D state” is older than your monitoring stack. Uninterruptible sleep has existed for decades as the kernel waits for I/O completion; it’s a hint that your problem is below userspace.
• systemd introduced cgroup-based process tracking as a practical fix for the classic “daemon forked and left an orphan” problem that haunted SysV init scripts.
• PID files are a historical compromise. They were a workaround for supervisors that couldn’t reliably track processes. They still cause outages when stale.
• Timeouts are a relatively modern operational habit. Old init scripts often waited forever; modern service managers assume “forever” is unacceptable, and will escalate or fail.
• NFS has been freezing shutdowns since the 1980s. It’s not malicious; it’s just distributed systems being honest about their failure modes.
• Database “fast shutdown” modes exist for a reason. PostgreSQL, MySQL, and others distinguish between “finish work,” “checkpoint,” and “drop everything,” because the crash-recovery cost is real.
• Watchdogs became popular after enough silent hangs. A service that’s “running” but not making progress is worse than one that crashes quickly; supervisors added watchdogs to force a decision.

Practical tasks: commands, outputs, and decisions

Below are hands-on tasks you can run on a typical Linux host using systemd. Each task includes: the command, an example output, what it means, and the decision you make from it.

Task 1: Confirm the unit state and why systemd thinks it’s stuck

cr0x@server:~$ systemctl status nginx.service --no-pager
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: deactivating (stop-sigterm) since Tue 2026-02-05 02:11:27 UTC; 1min 32s ago
       Docs: man:nginx(8)
    Process: 18421 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
   Main PID: 17102 (nginx)
      Tasks: 5 (limit: 18958)
     Memory: 62.3M
        CPU: 3.114s
     CGroup: /system.slice/nginx.service
             ├─17102 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─17105 nginx: worker process
             └─17106 nginx: worker process

Feb 05 02:11:27 server systemd[1]: Stopping A high performance web server and a reverse proxy server...
Feb 05 02:11:57 server systemd[1]: nginx.service: State 'stop-sigterm' timed out. Killing.
Feb 05 02:11:57 server systemd[1]: nginx.service: Killing process 17102 (nginx) with signal SIGKILL.

What it means: systemd tried a graceful stop, hit a timeout, and escalated. It’s still deactivating, which suggests either the process didn’t die, or children escaped, or the kernel won’t reap it.

Decision: Move from “unit view” to “process view.” Identify which PID(s) still exist, and what state they’re in. Don’t keep issuing restarts; collect evidence.

Task 2: Show the unit’s exact stop behavior (timeouts, kill mode)

cr0x@server:~$ systemctl show nginx.service -p TimeoutStopUSec -p KillMode -p KillSignal -p SendSIGKILL -p ExecStop -p ExecStopPost
TimeoutStopUSec=1min 30s
KillMode=control-group
KillSignal=SIGTERM
SendSIGKILL=yes
ExecStop={ path=/sbin/start-stop-daemon ; argv[]=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=18421 ; code=exited ; status=0 }
ExecStopPost=

What it means: systemd will kill the entire cgroup if needed. The timeout is 90 seconds. The stop command uses a PID file; that’s a potential lie vector.

Decision: If the PID file is stale or wrong, stopping can stall. Verify the PID file and whether the cgroup still has processes.

Task 3: Inspect the cgroup membership (what systemd thinks it owns)

cr0x@server:~$ systemd-cgls /system.slice/nginx.service
Control group /system.slice/nginx.service:
├─17102 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─17105 nginx: worker process
└─17106 nginx: worker process

What it means: These are the processes systemd will signal when stopping. If you see unexpected processes, your service may be forking helpers that never exit.

Decision: If the cgroup is empty but systemd says “stopping,” you might be waiting on ExecStop hooks or a stuck mount/DNS call inside them.

Task 4: Check if the process is in uninterruptible sleep (D state)

cr0x@server:~$ ps -o pid,ppid,stat,wchan:30,cmd -p 17102
  PID  PPID STAT WCHAN                          CMD
17102     1 D    nfs_wait_bit_killable          nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

What it means: The process is in D state, waiting on a kernel function related to NFS. In D state, signals won’t be handled until the kernel call returns. SIGKILL won’t save you.

Decision: Stop trying to kill it. Find the NFS mount, the network/storage issue, or plan a failover. This is not an “nginx problem.”

Task 5: Identify open files/sockets that might be hanging stop

cr0x@server:~$ sudo lsof -p 17102 | tail -n 8
nginx   17102 root   10u  IPv4  541233      0t0  TCP *:http (LISTEN)
nginx   17102 root   11u  IPv4  541234      0t0  TCP *:https (LISTEN)
nginx   17102 root   12r  REG   0,37  10485760 917514 /var/log/nginx/access.log
nginx   17102 root   13r  REG   0,37   2097152 917515 /var/log/nginx/error.log
nginx   17102 root   14r  REG   0,41    131072  26233 /mnt/shared/certs/bundle.pem
nginx   17102 root   15r  REG   0,41    131072  26234 /mnt/shared/certs/key.pem

What it means: The service has files open on /mnt/shared which appears to be a separate filesystem (device 0,41). If that is NFS and it’s unhealthy, shutdown can block on file operations.

Decision: Verify mount type and health. If it’s a remote mount and it’s wedged, fix that first or detach traffic and relocate.

Task 6: Confirm mount type and whether it’s a network filesystem

cr0x@server:~$ findmnt -T /mnt/shared -o TARGET,SOURCE,FSTYPE,OPTIONS
TARGET      SOURCE               FSTYPE OPTIONS
/mnt/shared nfs01:/exports/shared nfs4   rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,sec=sys

What it means: It’s NFSv4.1 with hard mount semantics. Hard mounts can hang processes during server/network trouble. That’s sometimes the right choice; it’s never a free choice.

Decision: If NFS is flaky, decide between restoring NFS, failing the service to another node, or (last) rebooting the client to clear stuck I/O—knowing it can come right back if NFS is still down.

Task 7: Check if the host is under memory pressure or swapping (slow “stop” can be paging)

cr0x@server:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi        29Gi       512Mi       1.2Gi       1.6Gi       745Mi
Swap:            8Gi       6.9Gi       1.1Gi

What it means: Low available memory and heavy swap. Even a polite shutdown can crawl if the process is getting paged in/out.

Decision: Before restarting, consider relieving pressure (stop non-critical workloads, scale out, add memory, or lower load). Restarting a memory-thrashing service can make things worse if it reloads caches and triggers more swapping.

Task 8: Check disk I/O saturation (service “hang” can be storage latency)

cr0x@server:~$ iostat -xz 1 3
Linux 6.1.0 (server) 	02/05/2026 	_x86_64_	(8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.21    0.00    4.05   38.44    0.00   45.30

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz  aqu-sz  %util
nvme0n1         120.0   4800.0     0.0    0.0  120.22    40.0    95.0   9200.0     2.0    2.1  210.45    96.8   35.20  99.8

What it means: Nearly 100% device utilization, huge await times, and high iowait. Your “stuck stop” may be a victim of a saturated disk.

Decision: Don’t restart services into a burning I/O queue. Identify the top I/O consumers, relieve pressure, or fail over. Restarting often increases I/O (log replay, cache warm-up, reindex, etc.).

Task 9: Find which processes are hammering the disk

cr0x@server:~$ sudo iotop -oPa -n 1
Total DISK READ: 15.20 M/s | Total DISK WRITE: 42.10 M/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
22340 be/4 postgres   1.20 M/s   18.30 M/s  12.00 %  85.00 % postgres: checkpointer
22410 be/4 postgres   0.00 B/s   10.20 M/s   0.00 %  60.00 % postgres: wal writer
30112 be/4 root       0.00 B/s    9.80 M/s   0.00 %  55.00 % rsync -a /var/log/ /mnt/backup/

What it means: Background tasks (database writers, rsync backups) are dominating write I/O. Your stuck service might be collateral damage.

Decision: Pause or throttle non-essential I/O (backup jobs, batch work). If it’s the database doing necessary recovery, let it finish before you touch dependent services.

Task 10: Check the journal for stop/start hooks that are blocking

cr0x@server:~$ journalctl -u nginx.service -n 50 --no-pager
Feb 05 02:10:55 server systemd[1]: Stopping A high performance web server and a reverse proxy server...
Feb 05 02:10:55 server start-stop-daemon[18421]: stopping nginx (pid 17102)...
Feb 05 02:11:25 server start-stop-daemon[18421]: waiting for nginx to die...
Feb 05 02:11:55 server start-stop-daemon[18421]: waiting for nginx to die...
Feb 05 02:11:57 server systemd[1]: nginx.service: State 'stop-sigterm' timed out. Killing.

What it means: The stop helper is waiting. That’s not evidence of application bug; it’s evidence that the process can’t exit or can’t be reaped.

Decision: Correlate with process state (ps) and kernel waits (wchan). If it’s D-state, fix the underlying I/O wait.

Task 11: Verify DNS health (yes, DNS can hang shutdown too)

cr0x@server:~$ resolvectl status
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (ens192)
    Current Scopes: DNS
         Protocols: +DefaultRoute
Current DNS Server: 10.10.0.53
       DNS Servers: 10.10.0.53 10.10.0.54
        DNS Domain: corp.internal

What it means: You have systemd-resolved in stub mode. If the resolver is down or slow, services that do reverse lookups during shutdown (logging, auth, stats) can stall.

Decision: If you see name resolution delays in logs, test queries and fix resolver reachability before restarting dependent services.

Task 12: See dependency failures and ordering issues

cr0x@server:~$ systemctl list-dependencies --reverse postgresql.service
postgresql.service
● ├─app-api.service
● ├─worker.service
● └─reporting.service

What it means: These services depend on PostgreSQL. Restarting PostgreSQL will ripple outward—unless you intentionally stop/drain dependents first.

Decision: Decide whether you should restart the dependency (Postgres) or the dependent (app-api). Often the “stuck” symptom lives in the dependent; the cause is the dependency. Handle order intentionally.

Task 13: Check for a stale PID file that blocks start/stop

cr0x@server:~$ sudo cat /run/nginx.pid
99999

cr0x@server:~$ ps -p 99999 -o pid,cmd
  PID CMD

What it means: PID file points to a non-existent process. Some stop scripts will wait or error; some start scripts refuse to start, thinking the daemon is still alive.

Decision: If you’ve confirmed nginx isn’t running, remove the stale PID file and start cleanly. If it is running but PID changed, fix the unit to avoid PID-file fragility (prefer systemd’s cgroup tracking / Type=notify where appropriate).

Task 14: Try a safe reload instead of restart (when possible)

cr0x@server:~$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

cr0x@server:~$ sudo systemctl reload nginx.service

What it means: A reload keeps the master process and swaps config, often avoiding connection drops. It’s not always supported, but when it is, it’s the gentlest lever.

Decision: If the issue is configuration-related, prefer reload. If the process is wedged, reload won’t help; don’t pretend it will.

Task 15: Reset systemd’s “failed” view before retrying (so you see fresh errors)

cr0x@server:~$ systemctl is-failed worker.service
failed

cr0x@server:~$ sudo systemctl reset-failed worker.service

What it means: Resetting failed clears the latch so monitoring and humans see the next failure cleanly, not yesterday’s ghost.

Decision: Do this before your next start attempt during an incident, otherwise you’ll chase stale symptoms.

Task 16: When systemd is stuck, check for pending jobs and blocked transactions

cr0x@server:~$ systemctl list-jobs
JOB  UNIT             TYPE  STATE
421  nginx.service     stop  running
422  app-api.service   stop  waiting

What it means: app-api is waiting because nginx stop is still running. systemd transactions serialize certain operations; one stuck unit can back up others.

Decision: Fix/clear the stuck unit or isolate by operating on units not blocked by that transaction. Avoid launching a dozen new jobs into the jam.

Escalation ladder: from polite to firm (without panic)

When something is stuck, humans reach for big red buttons. Resist. Use an escalation ladder, because you want the least destructive action that restores service while preserving evidence.

Level 0: Decide if a restart is even the right tool

Restarting is appropriate when:

• The process is alive but misbehaving (deadlock, memory leak, stuck thread pool), and you’ve got redundancy or a drain plan.
• A configuration change requires it (and reload isn’t available).
• A dependency is fixed and you need a clean handshake (e.g., reconnect to database, re-mount storage).

Restarting is a bad idea when:

• The process is in D state (I/O wait). Killing won’t work; repeated attempts just waste time and add confusion.
• The host is under extreme memory or I/O pressure. Restart adds load.
• You can fail over to a healthy node faster than you can “fix” this one.

Level 1: Graceful stop/start with traffic draining

If you’re behind a load balancer, drain first. Don’t “restart in place” and hope clients are patient.

cr0x@server:~$ sudo systemctl stop app-api.service

Decision: If stop completes quickly and cleanly, start it. If it hangs, move to inspection rather than repeating the command.

Level 2: Use reload or rotate workers if supported

For some daemons (nginx, haproxy, some logging agents), reload is enough. For others, you can sometimes “rotate” workers (e.g., send a signal to spawn new workers and retire old ones). This keeps sockets open and reduces client pain.

Level 3: Confirm what’s blocking shutdown (locks, I/O, dependencies)

This is where you use journalctl, lsof, ps wchan, and resource tooling. The goal: identify whether you’re waiting on storage, network, or application logic.

Level 4: Targeted termination of the right processes

If the process is in an interruptible state and simply ignoring SIGTERM, you can escalate with intent. Do it in the service’s cgroup, not by random grepping.

cr0x@server:~$ sudo systemctl kill -s SIGTERM nginx.service

cr0x@server:~$ sudo systemctl kill -s SIGKILL nginx.service

What it means: You’re asking systemd to signal all processes in the unit’s cgroup. This is cleaner than hunting PIDs, and it respects service boundaries.

Decision: Only do SIGKILL when you’ve decided that state loss is acceptable and the process isn’t in D state. If it’s D state, SIGKILL won’t land.

Level 5: If systemd itself is wedged, use cgroup cleanup carefully

Sometimes a unit is “gone” but the cgroup is messy, or a process escaped. You can inspect and act, but keep your hands steady.

cr0x@server:~$ systemctl show nginx.service -p ControlGroup
ControlGroup=/system.slice/nginx.service

cr0x@server:~$ cat /sys/fs/cgroup/system.slice/nginx.service/cgroup.procs
17102
17105
17106

Decision: If those PIDs are truly owned by that service, you can stop the unit and kill via systemd. If PIDs are in D state, stop here and fix I/O; cgroup killing won’t help.

Level 6: Last resort—reboot with intent, not frustration

If the kernel is wedged on storage (hung device, broken NFS, dead multipath path) and you can’t recover from userspace, reboot might be the right operational choice. But do it like an engineer:

• Fail over traffic first.
• Capture evidence: dmesg, journal, iostat samples, mount state.
• Communicate expected impact and recovery steps.

Small joke #2: A reboot is like turning it off and on again—except the “again” includes explaining it to your change board.

Three corporate mini-stories from the trenches

Incident caused by a wrong assumption: “Restarting the service won’t affect data”

A mid-sized company ran a payments API backed by a queue and a relational database. The API nodes were “stateless,” or so everyone said. A deploy introduced a subtle bug: worker threads would pile up waiting on a DB connection that never returned. Latency climbed; error rate followed.

The on-call saw stuck threads and did the standard move: restart the API service on one node. It came back. For about three minutes. Then it wedged again. So they restarted more nodes. Fast. The load balancer dutifully shifted traffic to the remaining nodes, which promptly saturated.

The assumption that bit them: “API is stateless.” In reality, each node also ran a local agent that buffered in-flight requests to disk for retry during transient failures. During the restart storm, the agent’s retry spool raced, re-submitted old requests, and the DB’s idempotency keys weren’t enforced consistently across code paths.

They didn’t lose money—barely. They did lose a weekend to reconciliations and incident reviews. The fix wasn’t “don’t restart.” The fix was to model the system’s state honestly: where it lives, how it’s retried, and which components must be drained before restart. The runbook now includes “disable local retry agent” and “verify idempotency enforcement” before any mass restart.

The practical lesson: you can’t safely restart what you haven’t inventoried. “Stateless” is not a vibe; it’s an architecture decision you can prove.

Optimization that backfired: “Shorter timeouts will make restarts faster”

A different org had a fleet of Linux hosts running a log forwarder and a metrics agent. The team tuned systemd units to reduce shutdown time during rolling maintenance: TimeoutStopSec=10 everywhere. It felt crisp. It looked efficient. It was also wrong.

One day, a remote storage hiccup caused intermittent delays in writing log buffers. The forwarder needed ~20–30 seconds to flush safely. But with a 10-second stop timeout, systemd escalated to SIGKILL regularly. The result wasn’t immediate downtime; it was worse: silently dropped audit logs and half-delivered batches that broke downstream parsing.

The incident was hard to spot because the “restart succeeded.” Green dashboards. Happy unit status. Meanwhile compliance folks noticed gaps, and the downstream team noticed “duplicate-ish” event patterns.

The rollback was to restore sane stop timeouts and add explicit flush metrics. They also learned to differentiate services: a web proxy can be killed faster than a buffering data pipeline. Uniform “optimization” is how you make uniform mistakes at scale.

The practical lesson: timeouts are part of your data integrity story. Make them service-specific and proven by failure tests, not by aesthetics.

Boring but correct practice that saved the day: “Drain, verify, then restart one node”

A SaaS company ran a cluster of application nodes behind a load balancer, plus a separate storage tier. A node started failing health checks intermittently. The app process wasn’t dead; it was “alive but stuck,” responding slowly and missing watchdog deadlines.

The on-call followed the boring runbook. First, they drained the node from the balancer and confirmed traffic dropped to near-zero. Then they captured local evidence: a five-minute window of iostat, current systemctl status, and the last 200 lines of the journal for the unit.

Only then did they restart the service on that single node. It still hung on stop. Instead of escalating blindly, they checked process states and found D-state waits tied to a stale iSCSI path. They detached the node from the storage session (planned), marked it out of rotation, and let the cluster carry on.

Later, during daylight, they fixed multipath settings and added alerts for path flaps. No heroic rebooting. No mass restarts. The customer impact was minimal because they treated “one bad node” as an isolation problem, not a fleet-wide drama.

The practical lesson: draining and single-node canaries feel slow, but they prevent the classic “I turned one problem into twelve” experience.

Common mistakes: symptom → root cause → fix

1) Symptom: systemctl stop hangs forever (or until timeout)

Root cause: The process is in D state waiting on storage/network I/O (NFS, iSCSI, dead disk), or an ExecStop script is blocked on DNS/mount.

Fix: Confirm with ps ... wchan and findmnt. Restore the dependency (storage/network/DNS) or fail over; don’t spam signals.

2) Symptom: restart “works” but service becomes slow immediately

Root cause: Underlying resource pressure (I/O saturation, memory swapping) plus cold caches after restart.

Fix: Check iostat, free, and top I/O consumers. Reduce load or fix the bottleneck first; restart last.

3) Symptom: systemd says “active (running)” but the service is dead/unresponsive

Root cause: Incorrect unit type or readiness signaling (e.g., Type=forking with a stale PID file), or no health check gating.

Fix: Validate PID tracking; prefer Type=notify when supported. Add watchdog/health checks or a socket check and alert on them.

4) Symptom: systemctl start fails with “Address already in use”

Root cause: Old process still listening (escaped cgroup, orphan), or socket-activated unit confusion.

Fix: Find listeners with ss -ltnp, confirm cgroup membership, kill the correct process via systemd or stop the socket unit if applicable.

5) Symptom: service keeps restarting in a loop

Root cause: Restart=always plus a persistent failure (bad config, missing dependency, permissions). The loop can create load and log storms.

Fix: Stop the unit, inspect logs, fix the real error, then start. Consider adding backoff or StartLimitIntervalSec / StartLimitBurst.

6) Symptom: stop succeeds but start blocks at “activating”

Root cause: Service readiness check never completes (waiting for notify), or it’s waiting on a dependency that is “up” but unusable (DNS returns, but slow; mount exists, but stale).

Fix: Check systemctl status + unit properties (Type=, notify), validate dependency health with direct commands (DNS lookup, mount read/write test).

7) Symptom: killing the PID does nothing

Root cause: PID is wrong (stale PID file), or process is stuck in uninterruptible sleep.

Fix: Verify PID with systemctl status and systemd-cgls. If D state, treat it as an underlying I/O failure.

8) Symptom: “Unit is masked” or “Refusing to operate on alias name” during incident

Root cause: Someone masked a service to stop it coming up, or you’re targeting an alias/symlink rather than the real unit.

Fix: Confirm with systemctl status and systemctl cat. Unmask intentionally, document why it was masked, and start the canonical unit.

Checklists / step-by-step plan

Checklist A: Safe restart of a typical stateless service (web/API)

1. Confirm impact scope: Is this node redundant? If not, pause and plan a maintenance window or failover.
2. Drain traffic: remove node from load balancer; confirm active connections fall.
3. Capture evidence: systemctl status, last 100 lines of journalctl -u, and a ps snapshot of the service PIDs.
4. Try graceful action: reload if supported; otherwise stop with systemd.
5. If stop hangs: check D state, mounts, I/O pressure. Fix cause before force.
6. Start and verify: unit active, port listening, health endpoint OK, error rate stable.
7. Return traffic gradually: re-add node; watch latency and saturation.

Checklist B: Safe restart of a stateful service (database/queue)

1. Confirm replication / HA status: know primary vs replica, quorum, and failover rules.
2. Quiesce writes if needed: pause consumers, stop batch jobs, or enable maintenance mode.
3. Check disk space and I/O health first: low space or high latency turns “restart” into “recovery marathon.”
4. Use native admin commands when available: e.g., database fast shutdown vs hard kill, to avoid long crash recovery.
5. Stop via systemd and monitor: watch logs for checkpoint/flush messages.
6. If it hangs: don’t SIGKILL reflexively. Identify whether it’s checkpointing, waiting on fsync, or stuck on underlying storage.
7. After start: verify consistency signals (replication catch-up, WAL replay complete), and only then re-enable writers/consumers.

Checklist C: When you suspect storage/network is the real culprit

1. Check for D-state processes: if yes, you’re likely below userspace.
2. Check mounts and remote filesystems: NFS, iSCSI, SMB—confirm health, latency, and error logs.
3. Check kernel messages: timeouts, link flaps, SCSI resets, NVMe errors.
4. Isolate the host: drain traffic and stop making it worse.
5. Recover dependency or fail over: restore paths, restart mount services, or move workload.
6. Reboot only with a plan: after evidence capture and with confirmed failover.

FAQ

1) Why not just reboot? It’s faster.

Sometimes it’s faster for you, not for the system. Rebooting resets everything, can trigger long recovery (fsck, RAID resync, database replay), and destroys evidence. Reboot when you’ve identified a kernel-level wedge or you’ve already failed over and want a clean slate.

2) What does “D state” mean, and why do I care?

D state is uninterruptible sleep: the process is waiting in the kernel, typically for I/O. Signals won’t be handled until the kernel call completes. If a process is stuck in D state, killing it won’t work; you need to fix the underlying I/O path or reboot the host.

3) Should I use killall or pkill during incidents?

Usually no. They’re blunt instruments and love collateral damage. Prefer systemctl kill for a specific unit so you hit the right cgroup. If you must use pkill, scope it tightly and verify with ps and systemd-cgls.

4) Why does systemd say it killed the process, but it’s still there?

If the process is in D state, it may remain present because it can’t complete the kernel operation and exit. Another possibility: you’re looking at a different PID (stale PID file) or a child process outside the unit’s cgroup.

5) Is lowering TimeoutStopSec a good idea to avoid hangs?

Not as a blanket policy. For buffering pipelines and stateful services, short timeouts convert graceful shutdown into data loss. Set timeouts per service, based on measured shutdown behavior under load and under degraded dependencies.

6) When is SIGKILL acceptable?

When you’ve decided that preserving state is less important than restoring availability, and you understand the recovery cost. It’s also acceptable for truly stateless services where the process is just being rude. It’s not a fix for D-state hangs.

7) What’s the difference between restart and try-restart?

restart starts the service even if it’s not running. try-restart restarts only if it’s already running. In production, try-restart is safer when you don’t want to accidentally start something that was intentionally stopped.

8) How do I avoid cascading failures when restarting dependencies?

Know the dependency graph. Drain or stop dependents before restarting a core dependency (database, queue, DNS, storage). Bring dependencies back first, then dependents in small batches while watching error rates and saturation.

9) systemd shows “activating” forever. What’s the usual culprit?

Readiness signaling. The service may be waiting for a notify event, a PID file, or a post-start hook that calls out to the network. Inspect unit type and ExecStartPost behavior; check logs and timeouts.

10) How do I keep evidence if I need to intervene quickly?

Grab the minimum viable evidence before applying force: unit status, last ~200 journal lines, process state snapshot, and a quick I/O sample. That’s enough to diagnose most “stuck stop” root causes later.

Next steps you can actually do tomorrow

Safe restarts aren’t about being cautious; they’re about being precise. When a service is stuck, your first job is classification: is it an application hang, a dependency hang, or a kernel/storage hang? Only one of those is solved by “restart harder.”

Do these next:

1. Write an escalation ladder for your top 5 services: reload vs restart, drain steps, and when SIGKILL is allowed.
2. Baseline shutdown time under normal load and under mild degradation, then set TimeoutStopSec accordingly.
3. Add one D-state alert: count processes in D state and page when it spikes. It’s an early warning that your storage/network is lying to you.
4. Inventory state: identify which “stateless” services actually persist, buffer, or retry locally.
5. Practice on one node in a staging-like environment: force a stuck mount, watch how stop behaves, and update the runbook with what you learn.

The goal isn’t to never reboot. The goal is to reboot only when you mean it—and to know exactly what you’re buying with that blast radius.