Schedule Scripts That Actually Run: Task Scheduler Done Right

What “reliable scheduling” actually means

Scheduling is not “run this command at 02:00.” Scheduling is a contract: run at roughly this cadence, under a known identity, with a known environment, producing known artifacts, and emitting enough evidence that you can prove what happened. Or didn’t.

The four promises your scheduled job must keep

• It runs: The scheduler triggers it, even after reboot, even after time changes, even when the machine is busy.
• It runs once: No overlaps, no duplicates, no “two midnights” because DST got spicy.
• It runs the same way: Same PATH, same working directory, same config, same permissions, same locale, same umask.
• It leaves evidence: Logs, exit codes, timestamps, metrics, and alerts when it misses.

Everything else is secondary. Yes, even “runs fast.” Fast is nice. Correct is rent money.

One opinion that will save you pain: treat scheduled scripts as services. Not “a bash one-liner.” Services have unit files, resource limits, logs, retries, and owners. The job can be a script, but the operation should feel like a service.

Paraphrased idea (if you’ve done ops, you’ve heard the sentiment): Failures happen; reliability comes from designing systems that detect and recover from them. — paraphrased idea attributed to the SRE school of thought popularized by Google.

Quick history & facts (because context prevents bad decisions)

1. cron is old enough to have opinions. It dates back to early UNIX (late 1970s era). That’s why it’s simple, stable, and also why it assumes a world without container sprawl.
2. Vixie Cron became the “default cron” story across many Linux distros for years, shaping behaviors like environment handling and mail-on-output expectations.
3. Windows Task Scheduler has been around since the Windows 9x/NT days and evolved into a fairly capable engine with triggers, conditions, and “run whether user is logged on.” It’s not just a GUI anymore.
4. DST creates “missing” and “duplicate” times. In many time zones, 02:30 may never happen one day a year, and happen twice another day. That’s not a theoretical problem; it’s an incident generator.
5. systemd timers exist partly because cron can’t express modern constraints well (dependencies, sandboxing, per-unit logging, and “run missed jobs after reboot”).
6. anacron was invented for laptops and other machines that aren’t always on at the scheduled moment, because cron only triggers when the machine is running.
7. the “thundering herd” is a scheduling artifact: fleets configured with “run at midnight” can stampede shared services. Randomization/jitter is a reliability feature.
8. mailing job output used to be normal. Many cron setups relied on local mail. Modern systems often don’t have an MTA installed, so output vanishes into the void.

Pick your scheduler like you pick a filesystem

Linux/Unix: cron vs systemd timers vs “something else”

Use cron when: you need universal compatibility, very simple periodic triggers, and you can supply the missing production features yourself (locking, logging, environment control, monitoring).

Use systemd timers when: you’re on a systemd-based distro and you want reliability primitives built-in: journald logs, dependencies, resource controls, sandboxing, and “Persistent=true” to catch up after downtime.

Use Kubernetes CronJobs when: the job belongs in the cluster and needs container isolation, resource requests/limits, and cluster-native retry policies. But remember: you just moved failure modes, you didn’t delete them.

Use a workflow engine when: you need DAGs, retries per step, backfills, and audit trails. If your “script” has become a small business, stop pretending it’s a hobby.

Windows: Task Scheduler is fine, if you treat it like prod

Windows Task Scheduler can be very reliable, but it will absolutely let you shoot yourself in the foot with credentials, “start in” directories, and user session assumptions. If you schedule PowerShell, do it like you mean it: explicit paths, explicit execution policy decisions, log everything, and don’t rely on mapped drives.

The scheduler is not your reliability layer

Even the best scheduler can’t fix:

• a script that isn’t idempotent,
• a job that can overlap,
• a job that depends on “whatever PATH is today,”
• a job that “succeeds” while silently producing garbage.

Pick a scheduler for triggers and orchestration. Build reliability into the job.

Joke 1/2: A cron job without logs is like a submarine without sonar: it’s quiet right up until you hit something expensive.

Design patterns for scripts that survive production

1) Make the environment boring on purpose

Cron runs with a minimal environment. systemd units may run with a different environment than your interactive shell. Windows tasks run with a different profile than your RDP session. The fix is the same everywhere: declare what you need.

• Use absolute paths for executables and files.
• Set PATH explicitly (and minimally).
• Set locale (LC_ALL=C) if output parsing depends on it.
• Set umask explicitly if you create files others must read.
• Set a working directory explicitly (or don’t rely on one).

2) Treat time as hostile

Time zones, daylight savings, leap seconds, clock skew, and NTP steps: time will betray you. If your job is date-based, decide whether “date” means UTC or local time, and be explicit.

• Prefer UTC for timestamps and partition names.
• If a business process needs local time (“send at 8am local”), use local time but guard DST days.
• Record both “scheduled time” and “actual start time” in logs.

3) Use locking to prevent overlaps

Overlaps are the classic slow-burn failure: everything is fine until one run takes longer than usual, the next run starts, and now you have two processes competing for the same files, database rows, or storage bandwidth.

On Linux, use flock. On Windows, use a lock file with an exclusive open approach, or use OS primitives (mutex) if you’re in a real language. Don’t DIY locking with “check then create”; that’s race-condition cosplay.

4) Make scripts idempotent, or at least safely repeatable

Schedulers retry. Humans rerun. Machines reboot mid-job. If rerunning causes corruption, you don’t have automation—you have a slot machine.

• Write outputs to a temp path, then atomically rename into place.
• Use “marker files” carefully; include version and timestamp.
• When interacting with databases, use transactions and unique keys for de-duplication.
• If deleting data, stage it (move to quarantine) before permanent removal.

5) Make “success” measurable

Exit code 0 is necessary and insufficient. A job that generates an empty backup file can still exit 0. Define what success means:

• Expected output exists and is non-empty.
• Checksum matches (where applicable).
• Row counts within expected bounds.
• Backup is restorable (periodic test restores).

6) Log like you’re going to be on call for it (because you are)

Every scheduled job should log: start time, end time, runtime, key parameters, and a clear final status. If it touches storage, log bytes written/read and any errors from the storage layer.

Route logs somewhere you can search. If you rely on local files, rotate them. If you rely on journald, make sure retention is sane.

7) Set timeouts and resource limits

A hung job is worse than a failed job because it blocks the next run and starves the box. Use timeouts for network calls and total runtime caps.

systemd is excellent here: TimeoutStartSec=, CPU and memory limits, IO scheduling, and sandboxing. Cron can do it too, but you’ll write more wrapper code.

8) Add jitter to avoid fleet-wide stampedes

If a thousand servers run “daily cleanup” at midnight, congratulations: you invented a distributed denial of service against your own storage.

Add a random delay (bounded), or schedule across a window. Jitter is cheap reliability.

9) Monitor the absence, not just the presence

Alerting on “job failed” is good. Alerting on “job did not run” is better. Missed schedules happen: disabled timers, broken crontabs, paused VMs, expired credentials.

Push a heartbeat metric or write a timestamp file that monitoring checks. Silence is not success.

Practical tasks (commands, output, decisions)

These are real operational checks. Run them when you’re building a schedule, and run them again when it breaks at 02:13. Each item includes: command, what the output means, and what decision you make next.

Task 1: Confirm the cron daemon is actually running

cr0x@server:~$ systemctl status cron
● cron.service - Regular background program processing daemon
     Loaded: loaded (/usr/lib/systemd/system/cron.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-02-03 00:12:10 UTC; 2 days ago
       Docs: man:cron(8)
   Main PID: 812 (cron)
      Tasks: 1 (limit: 18945)
     Memory: 1.4M
        CPU: 2.912s

Meaning: If it’s not active (running), your job never had a chance.

Decision: If inactive/failed, fix the service first (enable/start). Don’t touch the script yet.

Task 2: Verify the crontab entry is installed for the right user

cr0x@server:~$ crontab -l
MAILTO=""
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
15 2 * * * /opt/jobs/nightly-report.sh

Meaning: You can see schedule, shell, PATH, and whether output will be mailed.

Decision: If this is the wrong user (common), install it with the correct account or use system-wide cron with explicit user.

Task 3: Check cron logs for your job trigger

cr0x@server:~$ grep -E "CRON|nightly-report" /var/log/syslog | tail -n 5
Feb  5 02:15:01 server CRON[24719]: (cr0x) CMD (/opt/jobs/nightly-report.sh)
Feb  5 02:15:01 server CRON[24718]: (CRON) info (No MTA installed, discarding output)

Meaning: Cron did trigger. Also, output is being discarded because there’s no mail system and you didn’t redirect output.

Decision: Fix logging immediately: redirect stdout/stderr to a file or to syslog/journald.

Task 4: Prove the script runs non-interactively with the scheduler’s environment

cr0x@server:~$ env -i HOME=/home/cr0x USER=cr0x SHELL=/bin/bash PATH=/usr/bin:/bin /bin/bash -lc '/opt/jobs/nightly-report.sh'
/opt/jobs/nightly-report.sh: line 12: psql: command not found

Meaning: Your interactive PATH had psql; the job’s PATH doesn’t.

Decision: Use absolute path to psql or set PATH in the script/unit/crontab. Do not “source .bashrc” as a band-aid.

Task 5: Add locking to prevent overlaps (and test it)

cr0x@server:~$ flock -n /run/lock/nightly-report.lock /opt/jobs/nightly-report.sh; echo "exit=$?"
exit=0

cr0x@server:~$ flock -n /run/lock/nightly-report.lock /opt/jobs/nightly-report.sh; echo "exit=$?"
exit=1

Meaning: First run acquired the lock. Second run failed immediately (exit 1) because the lock is held.

Decision: Decide policy: skip if locked (often correct), or wait with a timeout (sometimes correct). Document it.

Task 6: Confirm exit codes propagate and are visible

cr0x@server:~$ /opt/jobs/nightly-report.sh; echo "job_exit=$?"
job failed: could not connect to database
job_exit=2

Meaning: The script returns a non-zero exit code and prints a clear error.

Decision: If exit codes are always 0, fix the script. Schedulers can only react to what you tell them.

Task 7: Build a systemd timer that catches up after downtime

cr0x@server:~$ systemctl cat nightly-report.timer
# /etc/systemd/system/nightly-report.timer
[Unit]
Description=Run nightly report

[Timer]
OnCalendar=*-*-* 02:15:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target

Meaning: Persistent=true runs the job after boot if it was missed. RandomizedDelaySec adds jitter.

Decision: If “must run daily no matter what,” use a timer with persistence or a workflow engine, not plain cron.

Task 8: Inspect the service unit for resource limits and logging

cr0x@server:~$ systemctl cat nightly-report.service
# /etc/systemd/system/nightly-report.service
[Unit]
Description=Nightly report job
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=cr0x
Group=cr0x
WorkingDirectory=/opt/jobs
Environment=PATH=/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/bin/flock -n /run/lock/nightly-report.lock /opt/jobs/nightly-report.sh
TimeoutStartSec=30m
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
StandardOutput=journal
StandardError=journal

Meaning: This is what “production scheduling” looks like: dependencies, lock, timeout, and logs in journald.

Decision: If your job competes with interactive workloads or storage-heavy services, set IO/CPU priorities here rather than hoping.

Task 9: Check when the timer last ran and whether it’s overdue

cr0x@server:~$ systemctl list-timers --all | grep nightly-report
nightly-report.timer  loaded active waiting  Thu 2026-02-05 02:23:11 UTC  3h ago  Thu 2026-02-06 02:15:00 UTC  20h left nightly-report.service

Meaning: You get last/next run times. If waiting is missing or last run is ancient, something is wrong.

Decision: If overdue, check whether the timer is enabled, the clock is sane, and the unit is failing or stuck.

Task 10: Read the logs for just this job

cr0x@server:~$ journalctl -u nightly-report.service -n 20 --no-pager
Feb 05 02:15:02 server nightly-report.sh[25101]: start ts=2026-02-05T02:15:02Z
Feb 05 02:15:02 server nightly-report.sh[25101]: connecting db=reporting
Feb 05 02:19:41 server nightly-report.sh[25101]: wrote /var/reports/nightly-2026-02-05.csv bytes=1842201
Feb 05 02:19:41 server nightly-report.sh[25101]: done status=ok runtime_s=279

Meaning: You can prove it ran, how long it took, and what it produced.

Decision: If logs don’t show a clear start/end, improve script logging before you improve anything else.

Task 11: Detect overlap or runaway processes

cr0x@server:~$ pgrep -af nightly-report.sh
25101 /bin/bash /opt/jobs/nightly-report.sh

Meaning: You see whether it’s currently running, and with what PID/command.

Decision: If multiple instances exist, add locking and consider a runtime cap/timeout.

Task 12: Check the lock holder if you’re stuck “locked forever”

cr0x@server:~$ ls -l /run/lock/nightly-report.lock
-rw-r--r-- 1 cr0x cr0x 0 Feb  5 02:15 /run/lock/nightly-report.lock

cr0x@server:~$ lsof /run/lock/nightly-report.lock
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
flock   25100 cr0x    3w   REG  253,0        0  987 /run/lock/nightly-report.lock

Meaning: The file existing doesn’t mean it’s locked; lsof shows whether a process holds it.

Decision: If a dead process doesn’t hold the lock, your locking approach is wrong (don’t use “lockfile exists” logic). Use flock properly.

Task 13: Validate time sync (because schedules assume time is real)

cr0x@server:~$ timedatectl
               Local time: Thu 2026-02-05 05:24:12 UTC
           Universal time: Thu 2026-02-05 05:24:12 UTC
                 RTC time: Thu 2026-02-05 05:24:11
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Meaning: If the clock isn’t synchronized, “2:15” becomes a suggestion.

Decision: Fix time sync before chasing phantom scheduler bugs.

Task 14: Check disk space and inode pressure (storage kills jobs quietly)

cr0x@server:~$ df -h /var /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       100G   94G  1.8G  99% /
tmpfs            16G  128M   16G   1% /tmp

cr0x@server:~$ df -i /var
Filesystem       Inodes  IUsed  IFree IUse% Mounted on
/dev/sda2       6553600 6501000  52600   99% /

Meaning: You can have free bytes but no inodes, or vice versa. Either can break “write output” tasks.

Decision: If usage is >95%, stop pretending this is a scheduling issue. Free space, rotate logs, fix runaway temp files.

Task 15: Spot IO bottlenecks that stretch runtimes and cause overlaps

cr0x@server:~$ iostat -xz 1 3
Linux 6.5.0 (server) 	02/05/2026 	_x86_64_	(8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.12    0.00    3.44   31.55    0.00   56.89

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz  aqu-sz  %util
sda              3.10     90.2     0.00   0.0   12.33    29.1     40.0   10240.0    12.0   23.1  210.45   256.0     8.41  98.7

Meaning: High %iowait and %util near 100% indicates the disk is saturated. High w_await suggests writes are queued.

Decision: If your job became “slow,” check IO. Then consider scheduling outside peak, throttling, or moving heavy writes to better storage.

Task 16: Confirm permissions match the scheduler identity

cr0x@server:~$ sudo -u cr0x test -w /var/reports && echo writable || echo not_writable
not_writable

Meaning: The user running the job can’t write the target path.

Decision: Fix directory ownership/ACLs, or run the job under the correct service account. Do not “just run it as root” unless you enjoy postmortems.

Task 17: Validate DNS/network dependencies for jobs that call services

cr0x@server:~$ getent hosts db.internal
10.40.12.8   db.internal

cr0x@server:~$ nc -vz db.internal 5432
Connection to db.internal (10.40.12.8) 5432 port [tcp/postgresql] succeeded!

Meaning: Name resolution works; the port is reachable.

Decision: If DNS fails or port is blocked, fix network policy/health before rewriting the script.

Task 18: Windows-style gotcha check on Linux: find scripts with CRLF line endings

cr0x@server:~$ file -b /opt/jobs/nightly-report.sh
Bourne-Again shell script, ASCII text executable, with CRLF line terminators

Meaning: CRLF can break interpreters in subtle ways, especially with shebangs.

Decision: Convert to LF (dos2unix), commit it properly, and stop copying scripts through email.

Three corporate mini-stories (the kind you learn from once)

Mini-story 1: The incident caused by a wrong assumption (time zones are not a detail)

A mid-sized company ran a nightly “data freeze” job at 00:05 local time. The script tagged rows with freeze_date=$(date +%F), then downstream systems used that date as the partition key. For years it worked, because “local time” and “business day” felt like the same thing.

Then they expanded to a second region. Someone did the sensible thing for infrastructure and set servers to UTC. The scheduler still ran at “00:05” because that’s what the crontab said. But now “00:05” was UTC, not local. The freeze ran hours earlier than intended for the original region, and hours later for the new region.

The real damage wasn’t the run time—it was the date stamp. Some rows that should have been tagged for the next business day were stamped with the previous day’s date. Partitions were “missing” data, dashboards showed a sudden drop, and the finance team started a small, polite panic.

The first fix proposed was classic: “Set the timezone back.” It would have helped one region and hurt the other. The second fix was better: define a business day boundary in code (explicit TZ for date calculation) and record timestamps in UTC. The job now computed the partition key using a configured business timezone and logged both the key and the UTC timestamp.

The lasting lesson was simple and annoying: you can’t outsource semantics to date. If a job’s output depends on “what day it is,” you must define which clock you mean.

Mini-story 2: The optimization that backfired (compression is not free)

A different org had a backup job that dumped a database and compressed it. Storage costs were rising, so someone “optimized” by turning compression to the max and increasing parallelism. The backups got smaller. Everyone congratulated the graph.

Two weeks later, 02:00 became the new peak traffic hour. The backup job saturated CPU and disk IO, and it did so with perfect consistency. Other scheduled tasks—log rotation, ETL, even security scans—started overlapping and timing out. The scheduler wasn’t failing; it was faithfully executing a plan that no longer fit reality.

The first symptom was not “backup failed.” The first symptom was “random services slow in the morning.” The on-call chased application latency, then database locks, then network. Only after someone graphed host IO wait did the pattern show up: the “optimized” backup was turning the host into a sad, vibrating brick.

The fix wasn’t to undo compression entirely. It was to bound the blast radius: lower compression level, cap CPU usage, give it an IO priority, and move it to a different time window with jitter across the fleet. They also introduced a maximum runtime; if the job exceeded it, it failed loudly and paged, instead of quietly stealing the whole night.

Optimization that ignores contention isn’t optimization. It’s moving cost from “disk space” to “everyone’s sleep.”

Mini-story 3: The boring but correct practice that saved the day (idempotency + atomic writes)

A team ran a scheduled job that generated a CSV used for customer billing. The previous version wrote directly to /srv/billing/current.csv. One night the machine rebooted mid-write after an unrelated kernel update. The file existed, the job “ran,” and downstream systems happily consumed a truncated CSV. Bad invoices followed. Not catastrophic, but expensive in human time.

The team changed one thing: the job now wrote to a uniquely named temporary file, validated row counts and a checksum, then atomically renamed it into place. They also kept the previous known-good file for a few days. It was boring. It was correct.

Months later, a storage hiccup caused a brief IO error mid-run. The job failed before rename, leaving the old current.csv intact. Downstream systems continued using the last good output while the job alerted the on-call that it couldn’t produce a new artifact.

No scramble. No “what changed?” calls. No forensic recovery of partial data. Just a clean failure and a stable output contract. Reliability often looks like refusing to be clever.

Fast diagnosis playbook

When a scheduled job doesn’t run (or runs wrong), you need a sequence that finds the bottleneck quickly. Not a vibe-based debugging session.

First: Did the scheduler trigger anything?

• cron: check syslog entries for the CMD line; confirm crontab entry exists and daemon is running.
• systemd: check list-timers, unit status, and last run time; confirm the timer is enabled.
• Windows: check Last Run Time / Last Run Result and the History tab (if enabled).

If there’s no trigger record, stop. Fix scheduling and enablement. Don’t touch the script.

Second: Did it start but die immediately?

• Read logs: journald or your redirected log file.
• Look for “command not found,” permission denied, missing config, wrong working directory.
• Re-run under a minimal environment to reproduce.

If it dies immediately, it’s almost always environment, identity, or permissions.

Third: Did it run but take too long?

• Check overlap: multiple PIDs, lock contention, “already running” messages.
• Check resource bottlenecks: disk full, inode exhaustion, IO wait, CPU steal, network timeouts.
• Check downstream dependencies: database locks, API rate limiting, DNS.

If runtime stretched, fix contention and add timeouts/limits before you add retries.

Fourth: Did it “succeed” but produce junk?

• Validate artifacts: size, checksum, row counts, schema versions.
• Look for partial writes: file timestamps, atomic rename usage.
• Check “success criteria” in code: are you actually checking anything?

Silent bad output is worse than failure. Make bad output impossible to publish.

Joke 2/2: The only thing more reliable than a cron job is a cron job that fails the moment you stop watching it.

Common mistakes: symptoms → root cause → fix

1) Symptom: “It runs manually but not on schedule”

Root cause: PATH/environment differences; missing working directory; script relies on interactive shell init files.

Fix: Use absolute paths, set PATH explicitly, set WorkingDirectory in systemd, avoid sourcing .bashrc. Reproduce with env -i.

2) Symptom: No logs, no output, no errors

Root cause: Output discarded (no MTA for cron), or logs written somewhere not accessible to the scheduler identity.

Fix: Redirect stdout/stderr to a log file or journald. Ensure log directory permissions and rotation.

3) Symptom: Job runs twice (or overlaps) and corrupts data

Root cause: No locking; long runtime; retries without idempotency; DST duplicate hour.

Fix: Add flock locking. Add runtime limits. Make outputs atomic and idempotent. Consider UTC scheduling or explicit TZ handling.

4) Symptom: “Randomly fails” with network errors

Root cause: DNS flaps, transient dependency failure, no retries/backoff, network not ready at boot.

Fix: Add dependency checks, bounded retries with backoff, and in systemd use After=network-online.target plus Wants=network-online.target.

5) Symptom: Runs, but downstream says output is empty/partial

Root cause: Writes directly to final filename; consumer reads while producer writes; reboot mid-write.

Fix: Write to temp + fsync if needed + atomic rename. Keep last-known-good output.

6) Symptom: Works for weeks, then stops forever

Root cause: Credential expiration (API tokens, Kerberos, database password rotation); locked forever due to incorrect lock implementation; disk filled gradually.

Fix: Alert on “job did not run” and “output missing.” Use OS-backed locks (flock). Monitor disk usage and rotate logs.

7) Symptom: “It’s scheduled, but it never catches up after downtime”

Root cause: cron doesn’t backfill missed runs; timers not persistent; laptop/VM was off.

Fix: Use systemd timers with Persistent=true, or design the job to run based on “last successful checkpoint” rather than wall-clock only.

8) Symptom: CPU spikes at schedule time across the fleet

Root cause: Identical schedules cause thundering herd; no jitter.

Fix: Add randomized delay in systemd timers or a bounded random sleep in the wrapper; spread schedules.

Checklists / step-by-step plan

Checklist A: Before you schedule any script

1. Define success: what artifact/side-effect proves it worked?
2. Define cadence: “daily” is not precise. Is catch-up required?
3. Define identity: which user/service account runs it? What permissions?
4. Define dependencies: network, database, storage mounts, secrets.
5. Make it repeatable: idempotent or safe to rerun.
6. Make it non-overlapping: locking plus maximum runtime.
7. Make logs unavoidable: stdout/stderr captured; start/end status.
8. Make time explicit: UTC vs local; DST behavior documented.
9. Plan monitoring: alert on failure and on missing run.

Checklist B: A “good enough” wrapper pattern (Linux)

Even if you keep cron, wrap the script. Your wrapper is where production discipline lives: environment, locking, logging, timeouts.

cr0x@server:~$ cat /opt/jobs/wrappers/nightly-report-wrapper.sh
#!/bin/bash
set -euo pipefail

export PATH="/usr/local/bin:/usr/bin:/bin"
export LC_ALL="C"
umask 027

log_dir="/var/log/jobs"
mkdir -p "$log_dir"
log_file="$log_dir/nightly-report.log"

exec >> "$log_file" 2>&1

echo "start ts=$(date -u +%FT%TZ) host=$(hostname -f)"

timeout 30m flock -n /run/lock/nightly-report.lock /opt/jobs/nightly-report.sh

echo "done ts=$(date -u +%FT%TZ) status=ok"

Why this works: it fails fast (set -euo pipefail), captures logs, enforces a timeout, and prevents overlap. It also makes the environment predictable.

Checklist C: systemd timer rollout plan (single host to fleet)

1. Write the .service unit with explicit User/Group, PATH, WorkingDirectory, timeout, and logging.
2. Write the .timer with OnCalendar, Persistent=true, and jitter.
3. Test: systemctl start job.service manually; verify logs and exit status.
4. Enable timer: systemctl enable --now job.timer.
5. Watch: list-timers and journald for at least one full cycle.
6. Add monitoring: alert if last success > expected window; alert on non-zero exit.
7. Only then: roll to more hosts. Don’t do fleet-wide midnight changes. You’re not a chaos engineer; you’re trying to sleep.

Checklist D: Storage-aware scheduling rules

1. Don’t schedule write-heavy jobs at the same time as backups, compactions, or snapshot replication.
2. Watch free space and inode usage on output and temp directories.
3. Use atomic writes for shared artifacts.
4. Throttle compression and concurrency; measure IO wait.
5. Plan for log growth; rotate or ship logs.

FAQ

1) Cron or systemd timers: which should I use on Linux?

If you have systemd, prefer timers for anything you care about: built-in logging, dependency control, Persistent=true, and resource limits. Cron is fine for simple, non-critical periodic tasks, or when portability matters.

2) How do I stop jobs from overlapping?

Use OS-backed locks. On Linux, wrap the command with flock. Also set a maximum runtime (timeout) so “stuck” doesn’t become “forever locked.”

3) Why does my job run fine over SSH but fail in cron?

Your interactive shell sets PATH, locales, and sometimes credentials. Cron does not. Reproduce with env -i, then make the script declare what it needs (absolute paths, explicit config paths, explicit locale).

4) Should I redirect output to a file or to journald?

If you’re already using systemd, journald is usually the cleanest: searchable, tagged by unit, and centrally manageable. If you’re on cron, a file is fine—just rotate it and make sure permissions allow writing.

5) What’s the right way to handle secrets for scheduled jobs?

Don’t hardcode them in scripts or crontabs. Use a secrets manager if you have one, or at least root-readable config files with tight permissions. For systemd, consider environment files with controlled access. Rotate secrets and alert on auth failures.

6) How do I make a job “catch up” after downtime?

cron won’t backfill. Use systemd timers with Persistent=true, or design the job to process from a stored checkpoint (“last successful timestamp”) rather than assuming it always runs.

7) How do I handle DST for a job that must run at a local business time?

Decide behavior for the DST “missing” hour and “duplicate” hour. Common choices: run at a safe time (e.g., 03:15), or run in UTC and translate outputs. If local time is required, log the timezone, record UTC timestamps, and guard against duplicate runs with locking and idempotency.

8) My job is slow sometimes. Should I add retries?

Not first. Slowness is often contention (IO saturation, database locks) or a dependency issue. Measure where time goes, cap runtime, and prevent overlaps. Retries without controls can multiply the load and make the incident louder.

9) What’s the minimum monitoring I should add?

Two signals: (1) last successful run timestamp, (2) last run exit status. Alert if the job is late/missed or exits non-zero. Also track artifact freshness if it produces a file or report.

10) When should I stop using “scheduled scripts” and move to a workflow tool?

When you have dependencies between steps, need backfills, require detailed audit trails, or have complex retry logic. If you’re building a DAG with bash and email, you already answered your own question.

Conclusion: practical next steps

If you want scheduled scripts that actually run, don’t start by arguing about cron versus timers. Start by making your job survivable: explicit environment, locking, idempotency, logging, and monitoring for absence.

1. Pick the execution identity and lock it down (least privilege, predictable permissions).
2. Add locking (flock) and a maximum runtime (timeouts).
3. Make outputs atomic (temp file + rename) and define success checks.
4. Capture logs somewhere searchable, and rotate/retain them intentionally.
5. Alert on “failed” and “did not run,” not just “it printed an error.”
6. If you’re on Linux with systemd: migrate critical jobs to timers with Persistent=true and jitter.
7. Run one controlled failure drill: break DNS or fill a temp directory in a test environment and confirm your job fails loudly and safely.

Scheduling is easy. Reliable scheduling is a discipline. Do the boring parts now so your future self can sleep through 02:15.