Ubuntu 24.04 “Permission denied” executing scripts: noexec mounts and how to fix (case #58)

A practical mental model: what “Permission denied” really means

When you try to run ./deploy.sh, a chain of decisions happens quickly:

• The shell checks that the file exists and that you have execute permission on it (the x bit).
• The kernel loads it via execve(). This is where mount options and MAC policies bite.
• If it’s a script, the kernel reads the shebang (#!/bin/bash) and tries to execute the interpreter with the script as an argument.
• Security layers can veto: filesystem mount flags (noexec), AppArmor profiles, systemd sandboxing, container runtime restrictions, and sometimes “helpful” hardening defaults.

The key point: you can have -rwxr-xr-x and still get Permission denied if the filesystem is mounted with noexec. The file mode says “you may execute,” but the mount says “nobody executes anything from here.” The mount wins.

One useful operator’s heuristic: if chmod +x didn’t help, assume the problem is not Unix permissions. Assume policy.

One quote to keep around: as a paraphrased idea from Gene Kranz (mission control discipline): “Tough and competent means you don’t accept vague failures—identify the constraint and design around it.”

Short joke #1: Noexec is like a hotel that lets you check in, but confiscates your shoes so you can’t leave. Secure? Sure. Convenient? Not so much.

Fast diagnosis playbook (check first/second/third)

If you’re on call, you don’t want a lecture. You want a sequence that narrows the search fast.

First: confirm the exact failing path and error

• Run it directly (./script) and also run via interpreter (bash script).
• Observe if the error changes: “Permission denied” vs “Exec format error” vs “No such file or directory”. Those map to different failure modes.

Second: check mount options for the directory

• findmnt -no OPTIONS --target /path is the fastest truth source.
• If you see noexec, stop arguing with chmod.

Third: check system-level policy if mounts look fine

• For services: systemd hardening options and sandboxing can emulate noexec behavior.
• For interactive sessions: AppArmor denies show up in the journal and dmesg.
• For containers: confirm the volume mount options and whether you’re executing from ConfigMap/Secret/overlay.

Then: pick the least dangerous fix

• Preferred: move the executable to an exec-enabled location (e.g., /usr/local/bin or a dedicated /opt).
• Acceptable: remount a specific mount with exec if risk is understood.
• Last resort: disable systemd/AppArmor hardening broadly. You’ll regret it later.

Interesting facts and history (because context prevents outages)

1. noexec is per-mount, not per-file. You can’t “chmod your way” out of a mount policy.
2. The kernel still allows reading and mapping. noexec blocks execve() and some forms of file-backed execution, but it doesn’t stop reading the file or copying it elsewhere.
3. Hardening /tmp is old wisdom. Mounting /tmp with noexec became common after worms and sloppy installers treated /tmp as a software distribution channel.
4. Scripts are not special. noexec applies to scripts and ELF binaries alike. The interpreter path is exec’d, but the script file itself is opened under exec rules.
5. “Permission denied” is not a permissions error. It’s a generic EACCES that can mean “policy” as easily as “mode bits.”
6. systemd changed the game. Services can have private mount namespaces and restrictions that don’t match what you see in your shell.
7. Enterprise images love noexec. CIS benchmarks and internal security baselines frequently recommend nodev,nosuid,noexec for world-writable areas.
8. Container platforms recreated the same problem. Kubernetes ConfigMap/Secret volumes are typically mounted with restrictive options; running code directly from them is intentionally awkward.
9. NFS adds policy ambiguity. “noexec” might be on the client mount, or execution might be blocked by server export policy and root-squash patterns that confuse people.

Hands-on tasks: commands, outputs, decisions (12+)

These are the things I actually do when someone says “it’s permission denied” and I can smell the impending time sink.

Task 1: verify the file is executable and not a directory typo

cr0x@server:~$ ls -l ./deploy.sh
-rwxr-xr-x 1 cr0x cr0x 1843 Dec 30 01:55 ./deploy.sh

What it means: the execute bit is set for owner/group/others. If you still get “Permission denied,” it’s likely not Unix mode bits.

Decision: move on to mount options and policy checks, not more chmod.

Task 2: try running through the interpreter (bypasses file exec, but not always noexec)

cr0x@server:~$ ./deploy.sh
bash: ./deploy.sh: Permission denied
cr0x@server:~$ bash ./deploy.sh
./deploy.sh: line 12: /tmp/helper: Permission denied

What it means: the script itself might be readable, but it tries to execute something else from a noexec location (/tmp/helper here).

Decision: identify every executable path the script uses; the failing one is often in /tmp, a mounted workspace, or a bind mount.

Task 3: check the mount options for the script’s directory

cr0x@server:~$ findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS --target /home/cr0x
/dev/nvme0n1p3 /home ext4 rw,relatime,noexec

What it means: /home is mounted noexec. Any direct execution from /home will fail.

Decision: either move executables out of /home or remove noexec (with a hard look at your threat model).

Task 4: confirm the kernel error code via strace (fast truth serum)

cr0x@server:~$ strace -f -e execve ./deploy.sh
execve("./deploy.sh", ["./deploy.sh"], 0x7ffce9c4a3b0 /* 45 vars */) = -1 EACCES (Permission denied)
strace: exec: Permission denied

What it means: the kernel rejected execve() with EACCES. This is consistent with noexec, AppArmor, or other policy.

Decision: check mount and MAC policy logs; don’t waste time on shell-level hypotheses.

Task 5: inspect all mounts that include noexec (find the landmines)

cr0x@server:~$ mount | grep -w noexec
/dev/nvme0n1p3 on /home type ext4 (rw,relatime,noexec)
/dev/nvme0n1p4 on /tmp type ext4 (rw,relatime,nodev,nosuid,noexec)

What it means: both /home and /tmp are noexec. That’s a common enterprise baseline.

Decision: decide where execution is supposed to happen: usually /usr/local/bin, /opt, or a dedicated application directory.

Task 6: confirm the exact mount flags with findmnt (more precise than mount output)

cr0x@server:~$ findmnt -no TARGET,PROPAGATION,OPTIONS /tmp
/tmp shared rw,relatime,nodev,nosuid,noexec

What it means: propagation being shared matters in container contexts; mount changes may propagate unexpectedly.

Decision: if you’re in a containerized setup, treat mount propagation as part of the blast radius.

Task 7: quick remediation test (temporary remount) to prove root cause

cr0x@server:~$ sudo mount -o remount,exec /tmp
cr0x@server:~$ findmnt -no OPTIONS --target /tmp
rw,relatime,nodev,nosuid,exec

What it means: you’ve flipped /tmp to executable for this boot session (until remount/reboot or fstab restores it).

Decision: if execution now works, you’ve confirmed noexec as the cause. Now revert and implement a safer permanent fix (usually not “make /tmp exec forever”).

Task 8: update fstab for a permanent change (only when policy allows)

cr0x@server:~$ grep -nE '\s/tmp\s' /etc/fstab
12:/dev/nvme0n1p4 /tmp ext4 defaults,nodev,nosuid,noexec 0 2

What it means: fstab hard-codes noexec for /tmp.

Decision: if you must change it, change it intentionally and document why. Better: keep /tmp noexec and relocate build artifacts.

Task 9: the boring safe fix—install scripts to an exec path

cr0x@server:~$ sudo install -m 0755 ./deploy.sh /usr/local/bin/deploy
cr0x@server:~$ /usr/local/bin/deploy --help
Usage: deploy [options]

What it means: you placed the script in a standard executable directory with appropriate permissions.

Decision: prefer this over making user-writable mounts executable. It reduces the “download and run” attack surface.

Task 10: check shebang correctness (wrong interpreter path looks like permission weirdness)

cr0x@server:~$ head -n 1 ./deploy.sh
#!/bin/bash
cr0x@server:~$ ls -l /bin/bash
-rwxr-xr-x 1 root root 1265648 Apr 10  2024 /bin/bash

What it means: the interpreter exists and is executable. If the shebang pointed to something missing, you’d often see “No such file or directory,” not “Permission denied,” but it’s still worth checking.

Decision: if scripts are intended to be portable, consider #!/usr/bin/env bash (with the usual caveats about PATH control in privileged contexts).

Task 11: detect AppArmor denies (policy can mimic noexec)

cr0x@server:~$ sudo journalctl -k -g apparmor --since "10 min ago"
Dec 30 02:05:41 server kernel: audit: type=1400 apparmor="DENIED" operation="exec" profile="snap.mytool.mytool" name="/home/cr0x/deploy.sh" pid=22109 comm="mytool" requested_mask="x" denied_mask="x" fsuid=1000 ouid=1000

What it means: AppArmor denied execution. This is not a mount issue; it’s confinement.

Decision: fix by adjusting the profile, changing the execution location to an allowed path, or using a non-confined packaging method. Do not disable AppArmor globally unless you enjoy compliance meetings.

Task 12: check systemd unit sandboxing for a service failure

cr0x@server:~$ systemctl cat myapp.service
# /etc/systemd/system/myapp.service
[Service]
ExecStart=/opt/myapp/bin/myapp
ProtectSystem=strict
PrivateTmp=true
NoNewPrivileges=true

What it means: PrivateTmp=true gives the service its own /tmp, which may be mounted differently than your interactive /tmp. Sandboxing changes the filesystem view.

Decision: reproduce inside the service context (see next task) and adjust the unit or file placement accordingly.

Task 13: run a shell under the service’s exact sandbox to reproduce

cr0x@server:~$ sudo systemd-run --unit debug-myapp --pty --property=PrivateTmp=true /bin/bash
Running as unit: debug-myapp.service
Press ^] three times within 1s to disconnect TTY.
root@server:/# findmnt -no OPTIONS --target /tmp
rw,nosuid,nodev,noexec,relatime
root@server:/# exit
exit

What it means: within the unit’s namespace, /tmp is noexec. Your script might work interactively but fail as a service.

Decision: avoid executing from /tmp in services. Put helper binaries under /usr/libexec, /opt, or package them properly.

Task 14: verify mount options for a bind mount (often overlooked)

cr0x@server:~$ sudo mount --bind /srv/build /mnt/build
cr0x@server:~$ findmnt -no SOURCE,TARGET,OPTIONS /mnt/build
/srv/build /mnt/build rw,relatime
cr0x@server:~$ sudo mount -o remount,bind,noexec /mnt/build
cr0x@server:~$ findmnt -no OPTIONS /mnt/build
rw,relatime,bind,noexec

What it means: bind mounts can have their own flags. Someone may have “helpfully” remounted a bind mount noexec while the underlying filesystem is exec.

Decision: always check the target mountpoint you execute from, not just the backing filesystem.

Task 15: confirm NFS mount options (client-side noexec is common)

cr0x@server:~$ findmnt -no SOURCE,FSTYPE,OPTIONS --target /mnt/tools
nfs01:/exports/tools nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noexec

What it means: NFS share is mounted noexec on the client. Scripts stored there won’t run.

Decision: either mount with exec (if policy allows) or copy tools to a local exec-enabled directory during deployment.

Where noexec comes from on Ubuntu 24.04

Ubuntu 24.04 didn’t suddenly become hostile to scripts. What changed is that more environments default to hardening patterns, and more tooling assumes it can execute from “wherever.” These trends collide.

1) fstab and “security baseline” images

The simplest source is /etc/fstab. Enterprises bake images where separate partitions exist for /home, /tmp, sometimes /var, and they apply nodev,nosuid,noexec to reduce abuse of user-writable locations.

If you grew up on single-root filesystems, this looks exotic. If you’ve been yelled at by auditors, it looks like Tuesday.

2) systemd’s private mount namespaces

Systemd can give services their own world. That world can include:

• PrivateTmp= (service-specific /tmp)
• ProtectSystem= and ProtectHome= (read-only or hidden paths)
• ReadWritePaths= and TemporaryFileSystem= (custom mounts)

Result: you check findmnt in your shell, see exec, and still fail in the service. That’s not gaslighting; it’s namespaces.

3) container volume mounts and orchestrator defaults

In containers, “the filesystem” is a collage: overlay layers, bind mounts, tmpfs mounts, and orchestrator-provided volumes. Many of these are intentionally restrictive.

The classic: ConfigMap/Secret volumes in Kubernetes are not a place to run programs. They’re a place to put config. If your app downloads a plugin into one of these mounts and tries to execute it, the kernel will decline.

4) networked storage policies (NFS, CIFS)

Remote storage introduces “execution” as a policy question. On Linux clients, noexec is enforced by the client kernel on that mount. Separately, servers can enforce permissions and mapping that lead to confusing denies when a script tries to access an interpreter or helper binary.

5) MAC policies (AppArmor) and confinement (Snap)

Ubuntu favors AppArmor. Snaps add confinement layers. Both can produce EACCES with “Permission denied” even when mounts are fine. A confined process might be allowed to read a file but not execute it, or allowed to execute only from a known set of paths.

Fix patterns: safest first, permanent when you must

Fix pattern A: stop executing from user-writable, ephemeral locations

If your pipeline or installer drops binaries into /tmp, /var/tmp, a home directory, or a mounted workspace and then runs them, you’re relying on a security anti-pattern. It works until it doesn’t—then it burns an hour of your life.

Do this instead:

• Install executables under /usr/local/bin (admin-managed) or /opt/<app>/bin.
• For helper binaries, consider /usr/libexec/<app> (common on modern distros) or keep them next to the main binary in /opt.
• Keep writable runtime data in /var/lib/<app> and logs in /var/log/<app>.

This pattern doesn’t fight the OS. It cooperates.

Fix pattern B: if you need a writable build directory, create one that is explicitly exec-enabled

Some workflows legitimately need to build and run artifacts (compilers, JITs, build systems). If your policy sets /home noexec, you need a dedicated exec-enabled build area. The name should make the intent obvious, because ambiguity is how incidents breed.

cr0x@server:~$ sudo mkdir -p /srv/exec-work
cr0x@server:~$ sudo chown cr0x:cr0x /srv/exec-work
cr0x@server:~$ findmnt -no OPTIONS --target /srv
rw,relatime

What it means: /srv is typically exec-enabled unless you’ve hardened it too. Now your build artifacts have a place to live without weakening /home or /tmp.

Decision: direct builds to /srv/exec-work (or equivalent) and keep the rest of the system hardened.

Fix pattern C: remount with exec (surgical, with eyes open)

Sometimes you’re stuck: legacy vendor software expects to execute from /home or a mounted tools share. If you can’t refactor quickly, you can remount with exec. Do it as narrowly as possible.

Good: a dedicated mountpoint for tools with exec. Bad: making /tmp exec because “it fixed it.”

cr0x@server:~$ sudo mount -o remount,exec /home
cr0x@server:~$ findmnt -no OPTIONS --target /home
rw,relatime,exec

What it means: the policy is relaxed. Everything in /home becomes executable.

Decision: if you do this, you should compensate elsewhere (better monitoring, stricter software install controls, or move toward per-user tooling in containers/VMs).

Fix pattern D: use packaging and deployment hygiene instead of “curl | bash” energy

If your system installs by downloading to /tmp and running immediately, you’re going to trip over noexec and you’re also training people into risky habits. Package it, or at least download to a controlled exec path and verify signatures/checksums.

Short joke #2: The only thing worse than “works on my machine” is “works on my machine because my machine is insecure.”

Systemd sandboxes that look like noexec

Operators blame systemd for everything, and systemd earns a solid chunk of that blame. But when it breaks your script execution, it’s usually doing exactly what you (or your distro) asked for: limiting what a service can see and do.

How this failure presents

• Script runs fine in SSH session.
• Fails as a service with “Permission denied” executing a helper binary.
• Mount options appear fine from the host shell.

Why it happens

Systemd units can run in their own mount namespace. That namespace may mount /tmp as a private tmpfs, sometimes with noexec. Or it may use TemporaryFileSystem= to mount a writable tree with restrictive flags.

What to do

• Reproduce in the unit namespace using systemd-run with similar properties (see Task 13).
• Stop executing from /tmp. Put helper binaries in a stable directory and allow the service to read/execute them.
• Use ReadWritePaths wisely. If a service needs a writable directory, grant it narrowly instead of broad filesystem access.

A precise service-side fix example

If your service unpacks a tool and runs it, redesign so the tool is shipped as part of the package image. If you can’t, at least unpack into a directory you control with known mount properties (not /tmp inside a sandbox).

cr0x@server:~$ sudo systemctl edit myapp.service
# (editor opens)

And you’d add something like:

cr0x@server:~$ sudo systemctl cat myapp.service
# /etc/systemd/system/myapp.service
[Service]
ExecStart=/opt/myapp/bin/myapp
PrivateTmp=true
ReadWritePaths=/var/lib/myapp

What it means: you’re giving the service a writable state directory while keeping the rest constrained. You’re not trying to make its private /tmp executable just to run a random blob.

Decision: favor stable exec locations and narrow write permissions. That’s the ops version of eating vegetables.

Containers, Kubernetes, and the “mounted but not executable” trap

Containers love to mount stuff. Config. Secrets. Shared volumes. Scratch space. They also love to make those mounts restrictive, because one compromised pod should not become a cluster-wide hobby.

Common containerized failure mode

You bake an image. It runs fine. Then you mount a volume over /app or /app/bin in Kubernetes. Suddenly your startup script fails with “Permission denied.” You keep rebuilding the image like it’s going to apologize. It won’t.

Task: inspect mount options inside a container

cr0x@server:~$ sudo docker run --rm -it -v /tmp:/mnt/tmp ubuntu:24.04 bash -lc 'findmnt -no TARGET,OPTIONS /mnt/tmp'
/mnt/tmp rw,nosuid,nodev,noexec,relatime

What it means: the bind mount came through as noexec (or was remounted noexec by policy). Executing artifacts from it will fail.

Decision: don’t run from that mount. Copy executables into the container’s filesystem (image layer) or mount a volume designed for execution.

Kubernetes specifics (the quiet part)

ConfigMap and Secret volumes are not your plugin directory. If you need a script from a ConfigMap, a common pattern is: copy it to a writable, exec-enabled directory at startup, then execute from there. That still requires that the target directory is exec-enabled.

Task: prove noexec on a typical projected volume path (example)

cr0x@server:~$ findmnt -R /var/lib/kubelet/pods 2>/dev/null | head -n 5
TARGET                                                SOURCE         FSTYPE OPTIONS
/var/lib/kubelet/pods                                  /dev/sda1      ext4   rw,relatime
/var/lib/kubelet/pods/abcd.../volumes/kubernetes.io~secret/default-token tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=4096k

What it means: secret volumes are tmpfs and typically noexec. That’s by design.

Decision: treat these mounts as data-only. If you must execute a script, move it to a proper location with controlled permissions.

NFS and remote filesystems: execution policies and surprises

NFS brings its own flavor of weird. Mostly because it’s doing identity mapping, permissions, and caching over a network, and everybody assumes it’s just “a disk but farther away.”

Two realities to keep straight

• Client mount flags (noexec) are enforced by the client kernel. If the client says noexec, nothing runs, even if the server “would allow it.”
• Server-side permissions still matter for reading and traversing directories. A lot of “Permission denied” on NFS is actually directory execute permission (x on directories) or root-squash behavior breaking an installer running as root.

Task: differentiate “mount says noexec” from “server perms are wrong”

cr0x@server:~$ findmnt -no OPTIONS --target /mnt/tools
rw,relatime,vers=4.2,hard,timeo=600,retrans=2,noexec
cr0x@server:~$ ls -ld /mnt/tools
drwxr-xr-x 10 root root 4096 Dec 29 18:21 /mnt/tools

What it means: directory permissions look fine, but mount says noexec. That’s the blocking constraint.

Decision: remount with exec if allowed, or copy tools locally. Fixing Unix perms won’t help.

Task: remount NFS share with exec (if policy allows)

cr0x@server:~$ sudo mount -o remount,exec /mnt/tools
cr0x@server:~$ findmnt -no OPTIONS --target /mnt/tools
rw,relatime,vers=4.2,hard,timeo=600,retrans=2,exec

What it means: client now allows execution from that mount.

Decision: if this is a “tools share,” consider making it read-only and exec-enabled, and keep writable shares noexec. Mixing write+exec is where malware has a good day.

Common mistakes: symptom → root cause → fix

• Symptom: bash: ./script.sh: Permission denied even though ls -l shows executable.
  Root cause: filesystem is mounted noexec (often /home or a workspace).
  Fix: move script to /usr/local/bin or remount the specific filesystem with exec.
• Symptom: script runs interactively but fails in systemd service with “Permission denied” executing a helper in /tmp.
  Root cause: service has PrivateTmp or other sandboxing with a noexec tmpfs.
  Fix: stop running helpers from /tmp; ship helpers in /opt or /usr/libexec.
• Symptom: bash: ./script: /bin/bash^M: bad interpreter: No such file or directory (or similar).
  Root cause: CRLF line endings; not noexec, but it presents as “can’t execute.”
  Fix: convert to LF: sed -i 's/\r$//' script (then re-test).
• Symptom: Permission denied only when executed via Snap-installed tool.
  Root cause: AppArmor confinement denies execution from that path.
  Fix: execute from allowed paths, adjust confinement, or install a non-snap version where policy allows.
• Symptom: Kubernetes pod fails to execute a script stored in a ConfigMap.
  Root cause: ConfigMap volume mounted noexec (and often read-only).
  Fix: copy to a writable exec-enabled directory at startup, or bake the script into the image.
• Symptom: “Permission denied” executing from an NFS tools share; local disk works.
  Root cause: client mounted NFS with noexec.
  Fix: remount with exec or sync/copy tools locally during deployment.
• Symptom: Only one specific directory fails to execute; parent filesystem is exec-enabled.
  Root cause: bind mount remounted with noexec (common in containers and chroots).
  Fix: findmnt --target the exact path and remount that bind with exec.
• Symptom: You execute a script, it fails with “Permission denied” on a file it tries to run from /var/tmp.
  Root cause: /var/tmp is also hardened with noexec in some baselines.
  Fix: change the script to place executables in an exec-enabled directory; don’t assume /var/tmp is different from /tmp.

Three corporate mini-stories from the trenches

1) The incident caused by a wrong assumption: “chmod fixes permission denied”

A mid-size company rolled out Ubuntu 24.04 to a fleet of build runners. Security had quietly updated the base image: separate /home partition, mounted noexec. The rationale was solid—developers were downloading tools into home directories and running them with questionable provenance.

Then the CI pipeline failed. Not one job. All of them. The failure message was the usual shrug: “Permission denied” when running a build script. An engineer assumed it was a botched checkout: wrong owner, missing execute bit, maybe Git messed up. They added chmod -R +x in the pipeline. It didn’t help. It did, however, create a brand new problem: suddenly non-executable files had execute bits, which made later integrity checks noisy and confusing.

The incident escalated because people kept chasing the wrong layer. Someone tried running the script with sudo. Someone else blamed SELinux (on Ubuntu, in a room full of people, which is a special kind of optimism). A third person suggested “just disable security hardening.” That person was politely told to go get coffee and not touch anything.

The actual fix took five minutes once the right question was asked: “What does findmnt say for the workspace?” It said noexec. The pipeline copied the toolchain to a dedicated exec-enabled directory under /opt/ci-tools during runner provisioning, and scripts were executed from there. Home stayed noexec. Security was happy. CI was happy. The only casualty was everyone’s belief that chmod is a universal solvent.

2) The optimization that backfired: executing from /tmp to “make it faster”

An internal platform team tried to shave seconds off deploy time for a Java service that used a native helper binary (compression and checksum operations). The helper lived on a shared NFS tools mount. Occasionally NFS latency spiked during backups, making deploys flaky. Someone proposed a clever optimization: copy the helper into /tmp at startup and run it from there. “Local disk is faster,” they said. True, in a narrow sense.

In staging, it worked. In production, it failed instantly. Newer hardened nodes mounted /tmp with nodev,nosuid,noexec. The binary could be copied, but it couldn’t run. The service went into a crash loop. On-call saw “Permission denied” and assumed file ownership issues. They changed ownership. They changed modes. They re-deployed. The crash loop continued, smugly.

The fix was also simple: don’t execute from /tmp. The team created /var/lib/myapp/bin owned by the service user and ensured it lived on an exec-enabled filesystem. They also stopped copying at runtime and instead included the helper in the build artifact. The NFS dependency disappeared, and so did the startup trickery.

Postmortem takeaway: optimizations that change execution paths are security changes, whether you meant them to be or not. You can’t optimize your way around a policy boundary. The policy boundary will win, and it won’t even break a sweat.

3) The boring but correct practice that saved the day: “verify mount options during build and boot”

A regulated org ran a mixed fleet: some hosts were general-purpose, others were hardened according to a baseline. They’d been burned before by “mystery permission denied” incidents, so they added a mundane check to their host validation: during provisioning and daily health checks, they recorded mount options for key paths (/tmp, /home, app directories, build workspaces). The output was shipped to their logging system, not because it was exciting, but because it was searchable at 03:00.

One night a deployment failed on a subset of nodes. The service tried to execute a migration helper from /var/lib/myapp/tmp. On those nodes, /var/lib was a separate filesystem mounted noexec due to a misapplied baseline role. The code was fine. The package was fine. The filesystem policy was inconsistent.

Instead of guessing, on-call pulled the last-known-good mount snapshot for a working node and compared it with a failing node. The difference popped immediately. The team rolled back the baseline role on the affected nodes, then later fixed the role logic so only the intended partitions got noexec.

Nothing heroic happened. No one “debugged Linux” live on a critical system. They just had boring data that answered the question. This is what reliability looks like when it grows up.

Checklists / step-by-step plan

Checklist: triage a single failing script on Ubuntu 24.04

1. Run ls -l on the script; confirm it’s executable and owned as expected.
2. Run it directly and via interpreter; note whether the failing path is the script itself or a helper it calls.
3. Run findmnt -no OPTIONS --target <path> for the script and any helper binaries.
4. If noexec is present: stop. Decide whether to move the executable or change the mount policy.
5. If no noexec: check AppArmor denials in journalctl -k.
6. If it fails only in a service: reproduce in the unit context using systemd-run and inspect PrivateTmp/TemporaryFileSystem.
7. If it fails only in containers: inspect mount options inside the container and check what volume is covering the path.

Checklist: choosing a fix that won’t embarrass you later

1. Preferred: place executables in an admin-controlled exec location (/usr/local/bin, /opt).
2. Next best: create a dedicated exec-enabled workspace and document it.
3. Risky: remount broad user-writable mounts with exec.
4. Usually wrong: make /tmp exec permanently to satisfy a build/deploy tool.
5. Operational requirement: if you change mount policy, ensure it survives reboot (fstab/systemd mount units) and is consistent across the fleet.

Checklist: validating the fix

1. Re-run findmnt for the target path and confirm the option set is what you think it is.
2. Run the script under the same user as production (service user), not as root unless required.
3. If systemd service: restart the unit, then check systemctl status and logs.
4. If container: redeploy and confirm that the same path isn’t being shadowed by a volume mount in prod.

FAQ

1) Why do I get “Permission denied” when the file is chmod +x?

Because the filesystem mount can prohibit execution via noexec. The kernel rejects execve() regardless of mode bits. Verify with findmnt -no OPTIONS --target /path.

2) Is noexec a security feature or just an annoyance?

Both. It meaningfully reduces “drop a payload into a writable directory and run it” style attacks. It’s also an annoyance for sloppy install patterns that treat writable areas as executable staging.

3) Can I bypass noexec by running bash script.sh?

Sometimes you can run a script through an interpreter because you’re executing /bin/bash, not the script file as a binary. But many noexec scenarios still block execution of helper binaries the script calls, and some policies still enforce restrictions. Don’t rely on this as a “fix.”

4) Does sudo bypass noexec?

No. noexec is enforced by the kernel on that mount, for all users including root.

5) What’s the safest permanent fix?

Stop executing from hardened writable mounts. Install scripts/binaries to /usr/local/bin or /opt/<app> and keep writable runtime state elsewhere (/var/lib).

6) Why does it work in my shell but fail in a systemd service?

Because the service may run in a different mount namespace with different mount flags (PrivateTmp, TemporaryFileSystem) and path access restrictions (ProtectSystem, ProtectHome).

7) How do I know it’s AppArmor and not noexec?

Check kernel logs: sudo journalctl -k -g apparmor. AppArmor denials are explicit and include operation="exec" and the profile name. noexec won’t generate an AppArmor log entry.

8) Should I remove noexec from /tmp?

Usually no. If a tool requires executing from /tmp, treat that as a design flaw and relocate the executable. If you must change it, do so narrowly and with compensating controls.

9) What about NFS? Is “noexec” enforced by the server?

On Linux clients, noexec is a client mount option enforced by the client kernel. The server can still cause permission issues via standard Unix permissions and identity mapping, but noexec itself is typically client-side.

10) I changed mount options, but after reboot the problem came back. Why?

Your remount was temporary. Persistent behavior comes from /etc/fstab or systemd mount units. Fix the source of truth and ensure it’s consistent across hosts.

Conclusion: what to change Monday morning

If Ubuntu 24.04 tells you “Permission denied” when executing a script, believe it—but don’t assume it’s about chmod. Your first suspect should be noexec on the mount containing the script or one of its helper binaries. Your second suspect should be systemd sandboxing. Your third is AppArmor or container volume behavior.

Next steps that pay rent:

1. Standardize where executables live (/usr/local/bin, /opt) and stop executing from /tmp//home/workspaces.
2. Add a lightweight health check that records mount options for key paths across the fleet.
3. For services, audit systemd hardening settings and ensure your software doesn’t rely on executing from ephemeral directories.
4. For containers, treat mounted config as data-only; execute from the image or a purpose-built exec-enabled volume.

The OS is doing you a favor. It’s just doing it loudly, at night, while your pager watches.