You did the ritual: chmod -R 777, maybe a chown for good measure, restarted the container,
and it still says “permission denied.” The host path is writable. The UID matches. The filesystem isn’t read-only.
Yet your container acts like it’s locked out of its own house.
This is the part where engineers start blaming Docker, then Linux, then “security,” then each other. The truth is
boring: you’re being blocked by a mandatory access control system (SELinux or AppArmor), and it’s doing exactly what
it was built to do. The error message just doesn’t bother telling you.
Why “permission denied” survives chmod and chown
Unix permissions (owner/group/mode bits) are discretionary access control. They’re local and negotiable. If you own
the file, you can grant access. If you’re root, you can usually bulldoze through. Containers add namespaces and
capabilities, but at the end of the day, when a process tries to open a file, the kernel decides.
SELinux and AppArmor are mandatory access control (MAC). They’re not “file permissions.” They’re an additional
policy layer that can deny access even when the mode bits allow it. That’s the key: the kernel can return
EACCES for reasons that have nothing to do with ls -l.
Here’s the pattern:
- Classic permission issue: wrong UID/GID, wrong mode, wrong ACL. Fix with chown/chmod/setfacl.
- SELinux issue: labels (contexts) don’t allow the container domain to touch that path. Fix with labeling, booleans, or policy.
- AppArmor issue: the container’s profile forbids a path, a mount, a capability, or a syscall. Fix the profile or switch it.
You can spot MAC issues because your container behaves like a well-mannered burglar: it has the keys (UID/mode),
but the alarm system still calls the cops (SELinux/AppArmor).
Joke #1: If chmod 777 fixed everything, security would be a single Bash alias and we’d all be out of jobs.
SELinux vs AppArmor: different religions, same excommunications
SELinux in one production-minded paragraph
SELinux is label-based. Everything has a context (user:role:type:level). Policy decides which “domains” (process
types) can access which “types” (object labels) with what permissions. Containers typically run in a confined domain
like container_t, and host files must be labeled in a way that domain is allowed to touch. If you bind
mount random host paths into a container without proper labels, SELinux will block access even when the filesystem
says “sure.”
AppArmor in one production-minded paragraph
AppArmor is path-based. Profiles define which paths a process can read/write/execute, plus which capabilities and
kernel interfaces it can use. Docker often applies a default profile (commonly docker-default) unless
you override it. If your container needs to mount something, access /sys in a special way, or touch a
host path not anticipated by the profile, AppArmor can deny it.
Why you feel confused
They fail the same way at the application layer. Your app sees “permission denied.” Your logs show a stack trace.
Your team argues about file ownership. Meanwhile the real denial is in the kernel audit trail, and your container
doesn’t get a nicer error because the VFS layer doesn’t do therapy.
Your job is to figure out which policy engine is in play, then read the right logs, then make the smallest safe
change. The rest of this guide is that path, with fewer spiritual metaphors.
Interesting facts and short history (so the weirdness makes sense)
- SELinux started as a research project (originally from the NSA) and was designed to enforce policy even against root. That’s why “but I’m root” doesn’t impress it.
- AppArmor began life as SubDomain and became AppArmor after acquisition and rebranding; it gained popularity because it’s easier to reason about when you think in file paths.
- Docker didn’t invent confinement; it inherited what the kernel already had: namespaces, cgroups, LSM hooks, seccomp. Docker mostly wires them together and takes the blame.
- SELinux container labeling evolved a lot as containers became mainstream; early patterns were cruder, and modern distros have better defaults for
container_tand friends. - The
:zand:Zmount flags exist because bind mounts break the “labels match intent” assumption; these flags relabel content so containers can access it. - Audit logs aren’t “debug logs”; they’re security events. When teams centralize them properly, SELinux issues go from mysterious to routine.
- AppArmor’s path-based model can be tricked by rename/link games if profiles aren’t careful; SELinux’s label model avoids some of that but adds labeling operational burden.
- Overlay filesystems changed the container storage story; SELinux had to learn to label overlay layers correctly, and mislabeling can create failures that look like Docker bugs.
Fast diagnosis playbook (what to check first/second/third)
First: identify which MAC system is active (and for Docker, which one matters)
- If SELinux is enforcing and Docker is using it, start with AVC denials.
- If AppArmor is enabled and the container runs under a profile, start with AppArmor denials.
- If neither is active, go back to classic permissions and user namespaces.
Second: confirm the failing operation and the exact host path/device involved
- Is it a bind mount path? A named volume? A socket (like Docker socket)? A device node?
- Is the denial on read, write, execute, create, relabel, mount, or something else?
- Does it fail only on one host? If yes, suspect policy differences, not your YAML.
Third: pull the kernel/audit evidence, not guesses
- SELinux: look for
type=AVCin audit logs, then interpret contexts and requested permissions. - AppArmor: look for
apparmor="DENIED"and the profile name, then map to file paths/capabilities.
Fourth: pick the smallest fix that preserves confinement
- SELinux: use proper labels (
:z/:Z,chcon, orsemanage fcontext), avoid disabling SELinux. - AppArmor: adjust or create a profile, or switch to an unconfined profile only when you understand the blast radius.
Fifth: validate with a repeatable test and leave breadcrumbs
- Reproduce with a minimal container and a single mount.
- Document the label/profile expectation in Compose/Kubernetes manifests and in host provisioning.
- Make it survivable across reboots and relabels.
Practical tasks: commands, outputs, and decisions (12+)
These are the commands I actually reach for when a container can’t read/write a mount or gets blocked doing
something “obviously allowed.” Each task includes: command, sample output, what it means, and the decision you make.
Task 1: Is SELinux enabled and enforcing?
cr0x@server:~$ getenforce
Enforcing
Meaning: SELinux policy is actively denying disallowed actions (not just logging).
Decision: Treat “permission denied” as potentially SELinux until proven otherwise. Go find AVCs.
Task 2: Quick SELinux status sanity check
cr0x@server:~$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 33
Meaning: You have SELinux running in targeted mode (common on Fedora/RHEL derivatives).
Decision: Confirm Docker/container policy packages exist and your host labeling is correct.
Task 3: Is AppArmor enabled?
cr0x@server:~$ aa-status
apparmor module is loaded.
24 profiles are loaded.
22 profiles are in enforce mode.
2 profiles are in complain mode.
0 processes are unconfined but have a profile defined.
Meaning: AppArmor is actively enforcing profiles.
Decision: If the host is Ubuntu/Debian-ish, AppArmor is a prime suspect. Next: check container profile.
Task 4: Check what AppArmor profile Docker applied to a container
cr0x@server:~$ docker inspect --format '{{.Name}} -> AppArmor={{.AppArmorProfile}}' web-1
/web-1 -> AppArmor=docker-default
Meaning: The container is confined under docker-default.
Decision: If you see AppArmor denials, you’ll be editing/replacing that profile or overriding per-container.
Task 5: Confirm the exact mount and where Docker thinks it is
cr0x@server:~$ docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' web-1
bind /srv/web/data -> /var/lib/app/data
volume web-cache -> /var/cache/app
Meaning: You have both a bind mount and a named volume. SELinux/AppArmor behavior differs between them.
Decision: Focus on the bind mount first; it’s usually where labeling/path policy bites.
Task 6: Reproduce with a minimal write test inside the container
cr0x@server:~$ docker exec -it web-1 sh -lc 'id; touch /var/lib/app/data/.probe && echo ok'
uid=1000(app) gid=1000(app) groups=1000(app)
touch: cannot touch '/var/lib/app/data/.probe': Permission denied
Meaning: The application user cannot create a file in the mounted directory.
Decision: If host permissions look fine, proceed to MAC evidence (AVC/AppArmor) immediately.
Task 7: Check classic host permissions anyway (because humans)
cr0x@server:~$ ls -ldn /srv/web/data
drwxrwx--- 5 1000 1000 4096 Jan 3 10:12 /srv/web/data
Meaning: Mode and ownership match the container user (UID/GID 1000).
Decision: Stop arguing about chown. This is almost certainly SELinux/AppArmor (or userns/shiftfs, but start with MAC).
Task 8: On SELinux hosts, inspect the context of the host directory
cr0x@server:~$ ls -ldZ /srv/web/data
drwxrwx---. 5 1000 1000 unconfined_u:object_r:default_t:s0 /srv/web/data
Meaning: The directory is labeled default_t, the “I don’t know what this is” label. Containers typically can’t touch it.
Decision: Relabel the directory for container access (ideally persistently). Don’t disable SELinux.
Task 9: Find the SELinux denial in audit logs (the smoking gun)
cr0x@server:~$ sudo ausearch -m avc -ts recent | tail -n 5
type=AVC msg=audit(1704286512.911:412): avc: denied { create } for pid=23841 comm="touch" name=".probe" scontext=system_u:system_r:container_t:s0:c123,c456 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0
Meaning: Process domain is container_t; target is default_t; the denied permission is create.
Decision: Fix labeling of /srv/web/data (target context), not container user IDs.
Task 10: Quick temporary relabel using Docker mount flags (:z vs :Z)
cr0x@server:~$ docker run --rm -v /srv/web/data:/data:Z alpine sh -lc 'touch /data/ok && ls -l /data/ok'
-rw-r--r-- 1 root root 0 Jan 3 10:21 /data/ok
Meaning: The relabel worked; the container can write now.
Decision: Decide between :Z (private label for one container) and :z (shared label) based on whether multiple containers need the same path.
Task 11: Make SELinux labeling persistent with semanage fcontext
cr0x@server:~$ sudo semanage fcontext -a -t container_file_t "/srv/web/data(/.*)?"
cr0x@server:~$ sudo restorecon -Rv /srv/web/data
restorecon reset /srv/web/data context unconfined_u:object_r:default_t:s0->unconfined_u:object_r:container_file_t:s0
restorecon reset /srv/web/data/ok context unconfined_u:object_r:default_t:s0->unconfined_u:object_r:container_file_t:s0
Meaning: You’ve declared the expected label mapping and applied it. It will survive relabel operations.
Decision: Prefer this for managed servers. Use chcon only for quick tests.
Task 12: Confirm the new SELinux label
cr0x@server:~$ ls -ldZ /srv/web/data
drwxrwx---. 5 1000 1000 unconfined_u:object_r:container_file_t:s0 /srv/web/data
Meaning: The directory is now labeled to be accessible to containers.
Decision: Re-test container write. If it still fails, inspect other denials: maybe a subpath, socket, or different class.
Task 13: On AppArmor hosts, find the denial in kernel logs
cr0x@server:~$ sudo journalctl -k -g 'apparmor="DENIED"' -n 5
Jan 03 10:24:11 server kernel: audit: type=1400 audit(1704287051.112:96): apparmor="DENIED" operation="open" profile="docker-default" name="/srv/web/data/.probe" pid=24910 comm="touch" requested_mask="wc" denied_mask="wc" fsuid=1000 ouid=1000
Meaning: AppArmor profile docker-default denied write/create (wc) to a host path.
Decision: Either change the profile to allow that path, or choose a different approach (named volume, different mount, or profile override).
Task 14: Identify the container’s init process label (SELinux) or profile (AppArmor) from inside
cr0x@server:~$ docker exec -it web-1 sh -lc 'cat /proc/1/attr/current 2>/dev/null || true; cat /proc/1/attr/apparmor/current 2>/dev/null || true'
system_u:system_r:container_t:s0:c123,c456
docker-default (enforce)
Meaning: You’re seeing confinement metadata from the kernel. (On some hosts you’ll see one or the other.)
Decision: If SELinux says container_t, focus on labels. If AppArmor shows a profile, focus on profile rules.
Task 15: Check Docker daemon security options (SELinux/AppArmor/seccomp)
cr0x@server:~$ docker info --format '{{json .SecurityOptions}}'
["name=seccomp,profile=builtin","name=selinux","name=apparmor"]
Meaning: Docker is aware of SELinux and AppArmor; both are in play on this host.
Decision: Don’t assume “we’re an AppArmor shop” or “we’re an SELinux shop.” Your fleet can be mixed.
Task 16: Check if a named volume avoids the bind mount problem
cr0x@server:~$ docker run --rm -v web-cache:/cache alpine sh -lc 'touch /cache/ok && ls -l /cache/ok'
-rw-r--r-- 1 root root 0 Jan 3 10:28 /cache/ok
Meaning: The named volume works because Docker provisions it under a location already labeled/profile-allowed for containers.
Decision: Prefer named volumes unless you truly need host-path semantics (backups, audits, shared host tools).
Task 17: Verify mount options and filesystem type (NFS and friends get spicy)
cr0x@server:~$ findmnt -T /srv/web/data
TARGET SOURCE FSTYPE OPTIONS
/ /dev/mapper/rootvg ext4 rw,relatime,seclabel
Meaning: The filesystem supports SELinux labels (seclabel). Good.
Decision: If you don’t see seclabel (or you’re on NFS/CIFS), plan for special handling and test carefully.
Task 18: For overlay2 weirdness, check the container’s merged directory label (SELinux hosts)
cr0x@server:~$ cid=$(docker inspect -f '{{.Id}}' web-1)
cr0x@server:~$ sudo ls -ldZ /var/lib/docker/overlay2/*/merged 2>/dev/null | head -n 2
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c87,c912 4096 Jan 3 10:30 /var/lib/docker/overlay2/3a3d.../merged
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c21,c333 4096 Jan 3 10:30 /var/lib/docker/overlay2/9b71.../merged
Meaning: Overlay merged dirs have container labels with MCS categories. That’s expected on SELinux hosts.
Decision: If these labels are wrong or missing, you’re in “daemon/storage driver/policy mismatch” territory, not “chmod it” territory.
Bind mounts, volumes, and labeling: the real mechanics
Why bind mounts are where good intentions go to die
A named Docker volume is created under Docker’s managed directories, with the right default security context (SELinux)
or with paths already permitted (AppArmor), depending on distro defaults. A bind mount is a raw host path you chose.
The kernel doesn’t care that you chose it “for the container.” It’s just a host path with whatever label/profile
implications it already has.
This is the central operational distinction: volumes tend to “just work,” bind mounts often fail in confusing ways
because they cross security domains.
SELinux: the label is the permission, not the mode
With SELinux, your container process is running in a domain like container_t plus an MCS category set.
Files under /var/lib/docker (or the container storage root) are labeled to match that domain and often
include matching categories. Your random host path under /srv might be default_t or
var_t or some application type. Policy may forbid container access outright.
The fix is not “make it world-writable.” The fix is “apply an SELinux type the container domain is allowed to use.”
Common approaches:
- Use Docker mount labels:
:zfor shared content,:Zfor private content. - Set persistent labels:
semanage fcontext+restorecon, usingcontainer_file_t. - Use container-selinux tooling (varies by distro) to keep policy aligned with Docker behavior.
What :z and :Z actually do (and why they surprise people)
The :z/:Z options tell Docker to relabel the source path so the container can access it.
:Z typically gives the content a private label (including categories) for a single container. :z
makes it shareable among containers. If you slap :Z on a path that multiple containers use, one of them
may suddenly lose access. That’s not Docker being fickle; that’s you asking for private labels on shared content.
Another sharp edge: relabeling a host directory can have side effects for non-container processes. SELinux labels are
a system-wide truth, not a per-container wish. If that directory is used by another service with strict expectations,
you can break it.
AppArmor: paths and capabilities are the battlefield
AppArmor denials often look like:
- Cannot write to a mounted path (profile doesn’t allow it).
- Cannot
mountinside container (capability denied or mount rule denied). - Cannot access
/procor/sysfeatures an app expects. - Cannot use privileged operations even when running as root in the container (capabilities filtered).
The fix is generally to adjust a profile (allow the specific path) or to run with a different profile. The lazy fix
is “unconfined,” which is sometimes acceptable for a controlled troubleshooting window and almost never acceptable as
the steady state in production.
A note on NFS, CIFS, FUSE, and other networked fun
Network filesystems complicate this:
- Some mounts don’t support SELinux labels the way local filesystems do, or they need explicit mount options.
- Root-squash on NFS can turn “container root” into “nobody,” and then you’ll chase the wrong culprit.
- AppArmor may still deny paths regardless of the underlying filesystem.
When containers can’t write to NFS, check both MAC and NFS export rules. It’s rarely just one thing.
AppArmor profiles: what Docker does, what your distro does
docker-default is a compromise, not a promise
Docker’s default AppArmor profile is intended to block some obviously dangerous things while still letting most
containers run. It’s not tailored to your app, your mounts, or your compliance regime. It’s a generic seatbelt.
Useful, but not custom-fitted.
When you hit an AppArmor denial, you have choices:
- Change the workload: prefer named volumes; avoid weird mounts; reduce privileged behavior.
- Change the profile: allow the exact paths and operations needed.
- Change the container configuration: override the profile for that container.
Diagnosing AppArmor: what the denial tells you
The denial line usually includes:
- profile= which profile blocked it
- operation= open/mount/ptrace/etc.
- name= the path
- requested_mask / denied_mask what kind of access was attempted
Treat it as a targeted puzzle. If it’s denying write to a specific host path, either don’t mount that path, or
explicitly allow it. If it’s denying mount/capabilities, consider whether you actually need them. Most apps don’t.
Overriding AppArmor: use it like a scalpel
Docker supports overriding AppArmor per container via security opts. This can be a lifesaver for a one-off vendor
image that needs a specific capability, but it’s also how people accidentally run production as effectively “no MAC.”
The operational rule: if you override profiles, bake that decision into infrastructure code and threat model it. Do
not let it be a tribal workaround that lives in a wiki nobody reads.
Joke #2: AppArmor is like a corporate travel policy—your trip is “approved” until you try to expense the taxi.
Three corporate mini-stories from the trenches
Incident 1: the wrong assumption (“permissions are permissions”)
A mid-size company migrated a bunch of services from VMs to containers. They were careful about UID/GID mapping and
even standardized on running apps as non-root. Everything looked clean. Then one service started failing only on the
new RHEL-based hosts, while it ran fine on the older Ubuntu fleet.
The on-call did the usual: checked ownership, ran chmod, redeployed. The service still couldn’t write to
its data directory. After two hours, someone suggested “maybe SELinux,” which was met with the kind of silence you
get when you say “maybe it’s DNS.”
They found AVC denials: the bind-mounted directory was labeled default_t. The container domain was
container_t. SELinux was doing its job. The wrong assumption was that file permissions were the whole
story, so they kept adjusting the wrong knob.
The fix was simple: define persistent fcontext rules for the application’s host paths and apply them with
restorecon. The real improvement was procedural: the team added a host validation step in provisioning
that asserts label correctness for known bind mounts.
The post-incident action item was blunt: “Stop using chmod as a debugging strategy.” They printed it on a sticker
and stuck it on a laptop. It was only mildly effective, but morale improved.
Incident 2: an optimization that backfired (shared bind mounts + :Z)
Another organization ran a cluster of containers that shared a host directory for generated assets. Think thumbnails,
compiled front-end bundles, that sort of thing. They wanted faster rollouts and fewer duplicated artifacts, so they
bind-mounted the same host path into multiple containers across a node and called it “efficient.”
A security-minded engineer noticed occasional SELinux denials and decided to “fix it properly” by adding
:Z to the mount in the Compose file. It worked in development: one container, one directory, no issues.
They rolled it out.
Production got weird. Half the containers could write, half couldn’t, and which half changed after restarts.
Sometimes a deployment “fixed itself” when a container landed on a different node. The team spent a day suspecting
the storage backend, then Docker, then the application.
The root cause: :Z applies a private label intended for a single container’s MCS categories. When
multiple containers share the path, the private label matches one container’s categories and not the others. It’s
not deterministic across restarts because categories can differ.
They switched to :z for truly shared content, and for a subset of cases, moved to named volumes to avoid
host-path coupling. The “optimization” didn’t just backfire; it created a failure mode that looked like flaky I/O
and wasted a lot of senior time. The lesson was simple: in SELinux land, sharing is a policy decision, not a mount
convenience.
Story 3: boring but correct practice that saved the day (audit logs + runbooks)
A larger enterprise had a mixed fleet: some hosts with SELinux enforcing, some with AppArmor, a few hardened boxes
with both, and a steady flow of vendor containers that assumed they could do whatever they wanted.
They invested in a deeply unsexy practice: centralizing audit logs and teaching engineers to read them. Not a
compliance checkbox—an operational capability. They had a runbook: “permission denied in container” maps to
“check Docker mounts, then SELinux/AppArmor denials, then UID/GID.” Everyone had the same steps.
One night, a storage-adjacent service failed after a routine host patching wave. The app logs showed nothing useful.
The container logs showed “permission denied” on a data path. The on-call followed the runbook, found AVC denials,
and saw the target path labeled incorrectly after a filesystem migration.
Because they had persistent fcontext rules checked into their host configuration, fixing it was a restorecon
and a rollback of one misapplied mount. Service restored quickly. No heroics. No “temporarily set SELinux to permissive.”
Just a system behaving predictably.
The saving grace wasn’t brilliance. It was repeatability. The team didn’t “remember” SELinux; their tooling and
runbooks remembered it for them.
Common mistakes: symptom → root cause → fix
1) Symptom: container can’t write to bind mount, but host permissions are correct
Root cause: SELinux context on the host directory is not container-accessible (often default_t).
Fix: Use :z/:Z on the mount, or set persistent labels with
semanage fcontext -a -t container_file_t and restorecon.
2) Symptom: works on Ubuntu host, fails on RHEL/Fedora host
Root cause: SELinux enforcing on one host class, AppArmor (or none) on the other. Your manifests assume the wrong baseline.
Fix: Detect and codify host security configuration; add SELinux labeling to bind mounts on SELinux hosts.
3) Symptom: only one of several containers can write to a shared directory
Root cause: Using :Z (private label) for shared content; MCS categories don’t match across containers.
Fix: Use :z for shared mounts, or stop sharing and use per-container paths/volumes.
4) Symptom: “permission denied” when accessing a UNIX socket (e.g., /var/run/…)
Root cause: SELinux type on the socket file forbids container domain, or AppArmor profile blocks the path.
Fix: Avoid mounting privileged host sockets when possible; otherwise label/allow the socket path specifically. Consider a proxy service rather than direct socket mounts.
5) Symptom: container fails when trying to mount or use FUSE
Root cause: AppArmor denies mount operations or required capabilities; SELinux may deny device access.
Fix: Re-evaluate necessity. If required, create a tailored AppArmor profile and explicitly allow needed operations; avoid blanket unconfined.
6) Symptom: after reboot or relabel, the problem returns
Root cause: You used chcon or Docker relabeling as an ad-hoc fix without persistent fcontext rules; or provisioning recreates directories with default labels.
Fix: Use semanage fcontext + restorecon, and enforce directory creation via config management.
7) Symptom: you set SELinux to permissive and everything “works”
Root cause: You proved it was SELinux, then stopped at the most expensive solution.
Fix: Put SELinux back to enforcing and fix labels/policy. Permissive is for diagnosis windows, not production comfort.
8) Symptom: named volume works, bind mount fails, same path inside container
Root cause: Docker-managed volume path is already labeled/allowed; arbitrary bind mount isn’t.
Fix: Prefer named volumes. Use bind mounts only when host integration is required, and then label/allow correctly.
Checklists / step-by-step plan
Checklist A: when a container can’t write to a mount
- Identify the mount type and host path:
docker inspectmounts. - Reproduce with a simple
touchin the container to isolate app logic from permissions. - Check host mode/owner quickly (
ls -ldn) to rule out obvious UID mismatch. - Check SELinux state (
getenforce) and label (ls -ldZ) if enforcing. - Search for AVC denials (
ausearch -m avc) that match the path and operation. - If AppArmor is active, search kernel logs for
apparmor="DENIED"and identify the profile. - Apply the smallest fix: SELinux label adjustment or AppArmor profile change.
- Re-test with the minimal write probe, then with the real app.
- Make it persistent:
semanage fcontext/restoreconor profile deployment via config management. - Write down the expectation in the service manifest (“this bind mount requires container_file_t”).
Checklist B: hardening without breaking everything
- Prefer named volumes for app state unless you need a host path.
- Standardize on a small set of bind mount roots (
/srv/containers/<app>) and label them consistently. - Centralize audit and kernel logs; alert on bursts of AVC/AppArmor denials.
- Keep SELinux enforcing and AppArmor enforcing; treat exceptions as change-controlled.
- Don’t cargo-cult
--privileged. If you need one capability, add one capability. - In CI, run a “permission probe” container against expected mounts to catch label/profile drift early.
Checklist C: migration day (when you move hosts or storage)
- Before moving data directories, record contexts:
ls -lZon old path. - After migration, apply fcontext rules and
restoreconon new path. - Verify with a minimal container that does create/read/delete operations.
- Only then roll production traffic. This is not superstition; it’s avoiding a 2 a.m. surprise.
One quote that aged well in operations: “Hope is not a strategy.” — General Gordon R. Sullivan
FAQ
1) Why does Docker show “permission denied” instead of “SELinux denied”?
Because the kernel returns a generic access error to the syscall. The app and Docker see EACCES. The
real details live in SELinux/AppArmor logs, not in the application error message.
2) Should I disable SELinux or AppArmor to make containers work?
No, not as a steady-state fix. Use permissive/complain mode briefly to confirm suspicion, then fix labeling/profile
rules. Disabling MAC turns subtle container escapes into loud incidents later.
3) What’s the difference between :z and :Z on Docker mounts?
On SELinux hosts, they trigger relabeling. :z is intended for shared content (multiple containers).
:Z is intended for private content (one container). Using :Z on shared directories causes
cross-container failures.
4) I used chcon and it worked. Why did it break later?
chcon changes labels but doesn’t make them persistent across relabel operations or some provisioning
flows. Use semanage fcontext to define a rule, then apply it with restorecon.
5) Why does a named Docker volume work when a bind mount fails?
Named volumes live under Docker’s storage directories and inherit labels/paths that policies already expect for
container use. Bind mounts inherit whatever label/path policy the host path already has.
6) Can I run containers unconfined under AppArmor?
You can, but you should treat it like running without guardrails. Sometimes it’s a temporary diagnostic move or a
deliberate exception for a tightly controlled host. It should not be the default.
7) Why does this show up only after a host patch or OS upgrade?
Policy packages, default profiles, and labeling behavior can change with updates. Also, storage migrations and
directory recreation can reset contexts to defaults. The fix is to codify labels/profiles, not rely on “whatever the OS does.”
8) Is this the same problem as user namespace remapping or rootless Docker?
Different layer. Userns/rootless affects UID/GID mapping and capability boundaries. SELinux/AppArmor are LSM policy
layers. You can have both issues at once, which is why you must check evidence (audit logs) rather than guessing.
9) How do I decide between changing policy and changing the container design?
If you’re repeatedly carving holes in MAC policy for convenience, redesign. Prefer named volumes, avoid host sockets,
reduce privileges. If you have a legitimate integration need (e.g., shared host directory for backups), then make a
precise policy change and document it.
Conclusion: practical next steps
The permission errors nobody explains are rarely mysterious. They’re just reported badly. SELinux and AppArmor sit
below your container runtime and don’t negotiate with your chmod. They enforce policy, and the details
are in audit trails and kernel logs.
Next steps you can do this week, without starting a holy war:
- Teach your on-call runbook to check SELinux/AppArmor before touching file modes.
- Standardize bind mount roots and apply persistent SELinux fcontext rules where relevant.
- Prefer named volumes unless host-path semantics are truly required.
- Centralize audit and kernel denial logs so “permission denied” becomes a two-minute diagnosis, not a two-hour argument.
- When you must override confinement, do it intentionally, minimally, and in code—never as an emergency fossil that lives forever.
Containers are already complicated. Don’t make them spooky. Make them observable, labeled correctly, and boring.
Boring is what stays up.