You bind-mount a host directory into an LXC container and everything looks fine—until the container tries to write, chmod, chown, or even list the directory.
Then the logs politely inform you: permission denied. Production systems love polite failures because they waste your time with dignity.
This problem is rarely “a Linux permission issue” in the generic sense. In Proxmox, it’s usually a specific, predictable mismatch between
unprivileged container ID mapping and host-side ownership/ACLs, often seasoned with network storage semantics.
The fix is not random chmod therapy. The fix is to align identities.
What’s actually happening (and why your eyes lie)
Proxmox LXC unprivileged containers are a security feature: root inside the container is not root on the host.
It’s mapped to a high, unprivileged UID on the host, typically starting at 100000.
That is the entire point: if the container is compromised, “root” can’t write to /etc on the host, load kernel modules, or casually trash your data.
Bind mounts (Proxmox mp0, mp1, etc.) punch a controlled hole in that isolation. But they don’t magically translate file ownership.
If the host directory is owned by root:root (0:0) and the container root is mapped to 100000:100000, then inside the container the mount may look
like it’s owned by root, but the underlying permission checks happen against the mapped host IDs.
That’s why “but it’s owned by root inside the container” is a trap. Your container root is a regular user on the host. A very high-numbered regular user.
Jargon translation: the error is not about “mounting.” It’s about identity and authorization at the VFS boundary.
One short joke, because we’re about to talk about UID offsets for 30 minutes: bind mounts are like office keycards—your badge works great until you try the server room.
Fast diagnosis playbook
If you’re in the middle of an incident, don’t wander. Check these in order. This sequence finds the bottleneck quickly and minimizes “I changed three things and it got worse.”
1) Confirm the container is unprivileged and note the ID map
- If it’s unprivileged, assume UID/GID mapping is the issue until proven otherwise.
- If it’s privileged, permissions are still a thing, but the failure modes are different.
2) Identify the host path behind mp0 and inspect host ownership/ACLs
- Host-side ownership matters more than container-side ownership for bind mounts.
- ACLs can override chmod expectations, and network filesystems add their own rules.
3) Test write and chown behavior from inside the container
- If writes fail but reads work: ownership/ACL mismatch.
- If chown fails but writes work: you probably don’t need chown; fix the app or set correct ownership once.
4) Check storage backend semantics (NFS root_squash, SMB, ZFS datasets)
- NFS with root_squash is famous for turning “root” into “nobody,” especially in containers.
- ZFS datasets can enforce their own ACL mode and permission inheritance.
5) Only then consider “workarounds”
- Switching to a privileged container is the nuclear option. It solves symptoms by removing a safety feature.
- Random chmod -R 777 is not a fix; it’s a confession.
Interesting facts and context (the stuff you only learn after the outage)
- UID namespaces landed in the Linux kernel years before most distros enabled them by default; container tech was waiting on security and tooling to catch up.
- LXC predates Docker in mainstream use; it’s closer to “system containers” than “single-process app containers.” Different ergonomics, different pain.
- Proxmox picked unprivileged containers as a default recommendation because they reduce host blast radius drastically for the common “web app got popped” scenario.
- The default Proxmox mapping often starts at
100000and spans65536IDs. That number is not magic; it’s a typical subordinate ID range size. - POSIX permissions are evaluated on the host IDs after mapping. What you see inside the container is a translation layer, not the host’s truth.
- NFS root_squash was invented to prevent remote root users from becoming root on the NFS server. Containers don’t get a special exemption.
- Shiftfs existed as an out-of-tree Ubuntu solution to the UID-shifting pain. It’s not the mainstream answer today, and relying on it long-term is a bet.
- idmapped mounts are a newer kernel feature that can solve this cleanly, but your Proxmox/kernel version and tooling support decide whether it’s practical.
- ACLs are older than many engineers think in Linux enterprise environments; they’re not exotic, and they often explain “chmod says yes but it still says no.”
The mental model: UID/GID mapping, idmap ranges, and bind mounts
Unprivileged LXC works by using a user namespace. Inside the container, processes run with “container UIDs.”
The kernel maps those to “host UIDs” before doing permission checks on host filesystems.
Typical mapping looks like this:
- Container UID 0 (root) → Host UID 100000
- Container UID 1 → Host UID 100001
- …
- Container UID 65535 → Host UID 165535
So if you bind-mount /tank/shared from the host into /mnt/shared in the container, and the host directory is owned by root:root,
then container root is not the owner in host terms. Container root is host UID 100000. Host root is UID 0.
That mismatch produces the classic errors:
- Write fails: the directory is not writable by host UID 100000.
- chown fails: unprivileged container can’t arbitrarily change ownership on host filesystems.
- chmod appears to work or doesn’t: depends on mapping, filesystem, and whether ACLs deny it anyway.
There are basically four production-grade ways out:
- Make host ownership match the mapped IDs (the usual, boring, correct answer).
- Use ACLs on the host to grant access to the mapped IDs without changing ownership of everything.
- Use idmapped mounts (if your stack supports it reliably).
- Don’t bind-mount host paths; instead, use storage designed for sharing (NFS/SMB with explicit user mapping, or a dedicated dataset per container with correct ownership).
Switching to a privileged container is the “fifth way,” but it’s more like pulling the fire alarm to turn off the microwave. It works, and everyone will remember it.
Practical tasks: commands, expected output, and what decision to make
These are the hands-on checks I actually run. Each task includes: a command, what the output means, and what you do next.
Run host-side commands on the Proxmox node, container-side commands inside the LXC.
Task 1: Verify the container is unprivileged
cr0x@server:~$ pct config 103 | egrep 'unprivileged|lxc.idmap|mp0|mp1'
unprivileged: 1
mp0: /tank/shared,mp=/mnt/shared
Meaning: unprivileged: 1 confirms you have UID/GID mapping in play.
If you see lxc.idmap lines, the mapping is customized and you must respect it.
Decision: Continue assuming a mapping mismatch until you prove access is granted to the mapped host IDs.
Task 2: Inspect the container’s ID mapping from the host
cr0x@server:~$ cat /etc/pve/lxc/103.conf | egrep 'unprivileged|lxc.idmap'
unprivileged: 1
Meaning: No custom idmap lines usually means the default subordinate ranges apply.
Decision: Check the node’s subuid/subgid ranges next; that determines the host UID offset.
Task 3: Check subordinate UID/GID ranges on the Proxmox node
cr0x@server:~$ grep -E '^root:' /etc/subuid /etc/subgid
/etc/subuid:root:100000:65536
/etc/subgid:root:100000:65536
Meaning: Container UID 0 maps to host UID 100000, and you have a 65536-ID range.
Decision: Any host directory you bind-mount must be writable by host UID/GID 100000 (or by whatever host IDs map to the container user your app runs as).
Task 4: Find the host path for the mount point
cr0x@server:~$ pct config 103 | grep '^mp'
mp0: /tank/shared,mp=/mnt/shared
Meaning: The host path is /tank/shared. That is the path whose ownership and ACLs must permit the mapped IDs.
Decision: Inspect host ownership and permissions on that path (not inside the container first).
Task 5: Inspect host ownership, mode bits, and ACL presence
cr0x@server:~$ ls -ldn /tank/shared
drwxr-x--- 5 0 0 5 Dec 26 08:41 /tank/shared
Meaning: Owner UID/GID is 0:0 (host root). Mode 750 means only owner and group can write/read; others can’t.
Host UID 100000 is “others,” so the container root won’t write.
Decision: Either change ownership to 100000:100000, or add an ACL granting host 100000 access, or restructure the storage.
Task 6: Confirm what user the workload runs as inside the container
cr0x@server:~$ pct exec 103 -- ps -eo user,uid,group,gid,comm | head
USER UID GROUP GID COMMAND
root 0 root 0 ps
www-data 33 www-data 33 nginx
Meaning: Your app may not run as root. Here, nginx runs as UID 33 inside the container.
Decision: Map the container’s runtime UID to the host: host UID = 100000 + 33 = 100033. That’s the identity that needs access on the host path.
Task 7: Calculate the mapped host UID/GID for a container user
cr0x@server:~$ echo $((100000 + 33))
100033
Meaning: Container UID 33 corresponds to host UID 100033 (assuming default mapping).
Decision: Grant write permissions on the host to UID 100033 (and maybe GID 100033), not to UID 33.
Task 8: Prove the failure from inside the container (write test)
cr0x@server:~$ pct exec 103 -- bash -lc 'id && touch /mnt/shared/.permtest'
uid=0(root) gid=0(root) groups=0(root)
touch: cannot touch '/mnt/shared/.permtest': Permission denied
Meaning: Classic. The mount is visible, but host permissions deny the mapped ID.
Decision: Fix host ownership/ACLs. Do not “fix” this by chmod inside the container and hoping it persists.
Task 9: Check mount type and options on the host
cr0x@server:~$ findmnt -no SOURCE,FSTYPE,OPTIONS /tank/shared
tank/shared zfs rw,xattr,posixacl
Meaning: ZFS dataset with ACL support enabled. Good. This gives you more controlled options than chmod alone.
Decision: Prefer an ACL grant or dataset-per-container with mapped ownership, rather than widening mode bits globally.
Task 10: Fix ownership on the host to match the mapped IDs (simple case)
cr0x@server:~$ chown -R 100000:100000 /tank/shared
Meaning: Now container root (host 100000) owns the tree. Inside the container, root should be able to write.
Decision: Use this when the directory is dedicated to that container. Avoid it for shared multi-container data unless you’ve designed a shared identity plan.
Task 11: Alternatively, grant access via ACLs to the mapped user (less disruptive)
cr0x@server:~$ setfacl -m u:100033:rwx /tank/shared
cr0x@server:~$ getfacl -p /tank/shared | sed -n '1,12p'
# file: /tank/shared
# owner: root
# group: root
user::rwx
user:100033:rwx
group::r-x
mask::rwx
other::---
Meaning: Host user 100033 (container www-data) has full access, without changing root ownership.
Decision: Choose ACLs when you need host root to remain the owner or when multiple identities need controlled access. Remember to set default ACLs for new files if needed.
Task 12: Set default ACLs so new files inherit permissions (common “it worked yesterday” fix)
cr0x@server:~$ setfacl -d -m u:100033:rwx /tank/shared
cr0x@server:~$ getfacl -p /tank/shared | grep -E '^default:user:100033'
default:user:100033:rwx
Meaning: New files and directories created under /tank/shared inherit the ACL entry.
Decision: If your app creates subdirectories and suddenly can’t write to them, default ACLs are often the missing piece.
Task 13: Validate from inside the container after changes
cr0x@server:~$ pct exec 103 -- bash -lc 'touch /mnt/shared/.permtest && ls -ln /mnt/shared/.permtest'
-rw-r--r-- 1 0 0 0 Dec 26 09:02 /mnt/shared/.permtest
Meaning: The container can write now. Don’t obsess over the displayed UID inside the container; the host is enforcing correctly via mapping/ACL.
Decision: Move on to app-specific expectations: does it require chown? does it need group-write? handle those explicitly.
Task 14: Diagnose “chown: Operation not permitted” correctly
cr0x@server:~$ pct exec 103 -- bash -lc 'chown 33:33 /mnt/shared/.permtest'
chown: changing ownership of '/mnt/shared/.permtest': Operation not permitted
Meaning: Unprivileged containers generally cannot arbitrarily change file ownership on bind-mounted host paths. This is expected.
Decision: Fix the ownership on the host once, or adjust the application to stop trying to chown at runtime. If the app requires chown, you need to design around it (dedicated dataset and pre-owned tree, or a different sharing method).
Task 15: Check for NFS root_squash (if the host path is an NFS mount)
cr0x@server:~$ findmnt -no SOURCE,FSTYPE,OPTIONS /tank/shared
nas:/export/shared nfs4 rw,relatime,vers=4.2,hard,proto=tcp,timeo=600,retrans=2,sec=sys
Meaning: The “real” permissions are now governed by the NFS server, plus export options like root_squash and idmapping behavior.
Decision: If you suspect root_squash, stop trying to make “root in container” own things. Use a dedicated NFS export with an explicit UID/GID that matches the mapped host IDs, or avoid bind-mounting NFS into unprivileged containers entirely.
Task 16: See what UID the host uses when the container writes (forensics)
cr0x@server:~$ rm -f /tank/shared/hostview.test
cr0x@server:~$ pct exec 103 -- bash -lc 'echo hi > /mnt/shared/hostview.test'
cr0x@server:~$ ls -ln /tank/shared/hostview.test
-rw-r--r-- 1 100000 100000 3 Dec 26 09:06 /tank/shared/hostview.test
Meaning: The file is owned by host 100000:100000 because it was created by container root.
Decision: Use this to validate your mapping assumptions. If you expected 100033 but see 100000, your process is running as root inside the container.
Fix patterns that hold up in production
Pattern A: Dedicated host directory per container, owned by mapped root (fast, clean)
If only one container uses the bind mount, the simplest plan is: create a dedicated directory, then chown it to the container’s mapped root ID.
This is boring. Boring is good.
cr0x@server:~$ mkdir -p /tank/ct103-data
cr0x@server:~$ chown -R 100000:100000 /tank/ct103-data
cr0x@server:~$ chmod 0750 /tank/ct103-data
Why it works: you align the host owner with container root’s mapped identity.
When to avoid: shared data between multiple containers with different UID ranges; you’ll end up with a junk drawer of mismatched ownership.
Pattern B: ACL grants for specific container service users (surgical, scalable)
This is the adult approach for shared directories: keep host ownership meaningful, grant access to mapped IDs via ACLs, set default ACLs for inheritance,
and stop treating chmod as your only tool.
cr0x@server:~$ setfacl -m u:100033:rwx /tank/shared
cr0x@server:~$ setfacl -d -m u:100033:rwx /tank/shared
Why it works: it grants exactly the access required, and it survives new file creation.
Trade-off: you must operationalize ACL visibility. If your team can’t be trusted to run getfacl, you’ll have recurring mysteries.
Pattern C: One dataset per container (especially with ZFS), ownership set once
With ZFS on Proxmox, per-container datasets are clean because they keep quotas, snapshots, and replication sane.
It also reduces ACL complexity because each dataset has one “owner identity story.”
Set dataset, mount to host path, chown to mapped IDs, then bind-mount into container. Keep the data boundary crisp.
Pattern D: idmapped mounts (modern, elegant, but check your platform)
idmapped mounts can present a filesystem tree with a different UID/GID mapping at mount time.
That’s basically “do the shifting at the VFS layer” instead of pre-chowning everything.
In practice, your decision depends on Proxmox version, kernel support, and whether you want to be the first person on your team to debug it at 3 a.m.
If you can’t answer “how do we validate this mapping after a kernel update?” stick with ownership/ACLs.
Pattern E: Stop bind-mounting and use a sharing protocol with explicit identity
Sometimes the correct fix is architectural: don’t bind-mount host directories into unprivileged containers when the backing storage is itself a remote filesystem,
or when multiple containers need shared write access with different users.
Use NFS/SMB inside the container with a well-defined service account and consistent UID/GID across systems. Or expose storage via a service layer.
Bind mounts are fantastic until they aren’t.
Three corporate-world mini-stories (anonymized, painfully plausible)
Mini-story 1: The incident caused by a wrong assumption
A platform team migrated a legacy VM-based workload into Proxmox LXC to save resources. They did the right things—kept it unprivileged, moved configs, set up monitoring.
They also bind-mounted a host directory containing application uploads.
In staging, it “worked.” In production, uploads started failing intermittently. The app threw generic I/O errors, and the on-call engineer did what humans do under stress:
they blamed the application, then the network, then “storage.”
The wrong assumption was simple: “root inside the container is root enough.” They looked at ls -l inside the container,
saw root:root, and assumed the container should be able to write.
Meanwhile, on the host, the directory was owned by 0:0 with 750 permissions.
Container root was host 100000—effectively “other.” Writes failed when the app rotated into new subdirectories with stricter modes.
The fix was one command sequence and one policy change: chown the dedicated upload directory to the mapped service account,
add default ACLs so new folders inherit write permission, and document the mapping rule in the runbook.
The outage postmortem was short, which is the best kind.
Mini-story 2: The optimization that backfired
Another org wanted to reduce duplication. They created a single shared host directory for “common assets” and bind-mounted it into a fleet of containers.
To “make it easy,” they ran a broad chmod -R 777 on the tree and moved on. Nobody likes permission tickets.
Two months later, a compromised container wrote a malicious binary into that shared directory. Another container—completely different team—executed a helper script
from the shared path during a deployment step. It wasn’t a sophisticated attack. It was just a writable shared directory doing exactly what writable shared directories do.
The remediation was not fun. They had to untangle which containers needed read-only access, which needed write access, and which should not have had the mount at all.
They ended up replacing the shared write mount with a read-only mount for most containers and a separate write location per container.
The irony: the “optimization” to save space created a security and reliability liability. It also increased operational load because auditing a world-writable directory
in a multi-tenant container environment is basically a hobby, not a control.
Mini-story 3: The boring but correct practice that saved the day
A financial services team (yes, those people) had a policy: every bind mount must have a ticket with (1) purpose, (2) data owner, (3) access mode, (4) mapped UID/GID,
and (5) rollback steps. Engineers rolled their eyes. Then they followed it anyway.
A kernel update rolled through the cluster, and one container started failing to write to its mount. The on-call opened the ticket,
saw the exact mapped UID/GID and the expected host path permissions, and ran the verification commands.
The host directory’s ACL had been overwritten by a “cleanup” script that normalized permissions across datasets.
Because the expectations were documented, the fix was deterministic: restore the ACL entries and defaults, validate with a write test,
and add a guard to the cleanup script to skip managed mountpoints.
Nobody debated whether to “just make it privileged.” Nobody guessed.
It was boring. It was correct. It prevented a messy escalation and kept the blast radius small.
Boring operational hygiene doesn’t get you promoted in the moment, but it does let you sleep.
Common mistakes: symptom → root cause → fix
1) Symptom: “Permission denied” on touch or writes inside the mount
Root cause: host directory not writable by mapped host UID/GID (often 100000+).
Fix: chown host directory to mapped IDs or grant ACLs to mapped IDs; validate with host-side ls -ln and container write test.
2) Symptom: “Operation not permitted” on chown inside the container
Root cause: unprivileged container cannot change ownership on bind mounts like a real root would.
Fix: set correct ownership on host beforehand; change the app to stop runtime chown; or use per-container datasets and pre-owned paths.
3) Symptom: Works on existing files, fails on newly created subdirectories
Root cause: missing default ACLs or umask creates directories without write permission for the mapped service user.
Fix: set default ACLs on the host path; ensure group ownership and setgid bit if using group-based access patterns.
4) Symptom: NFS-mounted host path behaves strangely; root can’t write even after chown
Root cause: NFS export options (root_squash), or UID mismatch between client and server, or idmapping issues on NFSv4.
Fix: use a dedicated NFS export with explicit service UID/GID matching mapped host IDs; consider mounting NFS inside the container with a consistent identity plan.
5) Symptom: chmod/chown changes “don’t stick” or behave inconsistently
Root cause: ACLs or filesystem-level permission model (ZFS ACL mode, SMB semantics) overriding chmod expectations.
Fix: inspect and manage ACLs intentionally; confirm filesystem ACL settings; avoid mixing POSIX and NFSv4 ACL expectations without a plan.
6) Symptom: You “fixed” it by making the container privileged, and now security is nervous
Root cause: you removed the safety mechanism instead of aligning identities.
Fix: revert to unprivileged if possible; implement ownership/ACL mapping or redesign storage; reserve privileged containers for cases with a written exception and compensating controls.
7) Symptom: Only one node in a cluster fails after migrating the container
Root cause: inconsistent /etc/subuid//etc/subgid ranges across nodes, or inconsistent ACLs/ownership on shared storage.
Fix: standardize subordinate ID ranges across nodes; treat them as cluster configuration. Validate on every node that might run the container.
8) Symptom: Directory appears owned by “nobody” or odd IDs inside the container
Root cause: mapping mismatch, remote filesystem identity mapping, or missing name service inside container.
Fix: reason in numeric IDs; use ls -ln; don’t chase usernames until the numeric mapping is understood.
Checklists / step-by-step plan
Step-by-step: fix a single-container bind mount safely
- On host: confirm
unprivileged: 1for the container. - On host: read
/etc/subuidand get the base (often 100000). - On host: identify
mp0path and inspectls -ldn. - Decide who needs write access inside the container (root vs service user).
- Compute mapped host UID/GID for that user (base + container UID).
- For dedicated directory:
chownhost path to mapped UID/GID; set mode bits conservatively. - For shared directory: add host ACLs for mapped UID/GID; also set default ACLs.
- Validate from inside container: write test and (if needed) directory creation test.
- Validate from host: check file ownership shows mapped IDs, not 0:0.
- Record the mapping and expectations in a runbook entry tied to the container ID and mount point.
Step-by-step: making shared storage not turn into a security dumpster
- Stop using world-writable permissions as a “temporary fix.” Temporary is how you build permanent incidents.
- Create a shared group model if appropriate, but remember mapped GIDs differ in unprivileged containers.
- Prefer read-only bind mounts for shared assets. Write access should be rare and justified.
- Use ACLs with explicit mapped IDs for the few containers that need write access.
- Set default ACLs and test creation of nested directories, not just a single file.
- Implement periodic audits: list ACLs and ownership for managed mount roots.
Second short joke, then back to work: a chmod 777 is like turning off your smoke detector because it keeps beeping—quiet, yes; wise, no.
FAQ
1) Why does it say “permission denied” even when the directory looks like root owns it inside the container?
Because ownership is translated. Container root is mapped to a non-root host UID (often 100000). The host filesystem checks permissions against that host UID.
2) What’s the quickest reliable fix?
If the directory is dedicated to that container: chown it on the host to the mapped UID/GID (typically 100000:100000 for container root) and keep permissions tight.
3) How do I map a container user (like www-data UID 33) to the host UID?
Take the subordinate base from /etc/subuid (e.g., 100000) and add the container UID (33). Result: 100033. Same math for GID.
4) Can I just run chown inside the container?
On bind-mounted host paths in unprivileged containers, usually no. You’ll get “Operation not permitted.” Set ownership on the host instead, or use ACLs.
5) Is switching to a privileged container acceptable?
Sometimes, but treat it like an exception with compensating controls. Privileged containers increase host risk substantially. Fixing UID/GID mapping is almost always better.
6) Why does it work on one Proxmox node but not after migrating the container?
Common reason: different /etc/subuid//etc/subgid ranges across nodes. The same container UID maps to different host UIDs, so permissions break.
7) What about NFS? Why is it extra painful?
NFS adds server-side permission enforcement and export options like root_squash. A container’s mapped UID may not match what the NFS server expects.
Solve it by designing consistent service identities or using dedicated exports with explicit UID/GID strategy.
8) Should I use ACLs or chown?
Use chown for dedicated per-container paths. Use ACLs when ownership must remain host-root or when multiple identities need controlled access.
If you need both, you’re probably modeling shared storage—treat it like shared storage, not a hacky bind mount.
9) What’s the role of idmapped mounts here?
They can solve UID/GID translation cleanly at mount time without chowning data. But operational support varies by kernel/tooling.
If you can’t test and monitor it confidently, stick to the proven ownership/ACL approach.
10) How do I keep this from recurring?
Standardize subuid/subgid ranges across the cluster, document mount ownership/ACL expectations per container, and add a simple validation test (create file, create directory).
Conclusion: next steps you can do today
“Permission denied” on a Proxmox LXC bind mount is almost never mysterious. It’s math plus policy:
your container user is mapped to a different host UID/GID, and the host path doesn’t allow it.
Do these next:
- Pick one container with a failing bind mount and write down: container ID, mpX path, unprivileged flag, and subuid base.
- Compute the mapped host UID/GID for the service user that needs access.
- Fix it with either a dedicated chown (single-tenant path) or ACLs + default ACLs (shared path).
- Validate with a write test and a “create subdirectory” test. Future you cares about the subdirectory test.
- Standardize
/etc/subuid//etc/subgidacross nodes before the next migration surprise.
Paraphrased idea from John Allspaw: the real work in operations is designing systems and processes that make failure understandable and recoverable.