Debian/Ubuntu Web Root Permissions: Stop 403s Without Making Everything 777 (Case #9)

Fast diagnosis playbook

When you see a 403, you’re hunting for one of three bottlenecks: the web server config denies it,
the filesystem denies it, or an intermediate layer (chroot, container mount, AppArmor) denies it.
Don’t guess. Prove.

First: confirm which layer is denying

• Check the web server error log for “permission denied”, “access forbidden by rule”, or “directory index forbidden”.
• Check the request mapping: what on-disk path is it trying to serve?
• Reproduce locally with the web server’s effective user (www-data on Debian/Ubuntu by default).

Second: walk the path, not just the file

Most permission failures aren’t on the file. They’re on a parent directory you forgot existed.
Every directory in the path needs the execute bit for traversal. Not “read.” Execute.

Third: check for “policy” blockers

• AppArmor profiles (Ubuntu) can deny reads even when mode bits look fine.
• Container volume mounts can change ownership/modes in surprising ways.
• Symlinks may point outside allowed roots (Apache) or into directories without traversal.

If you do those three steps, you’ll stop treating 403s like a supernatural event.

The permission model that actually works (and doesn’t end in 777)

On Debian/Ubuntu, your web server worker processes typically run as www-data.
Your deploy user is typically deploy or a CI runner. The mistake is trying to make one
set of permissions satisfy both “deploy writes everything” and “web server reads everything” by
turning the whole tree into a public restroom.

Here’s the sane model I recommend for most shops:

Model A: read-only web root, explicit writable subdirs

• Web root (/var/www/site) is owned by a deployment owner (or root), readable by www-data, not writable by it.
• Writable directories are carved out: var/, storage/, uploads/, cache directories, sockets if needed.
• Writables are owned by www-data (or have an ACL granting www-data write), and nothing else is.

That gets you predictable security properties: compromise of the web server doesn’t automatically
grant the ability to modify application code. It can still write user uploads because you decided
it can. Not because you panicked.

Model B: shared group for deployments (when you must allow writes)

• Create a group (e.g. web) that includes the deploy user and www-data.
• Set group ownership of the web root to web.
• Use chmod 2775 (setgid) on directories so new files inherit the group.
• Set a sane umask in your deployment process so files are group-readable (and group-writable if required).

Model B is more flexible, but it has sharp edges: one bad umask and you’ve built a permission lottery.
If you’re going to do it, do it with discipline and checks.

Exactly one quote, because the rest should be logs: “Hope is not a strategy.” — Gene Kranz

Joke #1: chmod 777 is like fixing a broken window by removing the entire wall. Technically there’s no glass left to break.

Facts & historical context (so the weird bits make sense)

• The execute bit on directories originally meant “search”: you may traverse into the directory and access inodes if you already know names.
• 403 vs 404 is often a choice: many servers intentionally return 404 on permission issues to avoid revealing existence of content.
• www-data as a convention became common in Debian-derived systems because packages needed a predictable non-login user for daemons.
• Apache’s historical stance on symlinks is conservative: following symlinks and allowing overrides has been a recurring source of security incidents.
• Nginx popularized “try_files” patterns that can change what path is actually accessed, confusing permission troubleshooting if you inspect the wrong file.
• POSIX ACLs were introduced to express permissions beyond owner/group/other, largely driven by enterprise NFS and multi-user systems where Unix bits were too blunt.
• Setgid on directories is old-school but still relevant: it’s one of the few portable ways to force group inheritance without depending on ACL defaults.
• umask defaults came from multi-user timesharing: default-writable-by-group was considered risky even when people were nicer.

Practical tasks: commands, outputs, decisions

These are “run this now” tasks. Each includes what the output means and the decision you should make.
Use them in order when you’re under pressure, and again later when you’re fixing the system properly.

Task 1: confirm the service user and active worker identity

cr0x@server:~$ ps -o user,group,pid,cmd -C nginx | head
USER     GROUP    PID CMD
root     root    1021 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data www-data 1022 nginx: worker process
www-data www-data 1023 nginx: worker process

Meaning: Workers run as www-data. That’s the identity that needs filesystem access.

Decision: All “can it read this?” tests should be run as www-data.

Task 2: get the on-disk path from the web server config

cr0x@server:~$ sudo nginx -T 2>/dev/null | sed -n '/server_name example.com/,+40p' | sed -n '1,40p'
server {
    listen 80;
    server_name example.com;
    root /var/www/example/current/public;
    index index.html index.php;
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
}

Meaning: The effective root is /var/www/example/current/public, not the repo checkout you were staring at.

Decision: Troubleshoot permissions on that path and its parents, including the current symlink target.

Task 3: check the error log for the actual denial wording

cr0x@server:~$ sudo tail -n 8 /var/log/nginx/error.log
2025/12/30 10:12:14 [error] 1022#1022: *884 open() "/var/www/example/current/public/index.html" failed (13: Permission denied), client: 203.0.113.10, server: example.com, request: "GET / HTTP/1.1", host: "example.com"

Meaning: Error 13 is kernel-level permission denial. This is not an nginx “deny all” rule.

Decision: Focus on filesystem permissions, ownership, ACLs, and path traversal. Save config debates for later.

Task 4: walk the full path permissions (the boring part that fixes it)

cr0x@server:~$ namei -l /var/www/example/current/public/index.html
f: /var/www/example/current/public/index.html
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root www
drwxr-x--- deploy web  example
lrwxrwxrwx deploy web  current -> releases/20251230T1005
drwxr-s--- deploy web  releases
drwxr-s--- deploy web  20251230T1005
drwxr-s--- deploy web  public
-rw-r----- deploy web  index.html

Meaning: /var/www/example is drwxr-x---. “Other” has no execute bit, and www-data is neither owner nor in group web (yet).

Decision: Either add www-data to group web (Model B), or change directory traversal rights so www-data can traverse but not write (Model A).

Task 5: verify group membership (and remember to restart workers if needed)

cr0x@server:~$ id www-data
uid=33(www-data) gid=33(www-data) groups=33(www-data)

Meaning: www-data is not in web. Group-based access won’t work until you add it.

Decision: If you go with the shared-group model, add membership and reload the service to pick up groups.

Task 6: test access as the web user (no guessing)

cr0x@server:~$ sudo -u www-data test -r /var/www/example/current/public/index.html; echo $?
1

Meaning: Exit code 1 means “not readable.” That matches the log.

Decision: Fix permissions on directories/files so the test returns 0.

Task 7: check directory execute bit specifically (common path trap)

cr0x@server:~$ sudo -u www-data ls -ld /var/www/example /var/www/example/current /var/www/example/current/public
ls: cannot access '/var/www/example/current': Permission denied
drwxr-x--- 5 deploy web 4096 Dec 30 10:05 /var/www/example

Meaning: The denial happens at /var/www/example traversal. Even if files were world-readable, you can’t reach them.

Decision: Grant traversal (execute) to www-data via group membership, ACL, or “other” execute where acceptable.

Task 8: implement Model B safely (shared group + setgid)

cr0x@server:~$ sudo groupadd -f web
cr0x@server:~$ sudo usermod -aG web www-data
cr0x@server:~$ sudo usermod -aG web deploy
cr0x@server:~$ sudo chgrp -R web /var/www/example
cr0x@server:~$ sudo find /var/www/example -type d -exec chmod 2775 {} \;
cr0x@server:~$ sudo find /var/www/example -type f -exec chmod 0644 {} \;

Meaning: Directories become group-inheriting (2 in 2775), readable/traversable by group. Files become readable by group.

Decision: Reload nginx so worker processes pick up the new supplementary group; then retest access as www-data.

Task 9: reload the service to pick up new groups (don’t skip this)

cr0x@server:~$ sudo systemctl reload nginx
cr0x@server:~$ ps -o pid,user,group,cmd -C nginx | head -n 4
  PID USER     GROUP    CMD
 1021 root     root     nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
 2110 www-data www-data nginx: worker process
 2111 www-data www-data nginx: worker process

Meaning: The primary group remains www-data (fine). Supplementary groups aren’t shown here.

Decision: Verify supplementary groups by testing access again; if it still fails, the worker may not have the new group (restart instead of reload).

Task 10: re-test access as www-data after the change

cr0x@server:~$ sudo -u www-data test -r /var/www/example/current/public/index.html; echo $?
0

Meaning: Read access works now.

Decision: Confirm the HTTP response is 200 and that you didn’t accidentally make writable areas too broad.

Task 11: inspect umask that may sabotage future deploys

cr0x@server:~$ sudo -u deploy sh -lc 'umask; touch /var/www/example/current/public/.umask-test && stat -c "%a %U:%G %n" /var/www/example/current/public/.umask-test'
0022
644 deploy:web /var/www/example/current/public/.umask-test

Meaning: With umask 0022, new files are 0644. That’s fine for read-only web roots. If you needed group-write, this would break it.

Decision: Prefer read-only roots. If you truly need group-write, set umask to 0002 in the deployment environment and prove it stays that way.

Task 12: identify writable directories (and lock everything else down)

cr0x@server:~$ sudo find /var/www/example/current -maxdepth 3 -type d -name storage -o -name uploads -o -name cache
/var/www/example/current/storage
/var/www/example/current/public/uploads
/var/www/example/current/var/cache

Meaning: These are the typical “app needs write” paths.

Decision: Make only these writable by www-data. Everything else should be read-only to the web server.

Task 13: set explicit ownership for writable dirs (Model A hybrid)

cr0x@server:~$ sudo chown -R www-data:www-data /var/www/example/current/storage /var/www/example/current/public/uploads /var/www/example/current/var/cache
cr0x@server:~$ sudo find /var/www/example/current/storage /var/www/example/current/public/uploads /var/www/example/current/var/cache -type d -exec chmod 0750 {} \;
cr0x@server:~$ sudo find /var/www/example/current/storage /var/www/example/current/public/uploads /var/www/example/current/var/cache -type f -exec chmod 0640 {} \;

Meaning: The app can write where it must, but can’t mutate code elsewhere.

Decision: If your deploy user needs to manage these directories too, use ACLs instead of loosening modes.

Task 14: verify no accidental world-writable permissions exist

cr0x@server:~$ sudo find /var/www/example -xdev -perm -0002 -type f -o -perm -0002 -type d | head

Meaning: No output is good. Output means something is world-writable, which is usually an incident waiting for a timeline.

Decision: If output exists, fix ownership/ACL and remove world-write. Don’t negotiate with it.

Task 15: diagnose AppArmor denials on Ubuntu (when modes look fine)

cr0x@server:~$ sudo journalctl -k -g DENIED -n 5
Dec 30 10:14:02 server kernel: [12345.678901] audit: type=1400 apparmor="DENIED" operation="open" profile="/usr/sbin/nginx" name="/srv/apps/example/current/public/index.html" pid=2110 comm="nginx" requested_mask="r" denied_mask="r" fsuid=33 ouid=1001

Meaning: The kernel denied the read due to AppArmor policy. Filesystem bits are not the problem.

Decision: Adjust the AppArmor profile (or move the web root to an allowed path). Don’t keep chmod’ing; it won’t help.

Task 16: catch symlink surprises (Apache especially)

cr0x@server:~$ readlink -f /var/www/example/current/public
/var/www/example/releases/20251230T1005/public

Meaning: You’re serving from a releases directory. Permissions must be consistent across each new release path.

Decision: Bake permission normalization into the deploy step (or set default ACLs on parent dirs) so every release is correct.

Joke #2: Unix permissions are simple—until you need them to be correct.

Nginx vs Apache: different defaults, same filesystem reality

Nginx failure modes that look like permissions

Nginx is usually blunt about filesystem denials: you’ll see “(13: Permission denied)” in the error log.
But config can still create “permission-ish” symptoms:

• try_files changes the target: you request / but nginx tries /index.php. Permissions must work on the fallback too.
• autoindex off + missing index file: you may get 403 if directory listing is forbidden and no index exists.
• alias vs root mistakes: wrong path concatenation can point you somewhere you didn’t permission.

Apache failure modes that look like permissions

Apache has its own brand of “403 because policy,” even when the filesystem is fine:

• Require directives: a strict Require all denied inherited from a parent context will happily 403 everything.
• Options FollowSymLinks: symlinks may be rejected, producing 403 while the target is readable.
• .htaccess overrides: missing AllowOverride or unexpected rules can deny requests in ways that mimic file access problems.

The trick is to stop treating “403” as one problem. It’s an output. The input is in the logs.

ACLs: the grown-up alternative to group-write-everything

Unix permission bits are a three-lane highway: owner, group, other. ACLs give you side streets:
“www-data can read this tree, deploy can write, CI can read, and nobody else touches it.” That’s
exactly what web roots often need.

When ACLs are worth it

• You have multiple deploy identities (human deploy user, CI runner, automation) and don’t want a massive shared group.
• You want www-data to have traversal and read, but never write, even if group ownership changes.
• You want new files to automatically get the right permissions without relying on umask discipline.

Two ACL rules that do most of the work

Set an access ACL (what applies now) and a default ACL (what new files inherit) on directories.
Example: allow www-data to read/traverse the web root, and allow deploy to write, without making it world-readable.

cr0x@server:~$ sudo setfacl -R -m u:www-data:rx /var/www/example
cr0x@server:~$ sudo setfacl -R -m d:u:www-data:rx /var/www/example
cr0x@server:~$ getfacl -p /var/www/example | sed -n '1,20p'
# file: /var/www/example
# owner: deploy
# group: web
user::rwx
user:www-data:r-x
group::r-x
mask::r-x
other::---
default:user::rwx
default:user:www-data:r-x
default:group::r-x
default:mask::r-x
default:other::---

Meaning: www-data gets read+traverse. New directories/files inherit a default ACL entry, so new releases won’t randomly 403.

Decision: Use ACLs when you need stable multi-identity access without loosening “other” permissions.

The gotcha: the ACL mask

ACLs come with a mask entry that limits effective permissions for named users/groups. People forget the mask exists,
then wonder why their carefully set rwx doesn’t actually apply.

cr0x@server:~$ sudo setfacl -m u:www-data:rwx /var/www/example/current/storage
cr0x@server:~$ getfacl -p /var/www/example/current/storage | grep -E 'www-data|mask'
user:www-data:rwx
mask::rwx

Meaning: Mask is permissive. If it were r-x, the effective permissions would be reduced.

Decision: When ACLs “don’t work,” inspect the mask before rewriting half your filesystem policy.

Deployments: keep CI/CD from re-breaking permissions

Permissions aren’t something you fix once. They’re something your deployment can ruin repeatedly,
with the enthusiasm of a toddler in a room full of markers.

Where permissions drift comes from

• New release directories created with different umask or user context.
• Artifact extraction preserving modes from build time that don’t match production needs.
• Container builds copying files as root and leaving them root-owned on shared volumes.
• Hotfixes made on the server as root at 2 a.m. (you know who you are).

Normalize permissions as a deployment step

This is boring and correct. Add a step to your deploy pipeline that enforces ownership/modes/ACLs
on the release directory before flipping the current symlink.

cr0x@server:~$ sudo -u deploy sh -lc '
release=/var/www/example/releases/20251230T1005
find "$release" -type d -exec chmod 2755 {} \;
find "$release" -type f -exec chmod 0644 {} \;
'

Meaning: Directories are traversable; files are readable. The setgid bit keeps group inheritance stable.

Decision: Use normalization to remove “works on my build agent” from the permission equation.

Prevent root-owned artifacts from landing in the web root

If you deploy with sudo, be explicit about the user context when writing files.
“I ran it with sudo” is how you end up with a web root that only root can read.

cr0x@server:~$ sudo -u deploy tar -xf /tmp/release.tar -C /var/www/example/releases/20251230T1005
cr0x@server:~$ stat -c "%U:%G %a %n" /var/www/example/releases/20251230T1005/public/index.html
deploy:web 644 /var/www/example/releases/20251230T1005/public/index.html

Meaning: Files are owned by the deployment identity, not root. That’s what you want.

Decision: Make “who writes files” a first-class deployment parameter, not an accident.

Three corporate mini-stories from the permission trenches

1) Incident caused by a wrong assumption: “Readable file means readable site”

A mid-sized company ran a Debian fleet with nginx in front of a static site generator. An engineer
rotated the deploy user and tightened permissions on /var/www across the board, aiming for least privilege.
They checked a couple of files. Everything was 0644. They shipped it.

Five minutes later, the site returned 403. On-call initially blamed nginx config drift because the
root file was clearly world-readable. The lead in the incident channel was “it can’t be permissions,
the file is 644.”

The actual problem lived one level up: the directory became 750 owned by the deploy user and a private group.
Nginx couldn’t traverse into it because “execute on directories” wasn’t in their mental model.
The error log did say “permission denied,” but people read what they expect to read.

The fix was small—grant traversal via group membership and setgid directories—but the lesson stuck:
permission checks must include the entire path. The post-incident action item wasn’t “be careful.”
It was “use namei in the runbook and test as www-data.”

2) Optimization that backfired: “Let’s make the web server write the build output”

A different shop wanted faster deploys for a PHP app. Someone proposed building assets on the server
itself to avoid moving large artifacts around. The quickest way was to run the build as the same
identity that served the app. “If the build runs as www-data, it will always have access.”

It worked. Deploy time improved. The graphs looked nicer. Then a dependency compromise hit the ecosystem
(not their fault), and their application process had write permissions to the code tree. That turned a
read-only compromise into a self-modifying one: web requests could drop files into served directories.

The incident response team contained it quickly, but it was a long week of verifying what changed on disk.
The “optimization” erased a security boundary. A boundary that had been free.

They reverted to building outside production, made the web root read-only to the serving identity, and
explicitly allowed writes only to upload/cache paths. Deploy time went slightly up; risk went way down.
That’s a trade you take every time.

3) Boring but correct practice that saved the day: default ACLs on release parents

An enterprise team ran multiple sites under /srv/apps with a Capistrano-style releases layout. Their CI created
a new release directory for every deploy. Over time they had a low-grade “sometimes 403 after deploy” problem.
Not every time. Just enough to be annoying and to erode trust in automation.

The root cause was umask drift across runners. One runner created directories as 0750, another as 0755.
Sometimes Apache could traverse; sometimes it couldn’t. Engineers tried to “standardize” by adding a chmod step,
but it ran after the symlink flip. Users would see a brief outage during deploy. They hated it.

The fix was not clever. It was correct. They set a default ACL on the parent releases directory granting the
web server user rx, and they enforced a pre-switch permission normalization. New release trees inherited sane access,
so the deployment no longer depended on the runner’s mood.

Nothing about it was exciting. It also made the “403 after deploy” page go extinct. Sometimes boring is a feature.

Common mistakes: symptom → root cause → fix

1) Symptom: 403 for a directory, but files work when requested directly

Root cause: Missing index file, directory listing disabled, or framework front controller not reachable.

Fix: Ensure an index exists (index.html / index.php), verify nginx index directive, and verify permissions on the fallback target in try_files.

2) Symptom: 403 with “(13: Permission denied)” in nginx/apache logs

Root cause: Filesystem denies access. Often a parent directory lacks execute (traverse) for www-data.

Fix: Use namei -l on the full path; grant traversal via group, ACL, or directory mode adjustments.

3) Symptom: Everything is 644/755, still 403

Root cause: AppArmor denial on Ubuntu, or Apache policy denying access.

Fix: Check kernel/audit logs for AppArmor DENIED; check Apache Require and Options directives. Don’t keep chmod’ing.

4) Symptom: Works until a deploy, then 403

Root cause: New release directory created with restrictive umask or wrong ownership; symlink points to a tree with different permissions.

Fix: Normalize permissions before switching symlink; apply default ACLs to releases parent; enforce deploy user context.

5) Symptom: Uploads fail unless you chmod 777 the whole web root

Root cause: Application needs write access only for uploads/cache, but you made the whole tree the target.

Fix: Make only specific writable dirs owned by www-data or grant ACL write there. Keep code read-only.

6) Symptom: “Permission denied” when using symlinked paths

Root cause: One of the symlink components or its target path has restricted traversal; Apache may also require explicit symlink options.

Fix: Use readlink -f and namei -l on the resolved path; adjust traversal permissions; for Apache, allow symlinks appropriately.

7) Symptom: Deploy user can write, web server can’t read (or vice versa)

Root cause: Over-reliance on owner bits; group strategy not implemented; setgid missing; ACLs absent.

Fix: Choose a model (read-only root + explicit writable dirs, or shared group + setgid, or ACLs) and implement it consistently.

Checklists / step-by-step plan

Checklist: stop the immediate 403 safely

1. Check the error log for the denial type (filesystem vs config).
2. Confirm the configured on-disk root path (nginx root/alias, Apache DocumentRoot).
3. Run namei -l on the exact file path that failed.
4. Test as www-data with sudo -u www-data test -r (or -x for directories).
5. If it’s a parent directory traverse problem, fix the narrowest directory needed (not the whole tree).
6. Reload/restart the web server if group membership changes were involved.
7. Re-test: command-line read test, then HTTP request.
8. Scan for unintended world-writable permissions after the emergency fix.

Checklist: implement a long-term permission policy

1. Pick a model:
   • Preferred: read-only web root; writable subdirs only.
   • Acceptable: shared group + setgid + enforced umask.
   • Best for complex orgs: ACLs with default entries.
2. Define writable paths explicitly (uploads/cache/sessions/logs/sockets).
3. Set ownership for writable paths (often www-data:www-data), keep code owned by deploy/root.
4. Normalize permissions during deploy before switching traffic/symlink.
5. Add a CI check or post-deploy check that runs read tests as www-data.
6. Document one runbook command: namei -l on the failing path.
7. Include AppArmor checks in Ubuntu runbooks if applicable.

Step-by-step plan: a clean baseline for a typical site

This is a practical baseline that keeps code read-only to the web server while allowing uploads and cache.
Adjust paths to your application.

1. Create a shared group for deploy + web server (optional but useful):

   cr0x@server:~$ sudo groupadd -f web
   cr0x@server:~$ sudo usermod -aG web deploy
   cr0x@server:~$ sudo usermod -aG web www-data
   

2. Set group ownership and setgid on the site tree:

   cr0x@server:~$ sudo chgrp -R web /var/www/example
   cr0x@server:~$ sudo find /var/www/example -type d -exec chmod 2755 {} \;
   cr0x@server:~$ sudo find /var/www/example -type f -exec chmod 0644 {} \;
   

3. Mark writable directories and grant write only there:

   cr0x@server:~$ sudo install -d -o www-data -g www-data -m 0750 /var/www/example/current/public/uploads
   cr0x@server:~$ sudo install -d -o www-data -g www-data -m 0750 /var/www/example/current/var/cache
   

4. Restart if group membership was changed (restart is safer than reload for this):

   cr0x@server:~$ sudo systemctl restart nginx
   

5. Validate with a read test as www-data:

   cr0x@server:~$ sudo -u www-data test -r /var/www/example/current/public/index.html; echo $?
   0
   

FAQ

1) Why does a missing execute bit on a directory cause 403 even if the file is 644?

Because “execute” on a directory means “traverse/search.” Without it, you can’t access entries inside,
even if you know the filename. The kernel blocks traversal before it even evaluates the file’s mode bits.

2) Should my web root be owned by www-data?

Usually no. If www-data owns the code, a web compromise often becomes a persistent compromise.
Prefer code owned by deploy/root and readable by www-data; make only necessary runtime directories writable.

3) What permissions should a typical static web root have?

Common baseline: directories 0755, files 0644, owned by a deploy user, group readable by the web server (via shared group or ACL).
Tighten “other” if you have a reason, but keep traversal for the web server.

4) Is using a shared group (deploy + www-data) insecure?

It can be fine if you keep the web server’s access read-only and restrict writes to specific directories.
It becomes risky when group-write is enabled broadly and the web server process can write into code paths.

5) Why did adding www-data to a group not fix the issue until restart?

Processes don’t magically update supplementary groups mid-flight. Worker processes keep their group list from when they started.
Reload might not replace workers in a way that picks up groups; restart is deterministic.

6) When should I use ACLs instead of chmod/chgrp?

Use ACLs when you have multiple identities needing different access, or when you want inherited permissions
without relying on umask and setgid correctness. ACLs are also good when you need to avoid opening “other” permissions.

7) Why do I get 403 only on some files after deploy?

Usually inconsistent creation modes: different umask, different user, or extracted artifacts preserving restrictive modes.
The fix is permission normalization in the deploy process and/or default ACLs on the releases parent.

8) What’s the safest quick fix under pressure?

Grant the minimum needed traversal/read to the web server identity. Start with the first failing parent directory shown by
namei -l. Avoid chmod -R across the whole tree; it’s how you create new problems while fixing the old one.

9) Why do I see 403 but logs show nothing?

You might be looking at the wrong log (wrong vhost, wrong container), or your log level is too low.
Also check that the request is even reaching the server you’re tailing (load balancers love practical jokes).

10) Does Debian/Ubuntu use SELinux here?

By default, no—AppArmor is more common on Ubuntu. If you installed SELinux, then it’s in play, but don’t assume it’s there.
Verify with logs and enabled modules instead of cargo-culting fixes from another distro.

Conclusion: practical next steps

Stop treating 403s like a moral failing. They’re a permission model mismatch, and Linux is extremely consistent once you ask the right question.
The right question is “can www-data traverse every directory in the path and read the target?” Not “did I chmod it recently?”

Next steps you can do today:

1. Add namei -l and “test as www-data” to your on-call runbook.
2. Pick one of the permission models and implement it consistently: read-only root with explicit writable dirs is the default adult choice.
3. Normalize permissions as part of deployments, before you switch traffic or symlinks.
4. If you run Ubuntu, include AppArmor denial checks so you don’t waste an hour chmod’ing into the void.

Keep your web root readable, your writable dirs intentional, and your chmod finger off the 7 key unless you’re writing a cautionary tale.