Ubuntu 24.04: Resolv.conf Keeps Changing — Fix systemd-resolved/NetworkManager Properly

The mental model: who owns DNS on Ubuntu 24.04

/etc/resolv.conf looks like a simple file. In modern Ubuntu, it’s often a symlink to something generated. That’s not a conspiracy; it’s plumbing. The real question is: which component is the DNS source of truth?

The main actors

• systemd-resolved: a local resolver service that can do caching, DNS-over-TLS (in some setups), and per-link DNS (split DNS). It commonly exposes a local stub at 127.0.0.53 and writes a “managed” resolv.conf.
• NetworkManager: the network orchestrator for desktops and many servers, especially when netplan uses the NetworkManager renderer. It collects DNS from DHCP/VPN/static config and can hand it to systemd-resolved or write /etc/resolv.conf directly depending on configuration.
• DHCP clients (often via NetworkManager): provide DNS servers and search domains, sometimes aggressively. If the DHCP server is wrong, your machine becomes wrong, reliably and at scale.
• VPN plugins: add routes and DNS. Some expect split DNS. Some dump a full DNS override. Some try their best and still lose to your local configuration.
• netplan: generates lower-level configuration. It doesn’t “do DNS” by itself at runtime, but it decides which renderer (NetworkManager or systemd-networkd) gets to do it.

What “resolv.conf keeps changing” usually means

It’s almost never random. It’s a deterministic result of one of these:

• /etc/resolv.conf is a symlink to a generated file, and you keep editing the symlink target’s output instead of the inputs.
• NetworkManager is configured to use systemd-resolved, but resolved is disabled or mislinked, so NetworkManager falls back to writing its own.
• Two DNS managers are enabled and both believe they’re responsible.
• A VPN comes up and pushes DNS, then DHCP renews and replaces it (or vice versa).
• A configuration tool (cloud-init, provisioning, config management) “fixes” DNS on boot.

Dry-funny truth: /etc/resolv.conf is the duct tape of Linux networking—someone always assumes it’s the right place to slap a fix, and it always peels off at 3 a.m.

Pick one authority. Wire the others to feed it. Then stop editing /etc/resolv.conf by hand unless your goal is to create a mystery for future-you.

Fast diagnosis playbook (check these first)

If DNS is broken or resolv.conf won’t stay put, don’t start by editing anything. Start by answering: “Who wrote it last?” and “Who does the system think is handling DNS?”

First: identify what /etc/resolv.conf actually is

Is it a real file or a symlink? If it’s a symlink, where does it point?

Second: determine whether systemd-resolved is active and what it thinks

If resolved is running, resolvectl status is your ground truth for what DNS servers are actually used per interface, including VPN links.

Third: check NetworkManager’s DNS mode

NetworkManager can be configured to use resolved, dnsmasq, or write resolv.conf. The wrong mode is the fastest way to get a file that changes every time a link flaps.

Fourth: reproduce and watch changes live

When you can’t explain a change, watch it happen: tail the file, look at timestamps, and correlate with journal logs. Most “mystery rewrites” are a DHCP renewal or VPN reconnect.

Interesting facts & history (because this mess has lore)

• Fact 1: /etc/resolv.conf predates most modern Linux networking stacks; it came from Unix traditions where DNS configuration was a simple static file.
• Fact 2: The “stub resolver” approach (local 127.x address) exists because it enables caching and per-interface DNS policies without teaching every application new tricks.
• Fact 3: Ubuntu has long used symlinks for /etc/resolv.conf management (via different tools over time) because multiple components want to supply DNS dynamically.
• Fact 4: split DNS became mainstream not because it’s elegant, but because corporate VPNs and cloud VPCs forced it. One resolver for everything stopped working.
• Fact 5: search domains are a productivity feature that doubles as a foot-gun: a long search list can slow resolution, leak queries, or produce surprising name collisions.
• Fact 6: historically, many libc resolvers only read /etc/resolv.conf and ignored fancy runtime services. That’s why the symlink still matters: it’s compatibility glue.
• Fact 7: DNS caching at the host layer can be a performance win, but it can also amplify outages when negative caching holds onto failures.
• Fact 8: DHCP “DNS option” behavior varies widely by network gear; some environments overwrite DNS on renew even when “manual DNS” is set in a UI.

One paraphrased idea from a reliability voice worth keeping in your head: paraphrased idea: “Hope is not a strategy” — attributed to Gene Kranz, often repeated in operations circles. DNS fixes based on hope are how you end up explaining yourself to a change review board.

Practical tasks: commands, expected output, and decisions

You want repeatable diagnosis. Here are concrete tasks that work on Ubuntu 24.04. Each one includes what the output usually means and what decision to make next.

Task 1: See if /etc/resolv.conf is a symlink

cr0x@server:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jun  2 10:14 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Meaning: It’s managed. Editing it manually is temporary at best.

Decision: Stop editing the file. Decide whether you want resolved (recommended) or not, then configure inputs accordingly.

Task 2: Identify the symlink target contents

cr0x@server:~$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
nameserver 127.0.0.53
options edns0 trust-ad
search corp.example

Meaning: The system expects apps to query the local stub, not upstream DNS directly.

Decision: If DNS is failing, investigate resolved status and upstream servers rather than replacing this with public resolvers.

Task 3: Check if systemd-resolved is running

cr0x@server:~$ systemctl status systemd-resolved --no-pager
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-12-29 08:12:41 UTC; 1h 2min ago

Meaning: resolved is active. Good. Now ask it what DNS it’s actually using.

Decision: If it’s inactive or masked, don’t try to keep the stub resolv.conf. Choose Recipe B instead.

Task 4: Inspect resolved’s view of DNS (global and per-link)

cr0x@server:~$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Current DNS Server: 10.10.0.53
       DNS Servers: 10.10.0.53 10.10.0.54
        DNS Domain: corp.example

Link 2 (ens192)
    Current Scopes: DNS
         Protocols: +DefaultRoute
Current DNS Server: 10.10.0.53
       DNS Servers: 10.10.0.53 10.10.0.54
        DNS Domain: corp.example

Meaning: resolved is the authority, and it has upstream servers. If lookups fail, upstream may be unreachable, blocked, or wrong.

Decision: If “Current DNS Server” is empty, or points to something you don’t expect (like a VPN link when you’re not on VPN), focus on NetworkManager/VPN configuration.

Task 5: Confirm NetworkManager is managing the interface

cr0x@server:~$ nmcli device status
DEVICE   TYPE      STATE      CONNECTION
ens192   ethernet  connected  Wired connection 1
lo       loopback  unmanaged  --

Meaning: NetworkManager is in the driver’s seat for ens192.

Decision: Configure DNS via NetworkManager connection profiles (or netplan that generates them), not by hand-editing resolv.conf.

Task 6: Check NetworkManager’s DNS mode

cr0x@server:~$ nmcli general status
STATE      CONNECTIVITY  WIFI-HW  WIFI     WWAN-HW  WWAN
connected  full          enabled  enabled  enabled  enabled

cr0x@server:~$ grep -R "^\[main\]$\|^dns=" -n /etc/NetworkManager/NetworkManager.conf /etc/NetworkManager/conf.d/*.conf 2>/dev/null
/etc/NetworkManager/conf.d/10-dns.conf:1:[main]
/etc/NetworkManager/conf.d/10-dns.conf:2:dns=systemd-resolved

Meaning: NetworkManager is configured to feed systemd-resolved. That’s the sane default in many Ubuntu setups.

Decision: If DNS is unstable, confirm resolved is enabled and resolv.conf points to the stub. If you instead want NM to own resolv.conf, switch to Recipe B cleanly.

Task 7: Inspect the active connection’s DNS settings

cr0x@server:~$ nmcli -g ipv4.method,ipv4.dns,ipv4.ignore-auto-dns,ipv4.dns-search connection show "Wired connection 1"
auto
10.10.0.53,10.10.0.54
no
corp.example

Meaning: You have explicit DNS servers plus DHCP auto-DNS is not ignored (no). That can lead to mixed DNS lists depending on DHCP behavior.

Decision: If you want deterministic DNS, consider setting ipv4.ignore-auto-dns yes and specify DNS/search explicitly (with the tradeoff that you must maintain it).

Task 8: See recent logs that explain who rewrote DNS

cr0x@server:~$ journalctl -u systemd-resolved -n 80 --no-pager
Dec 29 09:01:10 server systemd-resolved[812]: Using DNS server 10.10.0.53 for interface ens192.
Dec 29 09:05:34 server systemd-resolved[812]: Switching to DNS server 10.10.0.54 for interface ens192.

cr0x@server:~$ journalctl -u NetworkManager -n 120 --no-pager
Dec 29 09:05:31 server NetworkManager[701]: <info>  [..] dhcp4 (ens192): option domain_name_servers => '10.10.0.54 10.10.0.53'

Meaning: DHCP renewals changed the offered DNS order; resolved followed along.

Decision: Decide if DHCP should be trusted. If not, override DNS in the NM profile and ignore auto DNS.

Task 9: Watch /etc/resolv.conf change in real time

cr0x@server:~$ sudo inotifywait -m /etc/resolv.conf
Setting up watches.
Watches established.
/etc/resolv.conf MODIFY
/etc/resolv.conf ATTRIB

Meaning: Something is touching it; now correlate timestamps with logs.

Decision: Use the timestamp of the event to grep journal logs around that time; don’t guess.

Task 10: Test resolution through resolved explicitly

cr0x@server:~$ resolvectl query repo.corp.example
repo.corp.example: 10.20.30.40                      -- link: ens192

-- Information acquired via protocol DNS in 18.7ms.
-- Data is authenticated: no

Meaning: resolved can resolve that name using its current DNS path.

Decision: If apps still fail, your issue might be nsswitch, container DNS, or an app doing its own DNS (yes, some do).

Task 11: Test resolution the “classic” way (and compare)

cr0x@server:~$ getent hosts repo.corp.example
10.20.30.40     repo.corp.example

Meaning: Glibc NSS resolution is working for normal applications.

Decision: If resolvectl query works but getent fails (or vice versa), inspect /etc/nsswitch.conf and whether apps bypass NSS.

Task 12: Confirm netplan’s renderer choice (who should manage links)

cr0x@server:~$ sudo netplan get
network:
  version: 2
  renderer: NetworkManager

Meaning: NetworkManager is intended to manage networking on this host.

Decision: Avoid enabling systemd-networkd configurations that compete, unless you’re intentionally migrating.

Task 13: Check if cloud-init is rewriting network/DNS

cr0x@server:~$ cloud-init status
status: done

cr0x@server:~$ grep -R "manage_resolv_conf" -n /etc/cloud/cloud.cfg /etc/cloud/cloud.cfg.d/*.cfg 2>/dev/null
/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg:2:manage_resolv_conf: false

Meaning: If manage_resolv_conf is true, cloud-init may rewrite resolver state on boot.

Decision: In cloud images, explicitly disable cloud-init resolver management unless you actually want it managing DNS.

Task 14: Check if there’s a local dnsmasq or other resolver stealing port 53

cr0x@server:~$ sudo ss -lntup | grep -E ":53\s"
udp   UNCONN 0      0          127.0.0.53%lo:53        0.0.0.0:*    users:(("systemd-resolve",pid=812,fd=14))
tcp   LISTEN 0      4096       127.0.0.53%lo:53        0.0.0.0:*    users:(("systemd-resolve",pid=812,fd=15))

Meaning: resolved is listening on the stub. That’s expected in stub mode.

Decision: If you see dnsmasq or another service binding 127.0.0.53 or 0.0.0.0:53, you have a conflict. Decide which resolver should run and disable the other.

Pick your architecture (don’t mix these)

Architecture 1: systemd-resolved is the resolver, NetworkManager supplies it

This is the “Ubuntu-native” way for many installations. You get per-interface DNS, decent VPN behavior, and a clean story for where changes come from. Your /etc/resolv.conf typically points to the stub.

Do this when: you have VPNs, multiple links, split DNS requirements, or you want modern behavior that doesn’t depend on every application being well-behaved.

Architecture 2: disable systemd-resolved, NetworkManager writes /etc/resolv.conf

This is the “classic” way: applications read /etc/resolv.conf with upstream servers. It’s simpler, and sometimes that’s worth it. But you lose some split DNS capabilities unless VPN tooling compensates.

Do this when: you run minimal servers, you don’t need split DNS, or you have an application stack that behaves badly with stub resolvers.

Architecture 3: static /etc/resolv.conf (immutable or manually managed)

This is a control-freak option. It can be correct in tightly controlled environments (air-gapped, lab appliances, or special-purpose servers). It’s also a common way to break DHCP/VPN setups quietly.

Do this when: DNS servers never change and you understand the operational cost. Otherwise, don’t.

Second dry-funny truth: if you “just chattr +i” your way out of DNS churn, you’re not fixing a system—you’re starting a hobby.

Recipe A (recommended): systemd-resolved + NetworkManager, correctly

Goal state

• systemd-resolved enabled and running
• NetworkManager set to dns=systemd-resolved
• /etc/resolv.conf symlinked to the stub (or the “real” resolved file, depending on your preference)
• DNS configured in NetworkManager connection profiles (or inherited from DHCP/VPN as intended)

Step 1: ensure systemd-resolved is enabled

cr0x@server:~$ sudo systemctl enable --now systemd-resolved
Created symlink /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service.

Meaning: resolved will start at boot and is started now.

Decision: If enabling fails because it’s masked, unmask it and re-enable.

cr0x@server:~$ sudo systemctl unmask systemd-resolved
Removed "/etc/systemd/system/systemd-resolved.service".

Step 2: set NetworkManager to use resolved

Create (or verify) a drop-in config, not a hand-edited main file you’ll forget about later.

cr0x@server:~$ sudo tee /etc/NetworkManager/conf.d/10-dns-systemd-resolved.conf >/dev/null <<'EOF'
[main]
dns=systemd-resolved
EOF

cr0x@server:~$ sudo systemctl restart NetworkManager

Meaning: NetworkManager will update resolved instead of fighting over resolv.conf.

Decision: If you previously used dns=dnsmasq, remove it unless you intentionally want dnsmasq in front (most people don’t).

Step 3: fix the /etc/resolv.conf symlink

On a resolved-managed system, you generally want the stub file:

cr0x@server:~$ sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

cr0x@server:~$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 37 Dec 29 09:21 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

Meaning: applications see 127.0.0.53 and use resolved for policy.

Decision: If you have software that breaks with a stub resolver (some older containers, embedded runtimes, or weird chroots), consider linking to the “real” upstream list instead:

cr0x@server:~$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

That file contains upstream servers directly. You lose some benefits, but you maintain compatibility.

Step 4: configure DNS in the right place (NM profiles)

If you want static DNS regardless of DHCP:

cr0x@server:~$ sudo nmcli connection modify "Wired connection 1" ipv4.ignore-auto-dns yes
cr0x@server:~$ sudo nmcli connection modify "Wired connection 1" ipv4.dns "10.10.0.53 10.10.0.54"
cr0x@server:~$ sudo nmcli connection modify "Wired connection 1" ipv4.dns-search "corp.example"
cr0x@server:~$ sudo nmcli connection up "Wired connection 1"
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)

Meaning: NetworkManager will stop importing DNS from DHCP for this connection and use what you specified.

Decision: Use this only if DHCP DNS is untrustworthy or you need consistent behavior across networks.

Step 5: verify end-to-end

cr0x@server:~$ resolvectl status | sed -n '1,35p'
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Current DNS Server: 10.10.0.53
       DNS Servers: 10.10.0.53 10.10.0.54
        DNS Domain: corp.example

Meaning: resolved is in stub mode and has your intended upstreams.

Decision: If resolv.conf mode is “foreign”, something else is writing resolv.conf. Go straight to the Common mistakes section.

Recipe B: disable systemd-resolved and let NetworkManager write resolv.conf

This is for environments where you want old-school behavior and accept the tradeoffs. The key is to do it cleanly: disable resolved, replace the symlink with a real file (or a NM-managed file), and configure NetworkManager’s DNS plugin accordingly.

Step 1: stop and disable resolved

cr0x@server:~$ sudo systemctl disable --now systemd-resolved
Removed "/etc/systemd/system/multi-user.target.wants/systemd-resolved.service".

Meaning: resolved won’t run or manage stub behavior.

Decision: Make sure nothing else is still pointing to 127.0.0.53 after this.

Step 2: set NetworkManager to manage DNS directly

Depending on your environment, setting dns=default is often enough.

cr0x@server:~$ sudo tee /etc/NetworkManager/conf.d/10-dns-default.conf >/dev/null <<'EOF'
[main]
dns=default
EOF

cr0x@server:~$ sudo rm -f /etc/NetworkManager/conf.d/10-dns-systemd-resolved.conf
cr0x@server:~$ sudo systemctl restart NetworkManager

Meaning: NetworkManager will write resolver information via its default mechanism.

Decision: If you had other DNS plugins configured, remove them; mixed plugins create unpredictable results.

Step 3: replace /etc/resolv.conf with a regular file

cr0x@server:~$ sudo rm -f /etc/resolv.conf
cr0x@server:~$ sudo touch /etc/resolv.conf
cr0x@server:~$ sudo chmod 644 /etc/resolv.conf
cr0x@server:~$ ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 0 Dec 29 09:40 /etc/resolv.conf

Meaning: it’s no longer a symlink.

Decision: Bring the connection down/up so NM populates it, then verify contents.

cr0x@server:~$ sudo nmcli networking off
cr0x@server:~$ sudo nmcli networking on

cr0x@server:~$ cat /etc/resolv.conf
nameserver 10.10.0.53
nameserver 10.10.0.54
search corp.example

Meaning: applications will query upstream DNS directly.

Decision: Confirm no service expects 127.0.0.53; if you had containers or local resolvers configured, adjust them.

Recipe C: static resolv.conf (only when you really mean it)

A static /etc/resolv.conf can be appropriate for certain appliances or isolated servers. It’s not a general fix for “it keeps changing” because it doesn’t address who is trying to change it.

Make it static without playing filesystem games

Prefer policy over immutability. Tell NetworkManager to stop touching DNS for that connection:

cr0x@server:~$ sudo nmcli connection modify "Wired connection 1" ipv4.ignore-auto-dns yes
cr0x@server:~$ sudo nmcli connection modify "Wired connection 1" ipv6.ignore-auto-dns yes
cr0x@server:~$ sudo nmcli connection up "Wired connection 1"

Then manage the file explicitly:

cr0x@server:~$ sudo tee /etc/resolv.conf >/dev/null <<'EOF'
nameserver 10.10.0.53
nameserver 10.10.0.54
search corp.example
options timeout:2 attempts:2
EOF

Meaning: you’re taking responsibility for DNS updates. There is no automation safety net now.

Decision: Only do this when the DNS servers are stable and you have change control for them.

VPN and split DNS: the part that usually hurts

Most “my resolv.conf keeps changing” complaints are actually split-DNS complaints in a trench coat. A VPN comes up, pushes internal domains and servers, then your primary interface renews DHCP and reorders DNS. Or the VPN client insists on being the default route and drags DNS with it.

What split DNS should look like with systemd-resolved

With resolved, you want per-link domains and DNS servers. Internal domains go to VPN DNS; everything else goes to your normal DNS (or public resolvers if that’s your policy).

Diagnose by checking per-link domains:

cr0x@server:~$ resolvectl status | sed -n '1,200p'
Global
resolv.conf mode: stub

Link 2 (ens192)
       DNS Servers: 10.10.0.53 10.10.0.54
        DNS Domain: ~.

Link 8 (tun0)
       DNS Servers: 172.16.1.10
        DNS Domain: corp.example

Meaning:

• ~. indicates a default routing domain (all names) for that link. If your VPN link gets ~., it may steal all DNS.
• Domains like corp.example without ~. are routed only for that suffix.

Decision: If the VPN is stealing everything but shouldn’t, fix VPN plugin settings or NM connection properties to avoid setting VPN as default DNS route.

NetworkManager knobs that matter for VPN DNS behavior

These vary by VPN type, but the debugging approach is consistent: inspect the connection profile’s IP settings and routes, then ensure the VPN is not configured as “use this connection only for resources on its network” unless you actually want that.

Check the VPN connection details:

cr0x@server:~$ nmcli connection show "Corp VPN" | grep -E "ipv4\.|ipv6\.|vpn\."
ipv4.never-default:                         no
ipv4.ignore-auto-dns:                       no
ipv4.dns:                                   --
ipv4.dns-search:                            --

Meaning: If never-default is no, the VPN may become default route depending on plugin behavior. If ignore-auto-dns is no, it can import DNS and potentially override.

Decision: For split-tunnel behavior, set ipv4.never-default yes on the VPN connection and ensure domains are configured specifically.

cr0x@server:~$ sudo nmcli connection modify "Corp VPN" ipv4.never-default yes
cr0x@server:~$ sudo nmcli connection up "Corp VPN"

When apps ignore the system resolver

Some runtimes ship their own resolver or cache. Some containers mount their own /etc/resolv.conf. Some Java stacks do DNS caching that makes outages last longer than they should.

If your host resolves but the app doesn’t, compare these:

cr0x@server:~$ getent hosts internal.service.corp.example
10.50.0.15     internal.service.corp.example

cr0x@server:~$ sudo resolvectl query internal.service.corp.example
internal.service.corp.example: 10.50.0.15 -- link: tun0

Decision: If both work on the host, but the app fails, inspect the app’s runtime DNS caching and any container resolver configuration. Don’t “fix” it by replacing host DNS with 8.8.8.8 and hoping corporate zones magically exist there.

Common mistakes: symptom → root cause → fix

1) Symptom: /etc/resolv.conf resets to 127.0.0.53 after every reboot

Root cause: You’re on Recipe A (resolved), but you keep replacing the symlink with a regular file. Boot services restore the symlink.

Fix: Decide to use resolved and configure DNS via NetworkManager; keep /etc/resolv.conf symlinked to /run/systemd/resolve/stub-resolv.conf (or to /run/systemd/resolve/resolv.conf for compatibility).

2) Symptom: resolv.conf contains upstream servers, but resolvectl shows different servers

Root cause: “foreign” mode: something besides resolved is managing /etc/resolv.conf, but resolved is still running and making its own decisions for apps that query the stub.

Fix: Either disable resolved (Recipe B) or restore the resolv.conf symlink (Recipe A). Don’t run both patterns at once.

3) Symptom: DNS works until VPN connects; then public DNS lookups fail (or vice versa)

Root cause: VPN is set as default route and/or default DNS routing domain (~.), stealing all queries.

Fix: Configure split DNS: VPN should own only corporate suffixes; set ipv4.never-default yes and ensure domains are scoped. Verify with resolvectl status.

4) Symptom: DNS changes every few hours

Root cause: DHCP renewals reorder or replace DNS servers; NetworkManager imports them faithfully.

Fix: If DHCP DNS is wrong, override in NM connection and set ipv4.ignore-auto-dns yes. Otherwise, fix DHCP server configuration (the correct fix, just not always the easiest one).

5) Symptom: “Temporary failure in name resolution” sporadically, especially under load

Root cause: Upstream DNS is slow/unreachable, or search domain expansion causes multiple failed queries per lookup. Sometimes MTU/VPN fragmentation causes UDP DNS trouble.

Fix: Measure with resolvectl query timings, review search domains, and test TCP fallback. If VPN is involved, verify MTU and consider forcing smaller DNS packets via upstream policy (not by mutilating resolv.conf).

6) Symptom: Containers resolve differently than the host

Root cause: Container runtime injects its own /etc/resolv.conf. Also common: it points to 127.0.0.53 which is not reachable from the container’s network namespace.

Fix: Configure the runtime to use the upstream resolv.conf (for resolved setups, use /run/systemd/resolve/resolv.conf as the source), or configure a proper DNS server reachable from containers.

7) Symptom: You disable resolved, but apps still query 127.0.0.53

Root cause: /etc/resolv.conf still points to stub-resolv.conf, or you have stale configuration in chroots/containers.

Fix: Replace resolv.conf with a real file containing upstream servers, and restart affected services that cache resolver state.

Checklists / step-by-step plan

Checklist 1: Make DNS stable on a typical Ubuntu 24.04 workstation (recommended)

1. Confirm netplan uses NetworkManager: sudo netplan get → look for renderer: NetworkManager.
2. Enable resolved: sudo systemctl enable --now systemd-resolved.
3. Set NM to use resolved via /etc/NetworkManager/conf.d/ drop-in.
4. Symlink /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf.
5. Configure DNS per NM connection if you must override DHCP: nmcli connection modify ... ipv4.ignore-auto-dns yes.
6. Verify with resolvectl status and getent hosts.
7. Connect VPN and re-check per-link DNS domains; fix default-route stealing if it happens.

Checklist 2: Make DNS stable on a server where you want classic resolv.conf

1. Disable resolved: sudo systemctl disable --now systemd-resolved.
2. Configure NetworkManager dns=default (or your chosen plugin, but choose one).
3. Replace /etc/resolv.conf symlink with a regular file.
4. Cycle networking: sudo nmcli networking off && sudo nmcli networking on.
5. Verify /etc/resolv.conf contains upstream servers and getent hosts works.
6. For VPN clients, ensure they update resolv.conf in a controlled way or use explicit DNS settings.

Checklist 3: When “something keeps changing it” and you need proof

1. Watch changes: sudo inotifywait -m /etc/resolv.conf.
2. Capture timestamps and correlate with journalctl -u NetworkManager and journalctl -u systemd-resolved.
3. Check for cloud-init: cloud-init status and grep for manage_resolv_conf.
4. Search for config management edits: check your automation, systemd units, and cron jobs (yes, people still cron DNS fixes).
5. Once identified, remove the competing writer. Don’t negotiate with it.

Three corporate mini-stories from the trenches

Mini-story 1: The incident caused by a wrong assumption

The team had a fleet of Ubuntu servers, mostly stable, mostly boring. A new internal service rollout started failing in a way that looked like an application bug: intermittent 502s, retries, then everything “magically” fixed itself. The first assumption was the worst kind: “DNS can’t be the issue, we set it in resolv.conf.”

They had, in fact, “set it.” Months earlier, someone edited /etc/resolv.conf directly to force internal DNS servers, because DHCP in one environment handed out public resolvers. It worked until the next reboot, and then they edited it again. Eventually a config management rule was added to reapply it daily. That’s not DNS management; that’s whack-a-mole with root privileges.

After upgrading some hosts, systemd-resolved became active and /etc/resolv.conf became a symlink again. The daily config job replaced it with a file containing upstream DNS, but applications still used resolved in some contexts, and some services cached resolution differently. Two resolver paths existed on the same host. Failures depended on timing and which library path the process used.

The fix was not “set better DNS servers.” The fix was choosing an owner: they standardized on resolved + NetworkManager, removed the daily job, and moved DNS policy into NM profiles. Once the system had a single source of truth, the “application bug” disappeared. The postmortem was uncomfortable for exactly one reason: everyone knew they were editing resolv.conf like it was 2003.

Mini-story 2: The optimization that backfired

A different org had a performance initiative: reduce DNS lookup latency for microservices. Someone noticed repeated lookups to the same names and decided local caching would help. They enabled host-level resolver caching and pushed a config to point all workloads at the local stub resolver.

Latency improved. Everyone was happy. Then a partial DNS outage hit their internal DNS tier. The authoritative servers were flaky, not dead—timeouts, occasional SERVFAIL. The local caching resolver started negative caching failures. A single transient miss became a persistent “no” for the cache lifetime, and the failure rate across services went from “spiky and recoverable” to “flat and brutal.”

The worst part: the outage looked like an application regression because restarts didn’t help. Some hosts recovered quickly; others didn’t. The difference was cache state. People started draining nodes, rolling back deployments, and arguing with each other about which change caused it. The real culprit was the optimization that removed natural jitter and turned transient DNS problems into sticky ones.

They kept caching in the end, because it’s not inherently wrong. But they adjusted negative caching behavior where possible, improved upstream DNS reliability, and—this is the important part—added diagnostics: resolvectl status snapshots in incident runbooks. Optimizations are fine. Silent optimizations are how you buy yourself a new pager sound.

Mini-story 3: The boring but correct practice that saved the day

One enterprise had a dull, almost annoying standard: every Linux host must have a “DNS ownership” check as part of provisioning. It verified whether the host was using resolved or not, ensured the symlink state matched, and confirmed NetworkManager’s DNS mode. Engineers complained it was bureaucratic.

Then came a merger. Two networks, two VPN solutions, two sets of DHCP policies, and one combined fleet of laptops and build agents. DNS behavior became unpredictable overnight. The natural human response was to apply “quick fixes” to resolv.conf on the affected hosts.

The teams that followed the boring standard didn’t do that. They ran the ownership check, saw which part of the stack was supposed to manage DNS, and adjusted only the inputs: NM profile DNS settings and VPN split DNS scoping. That made their fixes durable. Reboots didn’t re-break things. VPN reconnects didn’t randomly flip resolver order.

Meanwhile, the ad-hoc-fix teams ended up with a zoo: some hosts had immutable resolv.conf, some had symlinks, some had disabled resolved but still referenced 127.0.0.53. The boring team shipped features while everyone else debugged “why only this one laptop can’t reach the artifact repo.” Correctness is often tedious. It also scales.

FAQ

1) Why is /etc/resolv.conf a symlink on Ubuntu 24.04?

Because DNS is dynamic now: DHCP, VPN, and multiple interfaces can all change DNS policy. The symlink points to a generated file so the system can update resolver state without hand-editing.

2) Is 127.0.0.53 “wrong”?

No. It’s the local stub resolver address used by systemd-resolved. Apps send DNS queries there; resolved forwards them to real upstream DNS servers based on per-link policy.

3) How do I stop resolv.conf from changing?

Stop treating /etc/resolv.conf as the configuration source. Choose an architecture: either use resolved and configure NetworkManager to feed it (Recipe A), or disable resolved and let NetworkManager write upstream servers directly (Recipe B).

4) What does “resolv.conf mode: foreign” mean in resolvectl?

It usually means resolved detected that /etc/resolv.conf is not its managed file/symlink, so something else is managing it. That’s a sign you have competing owners and should cleanly pick one.

5) I set DNS in the GUI, but it keeps reverting. Why?

Often because DHCP is still allowed to provide DNS and override/merge settings. Check nmcli -g ipv4.ignore-auto-dns .... Set it to yes if you want deterministic manual DNS on that connection.

6) VPN breaks my public DNS or my corporate DNS. What’s the best approach?

Use split DNS with resolved: corporate zones routed to VPN DNS, everything else to your normal DNS. Verify per-link domains with resolvectl status. Avoid letting the VPN claim the default routing domain unless you want all DNS to go through it.

7) Should I make /etc/resolv.conf immutable with chattr?

Only if you enjoy self-inflicted outages. It blocks legitimate DNS updates (DHCP renewals, VPN connects) and creates confusing failure modes. Prefer proper ownership and configuration.

8) How do I debug “DNS works on host but not in containers”?

Check what /etc/resolv.conf the container sees. If it points to 127.0.0.53 but the container can’t reach that address (different network namespace), configure the runtime to use upstream servers (often from /run/systemd/resolve/resolv.conf) or provide a reachable DNS server.

9) Is disabling systemd-resolved a bad idea?

Not inherently. It’s a tradeoff. It simplifies things for some server setups, but it can make split DNS and VPN behavior harder. If you disable it, do it cleanly and verify no one is still pointing at 127.0.0.53.

Conclusion: stable DNS, stable life

The durable fix for “resolv.conf keeps changing” is not to fight the file. It’s to choose a DNS owner and make every other component feed it instead of competing.

Next steps that actually hold up under reboots, VPN reconnects, and DHCP renewals:

• Run the fast diagnosis: symlink check → resolved status → NetworkManager DNS mode → watch changes live.
• If you want modern split DNS and sane multi-link behavior, implement Recipe A and keep the stub symlink.
• If you want classic behavior for a simple server, implement Recipe B and remove the stub entirely.
• Document which model you chose for your fleet. The most expensive DNS incidents start with “I thought this host was using…”