Proxmox “node not in cluster”: what happened and how to rejoin correctly

What “node not in cluster” actually means

In Proxmox VE, “node not in cluster” usually shows up when a node believes it should be clustered, but it can’t see (or can’t validate) the cluster state. That state is delivered by Corosync and stored/distributed through pmxcfs (the Proxmox Cluster File System) mounted at /etc/pve.

The message can mean one of these practical realities:

• Corosync isn’t running (or can’t bind to the cluster network).
• No quorum, so pmxcfs goes read-only or won’t mount fully, and cluster config can’t be trusted.
• The node’s cluster identity is stale (wrong corosync.conf, wrong node ID, wrong ring addresses, or wrong hostname/IP mapping).
• Split brain: the node formed/kept a different view of the cluster and is now incompatible.
• Time drift breaks Corosync authentication or makes tokens expire in creative ways.
• You changed the network (VLAN, MTU, bonding, firewall) and Corosync traffic is now sad.

Here’s the opinionated part: treat the message like a safety fuse. It’s Proxmox telling you, “I can’t prove the cluster state; I won’t pretend.” Your job is to figure out why it can’t prove it, then choose the least-destructive recovery path.

Also: never “fix” this by copying random files into /etc/pve from another node unless you understand exactly what you’re doing. /etc/pve isn’t a normal directory. It’s a distributed database with opinions.

Facts and context that explain the weirdness

These aren’t trivia. They’re the little historical landmines that explain why Proxmox behaves the way it does.

1. Corosync came from the HA Linux ecosystem and inherits a “membership and quorum first” worldview. If membership is uncertain, it would rather stop than guess.
2. Proxmox stores most cluster config in pmxcfs, a FUSE filesystem backed by an in-memory database replicated via Corosync. That’s why losing Corosync often breaks “simple” config tasks.
3. /etc/pve is not like /etc. It’s dynamically mounted. If it’s not mounted, Proxmox services that expect it may behave like you deleted the control plane.
4. Quorum rules are there to prevent split brain, not to annoy you. They’re also why a two-node cluster is operationally awkward unless you add a third voter (qdevice) or accept constraints.
5. Cluster node identity relies on node IDs and names carried in Corosync config and internal state; mismatching a hostname can look like a different machine showing up in the same uniform.
6. Proxmox historically optimized for on-prem reliability where “a node disappeared” is often a cabling or switch issue, not an autoscaling event. The UI and defaults reflect that.
7. Network multicast used to be a big deal in clusters. Modern Corosync typically uses unicast, but the “cluster network must be boring and predictable” rule survived for good reasons.
8. Time sync is a cluster dependency because authenticated messaging and timeouts are assumptions baked into the membership protocol. NTP problems can masquerade as network problems.
9. Proxmox’s HA stack assumes fencing matters. Without a reliable membership view, it avoids taking actions that might start the same VM twice.

One paraphrased idea that belongs on every ops wall: paraphrased idea: “Hope is not a strategy; design so failures are expected and survivable.” — Gene Kranz (paraphrased idea)

Fast diagnosis playbook

This is the “stop scrolling and find the bottleneck” sequence. You’re trying to answer three questions: Do we have quorum? Is Corosync healthy? Is /etc/pve mounted and consistent?

1) First: check if the node is simply isolated

• Can it reach the other nodes on the Corosync ring IPs?
• Is the cluster network interface up, correct VLAN/MTU, no firewall drop?
• Is time sane?

2) Second: check quorum and membership from a healthy node

• If the cluster has quorum elsewhere, the “not in cluster” node is the problem.
• If the cluster has no quorum, don’t “rejoin” anything yet. Stabilize quorum first.

3) Third: decide the recovery track

• Track A (best): node identity is still valid; fix networking/time/Corosync and it rejoins automatically.
• Track B (common): node was removed or its state diverged; you must delete and re-add it using pvecm delnode then pvecm add.
• Track C (worst): split brain or conflicting cluster states; you must pick a “source of truth” and carefully clean the other side.

Short operational rule: if you still have VMs running on the broken node, treat it as a separate failure domain until you confirm cluster membership. Avoid HA actions that could double-start workloads.

How Proxmox clustering works (the parts that matter)

Corosync: the membership oracle

Corosync is the cluster communications layer. It decides who is in the club. It also enforces quorum policy: if fewer than the required votes are present, the cluster becomes conservative. The cluster might still run existing workloads, but control-plane operations can be blocked or become read-only.

When Corosync is down or can’t communicate, Proxmox services are forced into an “I can’t verify state” posture. That’s where “node not in cluster” often originates.

pmxcfs: your config is a replicated filesystem

pmxcfs provides /etc/pve. It’s where VM configs, storage definitions, firewall rules, and cluster metadata live. It is not just a directory you edit; it’s a distributed state machine wearing a filesystem costume.

If /etc/pve isn’t mounted (or mounted but stuck), you’ll see odd symptoms:

• VM config files “missing” even though VMs still exist on disk.
• GUI showing stale or empty cluster state.
• Edits to cluster config failing or reverting.

Quorum: why two nodes are a trap unless you plan

In a three-node cluster, losing one node typically still leaves quorum. In a two-node cluster, losing either node loses quorum unless you add a quorum device (qdevice) or intentionally tweak expectations. That’s not Proxmox being dramatic; it’s math.

Dry-funny truth: a two-node cluster without a third voter is like a meeting with two executives and no minutes; whoever speaks last thinks they won.

Split brain: the failure mode you must avoid

Split brain means two sides of the cluster believe they are authoritative. If both sides can write to shared storage or both can start the same VM, you get data corruption or duplicates. Proxmox’s quorum behavior tries hard to prevent this. Your job is not to outsmart it with “temporary” hacks.

Practical tasks: commands, outputs, and decisions

Below are concrete tasks you can run. Each includes (1) a command, (2) what plausible output means, and (3) what decision to make next. Run them on the affected node and at least one healthy node when possible.

Task 1: Confirm what Proxmox thinks its node name is

cr0x@server:~$ hostnamectl --static
pve03

Meaning: Proxmox expects the node name to match cluster config. If the cluster knows this node as pve03 but your host now calls itself pve3, you’ve created a new identity.

Decision: If hostname changed, fix it back to the cluster-known name before you do anything else.

Task 2: Validate hostname resolution (cluster traffic often uses names)

cr0x@server:~$ getent hosts pve01 pve02 pve03
10.10.10.11   pve01
10.10.10.12   pve02
10.10.10.13   pve03

Meaning: Names resolve to the expected ring IPs. If this is wrong (old IPs, wrong subnet), Corosync membership breaks.

Decision: Fix DNS or /etc/hosts consistency across all nodes; don’t band-aid it on one node only.

Task 3: Check if /etc/pve is mounted (pmxcfs health indicator)

cr0x@server:~$ mount | grep -E 'pmxcfs|/etc/pve'
pmxcfs on /etc/pve type fuse.pmxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Meaning: If you see pmxcfs, the cluster filesystem is mounted. If you don’t, Proxmox is running “blind.”

Decision: If not mounted, focus on Corosync and pve-cluster service status next.

Task 4: Check pve-cluster and Corosync services

cr0x@server:~$ systemctl status pve-cluster corosync --no-pager
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
     Active: active (running)
● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled)
     Active: failed (Result: exit-code)

Meaning: Corosync failed: no membership, likely no quorum view. pve-cluster might still run locally but without a valid cluster state.

Decision: Inspect Corosync logs and config; don’t jump to “rejoin” yet.

Task 5: Read Corosync logs for the real error

cr0x@server:~$ journalctl -u corosync -b --no-pager | tail -n 25
Dec 25 10:11:12 pve03 corosync[1321]: [TOTEM ] Token has not been received in 2730 ms
Dec 25 10:11:12 pve03 corosync[1321]: [TOTEM ] A processor failed, forming new configuration.
Dec 25 10:11:14 pve03 corosync[1321]: [KNET  ] link: host: 2 link: 0 is down
Dec 25 10:11:15 pve03 corosync[1321]: [MAIN  ] Corosync Cluster Engine exiting with status 1

Meaning: Network path failure (knet link down) or severe packet loss/MTU mismatch. Corosync is not a fan of flaky networks.

Decision: Validate network interface, MTU, VLAN tagging, and firewall. Don’t touch cluster membership until packets flow.

Task 6: Confirm Corosync ring connectivity (ICMP is not enough, but it’s a start)

cr0x@server:~$ ping -c 3 10.10.10.11
PING 10.10.10.11 (10.10.10.11) 56(84) bytes of data.
64 bytes from 10.10.10.11: icmp_seq=1 ttl=64 time=0.356 ms
64 bytes from 10.10.10.11: icmp_seq=2 ttl=64 time=0.331 ms
64 bytes from 10.10.10.11: icmp_seq=3 ttl=64 time=0.342 ms

--- 10.10.10.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms

Meaning: Basic reachability exists. If ping fails, the cluster can’t work. If ping succeeds but Corosync fails, look at MTU, firewall, or asymmetry.

Decision: If ping is fine, test MTU and UDP behavior next.

Task 7: Check MTU consistency (silent killer for knet)

cr0x@server:~$ ip -br link show
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eno1             UP             3c:ec:ef:11:22:33 <BROADCAST,MULTICAST,UP,LOWER_UP>
vmbr0            UP             3c:ec:ef:11:22:33 <BROADCAST,MULTICAST,UP,LOWER_UP>
vmbr1            UP             3c:ec:ef:11:22:33 <BROADCAST,MULTICAST,UP,LOWER_UP>

cr0x@server:~$ ip link show vmbr1 | grep -E 'mtu|state'
2: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000

Meaning: If this node uses MTU 9000 but the switch or peer nodes are 1500, you get “works sometimes” behavior that Corosync punishes.

Decision: Make MTU uniform end-to-end. If you can’t guarantee jumbo frames, stay at 1500.

Task 8: Verify time sync (Corosync auth and timeouts hate drift)

cr0x@server:~$ timedatectl
               Local time: Thu 2025-12-25 10:14:02 UTC
           Universal time: Thu 2025-12-25 10:14:02 UTC
                 RTC time: Thu 2025-12-25 10:14:01
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Meaning: “synchronized: yes” is what you want. If it’s “no,” fix NTP before blaming Corosync.

Decision: Bring NTP back (chrony/systemd-timesyncd) and re-check membership.

Task 9: Check cluster status from a healthy node (authoritative view)

cr0x@server:~$ pvecm status
Cluster information
-------------------
Name:             prod-cluster
Config Version:   18
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Dec 25 10:15:10 2025
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.6
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Meaning: Cluster is healthy elsewhere. The broken node is the one out of membership.

Decision: Focus on the broken node’s config/network rather than cluster-wide surgery.

Task 10: Check membership list and whether the node is “known”

cr0x@server:~$ pvecm nodes
Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 pve01
0x00000002          1 pve02
0x00000003          1 pve03

Meaning: If the cluster still lists the node, it should be able to rejoin without deletion, assuming identity matches and network works.

Decision: If the node is missing here, you’ll likely need pvecm add after cleanup.

Task 11: Inspect the local Corosync config on the broken node

cr0x@server:~$ grep -E 'cluster_name|ring0_addr|nodeid|bindnetaddr' -n /etc/pve/corosync.conf
2:  cluster_name: prod-cluster
12:     bindnetaddr: 10.10.10.0
18:         ring0_addr: pve01
23:         nodeid: 1
28:         ring0_addr: pve02
33:         nodeid: 2
38:         ring0_addr: pve03
43:         nodeid: 3

Meaning: The node IDs and ring addresses should match what the cluster expects. If this file is missing or different, you’re not looking at the same cluster reality.

Decision: If the file differs from a healthy node’s /etc/pve/corosync.conf, you have config drift or /etc/pve isn’t properly mounted.

Task 12: Verify the Corosync authentication key exists and matches (only if you know you should)

cr0x@server:~$ ls -l /etc/corosync/authkey
-r-------- 1 root root 128 Dec 10 09:02 /etc/corosync/authkey

Meaning: Missing authkey = can’t authenticate cluster traffic. Wrong key = can’t join.

Decision: If the node was reinstalled or partially rebuilt, it may have a different key; you should re-add the node properly rather than copying keys blindly.

Task 13: See whether the node thinks it’s in a cluster at all

cr0x@server:~$ test -f /etc/pve/.members && echo "members file exists" || echo "members file missing"
members file exists

Meaning: Presence suggests cluster metadata exists, but it may be stale or read-only.

Decision: If /etc/pve isn’t mounted yet this file exists, you might be seeing an old local mount state; resolve service/mount issues first.

Task 14: Check if /etc/pve is read-only (quorum loss symptom)

cr0x@server:~$ touch /etc/pve/test-write 2>&1 || true
touch: cannot touch '/etc/pve/test-write': Read-only file system

Meaning: You don’t have quorum from this node’s perspective, or pmxcfs is protecting state.

Decision: Don’t force changes. Fix quorum/membership first. If this is a single-node survival situation, you may temporarily adjust quorum expectations (carefully, and only with a clear plan).

Task 15: Confirm firewall isn’t blocking Corosync traffic

cr0x@server:~$ pve-firewall status
Status: enabled/running

cr0x@server:~$ iptables -S | grep -E 'DROP|REJECT' | head
-A INPUT -j PVEFW-INPUT
-A PVEFW-INPUT -j DROP

Meaning: If the firewall is enabled and rules are strict, Corosync UDP traffic can be dropped.

Decision: Confirm cluster network is in a trusted zone or explicitly allow Corosync. Don’t disable the firewall permanently; fix the rules.

Task 16: Confirm Ceph (if used) isn’t the real incident

cr0x@server:~$ ceph -s
cluster:
  id:     7d2c1c2a-9b9c-4e5b-9a65-1f9b6c1a1b11
  health: HEALTH_WARN
          1 osds down

services:
  mon: 3 daemons, quorum a,b,c
  mgr: x(active), standbys: y
  osd: 12 osds: 11 up, 12 in

Meaning: A node leaving the Proxmox cluster might coincide with Ceph warnings, but Ceph quorum can still be fine. Separate control-plane membership from storage-plane health.

Decision: Fix cluster membership first if you need centralized management; fix Ceph OSD recovery in parallel if it’s affecting IO.

Second short joke (you only get two): If your first instinct is “reinstall the node,” congratulations—you’ve invented the IT version of turning it off and on again with a forklift.

Checklists / step-by-step plan: rejoin correctly, depending on what happened

There isn’t one “rejoin” procedure. There are three, and only one is safe for your situation. Choose based on facts you verified above.

Plan 0: Stop and protect workloads (always do this first)

• Identify if any VMs/CTs are running on the affected node and whether they use shared storage (Ceph, NFS, iSCSI, shared ZFS).
• If HA is enabled, consider temporarily disabling HA actions for impacted services while you stabilize membership to avoid duplicate starts.
• If the node is isolated but still running workloads, treat it as “degraded standalone.” Don’t let another node start the same workloads.

Plan A: The node is still in cluster membership, just disconnected (best case)

Use when: cluster quorum is healthy elsewhere; pvecm nodes lists the node; and the node’s identity (hostname, IP mapping, authkey) is unchanged.

1. Fix networking: VLAN/MTU/bonding, switch ports, firewall rules.
2. Fix time sync.
3. Restart Corosync and pve-cluster on the affected node, in that order if Corosync was down.

cr0x@server:~$ systemctl restart corosync
cr0x@server:~$ systemctl restart pve-cluster
cr0x@server:~$ pvecm status
Cluster information
-------------------
Name:             prod-cluster
Config Version:   18
Transport:        knet
Secure auth:      on

Quorum information
------------------
Quorate:          Yes

Meaning: If it flips to quorate and membership returns, you’re done. Now validate storage and HA status.

Decision: If Corosync won’t start or still can’t join, stop and move to Plan B or C depending on what you discover.

Plan B: The node must be re-added cleanly (common after reinstall or identity drift)

Use when: the cluster is healthy elsewhere, but the node’s cluster state is stale, missing, or mismatched (reinstalled OS, new authkey, renamed host, changed IPs).

Step 1: On a healthy node, remove the dead membership entry (if present)

cr0x@server:~$ pvecm delnode pve03
Removed node pve03 from cluster

Meaning: This removes cluster membership metadata for that node. It does not magically wipe its disks, but it does change cluster state.

Decision: If removal fails because of quorum, do not force it; restore quorum first.

Step 2: Clean cluster config on the node you’re rejoining (be precise)

This is the part people rush and regret. You are removing the node’s old cluster identity so it can join as a fresh member.

cr0x@server:~$ systemctl stop pve-cluster corosync
cr0x@server:~$ rm -f /etc/corosync/corosync.conf
cr0x@server:~$ rm -f /etc/corosync/authkey
cr0x@server:~$ rm -rf /etc/pve/corosync.conf /etc/pve/nodes /etc/pve/.members
cr0x@server:~$ systemctl start pve-cluster

Meaning: You’re wiping local cluster identity files. This should not delete VM disks. It can orphan local VM configs if they were only in /etc/pve and not backed elsewhere—so confirm first where your VM configs live.

Decision: If the node hosts unique, non-replicated VM configs you still need, stop and back up /etc/pve state before you remove anything.

Step 3: Join the cluster from the node you’re adding

cr0x@server:~$ pvecm add 10.10.10.11
Please enter superuser (root) password for '10.10.10.11':
Establishing API connection with host '10.10.10.11'
The authenticity of host '10.10.10.11' can't be established.
Fingerprint: SHA256:0mJ0QwqN7ZpVtJcXxq0b3oYzYl6s9Qx9fYqVQ4oSx2s
Are you sure you want to continue connecting (yes/no)? yes
Successfully added node 'pve03' to cluster.

Meaning: Proxmox copied the correct Corosync config/auth material and registered the node.

Decision: Immediately verify membership and quorum from both sides.

Step 4: Verify post-join state

cr0x@server:~$ pvecm status | grep -E 'Name:|Quorate:|Nodes:'
Name:             prod-cluster
Nodes:            3
Quorate:          Yes

Meaning: The control plane is back. Now you must validate storage mounts, VM configs, and replication jobs.

Plan C: Split brain or conflicting cluster states (the careful surgery)

Use when: you suspect two independent cluster states exist: for example, someone ran pvecm create on the isolated node, or restored /etc/pve from a backup onto one side without coordination.

What this looks like in practice:

• Same cluster name, different node IDs.
• Nodes appear twice or with odd IDs.
• Corosync logs mention incompatible config versions or authentication failures that persist even with correct networking.
• /etc/pve/corosync.conf differs across nodes when it absolutely should not.

The safe approach: pick a source of truth (the side with quorum and the most consistent state). Then treat the other side as a node that must be re-added (Plan B) after you ensure workloads on that side are stopped or safely isolated.

In split brain scenarios, you must also account for shared storage:

• If both sides can write to the same datastore, stop. Ensure only one side is active on shared storage.
• If using Ceph, confirm MON quorum and ensure the isolated node isn’t doing anything “creative” like forming a second Ceph cluster.
• If using ZFS replication, confirm dataset GUIDs and replication direction before re-enabling jobs.

Plan C is where change control pays rent. If you can schedule downtime, do it. If you can’t, at least schedule silence from everyone except the person holding the keyboard.

Common mistakes: symptoms → root cause → fix

1) Symptom: “node not in cluster” after a reboot; other nodes are fine

Root cause: Corosync failed to bind to the right interface after a network change (bridge name change, bond reordering, VLAN not up in time).

Fix: Verify ring addresses and interface readiness; correct /etc/network/interfaces or systemd-networkd; restart Corosync; confirm logs show membership.

2) Symptom: /etc/pve is mounted read-only

Root cause: No quorum (often two-node cluster with one node down) or node can’t see enough peers.

Fix: Restore quorum by bringing nodes back or adding a quorum device; don’t force writes. If you must run temporarily as a single node, do it with explicit quorum strategy and a plan to revert.

3) Symptom: Corosync log shows “token has not been received” and knet link flaps

Root cause: Network packet loss, MTU mismatch, or firewall filtering UDP between cluster peers.

Fix: Make MTU consistent, fix switch/vlan, allow Corosync traffic, avoid routing cluster traffic through “smart” firewalls doing stateful weirdness.

4) Symptom: Node was reinstalled and now can’t join; authentication errors

Root cause: New install has a different Corosync authkey and possibly a different hostname/SSH host key. It’s a different node, even if the hardware is the same.

Fix: Remove the old node entry from the cluster, clean cluster identity on the new install, re-add with pvecm add.

5) Symptom: Cluster shows duplicate nodes or strange IDs

Root cause: Split brain or manual file copying into /etc/pve; conflicting membership databases.

Fix: Choose a source-of-truth cluster and re-add the others cleanly. Do not merge by hand unless you enjoy archaeology.

6) Symptom: GUI shows cluster but CLI says not quorate (or vice versa)

Root cause: Stale UI cache, partial pmxcfs mount, or services restarted in a bad order.

Fix: Trust CLI (pvecm status, journalctl). Restart services in controlled order; confirm /etc/pve mount and read/write state.

7) Symptom: Everything worked until someone “optimized” the cluster network

Root cause: Jumbo frames enabled on some ports, not all; LACP hashing changed; offload settings changed; or the cluster network moved behind a firewall appliance.

Fix: Make the cluster network boring: consistent MTU, minimal middleboxes, stable latency. Put cleverness somewhere else.

Three corporate-world mini-stories (anonymized, plausible, and painfully familiar)

Mini-story 1: The incident caused by a wrong assumption

They had a three-node Proxmox cluster in a regional office, running a mix of Windows VMs and a couple of Linux appliances. One day, a node went dark. Facilities said it was “just a power cycle.” The junior admin assumed the node would rejoin once it booted.

It booted, but Proxmox showed “node not in cluster.” The admin did what many people do under pressure: copied /etc/pve/corosync.conf from a healthy node into the broken one and restarted services. It looked better for about five minutes. Then Corosync started flapping.

The wrong assumption was subtle: they believed /etc/pve was a normal filesystem. It wasn’t. Their copy landed in a place that pmxcfs managed, and the node’s pmxcfs state wasn’t healthy yet. They basically stapled a configuration onto a moving target.

What fixed it was boring: they rolled back the manual copy, verified the ring network, and found the switch port had been moved to a different VLAN during unrelated work. Once the VLAN was corrected and time sync confirmed, the node rejoined without any membership surgery.

The postmortem takeaway was crisp: treat “node not in cluster” as a connectivity and quorum problem first, not a file problem. A cluster is a system, not a folder.

Mini-story 2: The optimization that backfired

A mid-sized company decided their Proxmox cluster network should be “fast.” Someone enabled MTU 9000 on the storage VLAN and figured, why not also set it on the corosync ring. Half the switches supported jumbo frames end-to-end, half did “sort of,” and a couple of older NICs needed driver tuning.

Nothing exploded immediately. That’s how these stories start. Over a few weeks, there were intermittent cluster membership changes. Short ones. A node would drop, come back, drop again. HA would hesitate. Logs mentioned missing tokens. Everyone blamed “Proxmox being flaky.”

The backfire was that Corosync’s traffic pattern doesn’t care about your theoretical bandwidth; it cares about predictability. The jumbo frame inconsistency introduced microbursts and occasional drops that ordinary application traffic didn’t notice. Corosync noticed. Loudly.

They fixed it by reverting the Corosync network back to MTU 1500 and leaving jumbo frames only where they could guarantee end-to-end support (and where it made a measurable difference). The cluster became boring again. Performance barely changed. Uptime did.

Mini-story 3: The boring but correct practice that saved the day

A different org ran a five-node Proxmox cluster with Ceph and strict change control. Every change to networking included a pre-flight checklist: confirm hostname resolution, confirm NTP sync, confirm Corosync ring reachability, confirm pvecm status quorate, and record the current corosync.conf version.

One afternoon, a switch reboot cascaded into a short outage on the cluster network. One node came back with “node not in cluster.” The team didn’t improvise. They pulled the checklist.

They immediately saw time sync on the affected node was broken: NTP was stuck because the node’s default route changed during boot and couldn’t reach the NTP servers. The node’s clock drifted enough that Corosync authentication and timeouts went sideways. Fixing routing and NTP restored membership cleanly.

No one “rejoined” anything. No one deleted cluster state. No one copied files around. The best incident response is sometimes just refusing to panic, and having the receipts to prove what changed.

FAQ

1) Does “node not in cluster” mean my VMs are gone?

No. It means the control plane can’t validate cluster state. VM disks are typically on local storage, shared storage, or Ceph; they don’t evaporate because Corosync is unhappy. But VM configs stored in /etc/pve may not be visible if pmxcfs isn’t mounted.

2) Can I just run pvecm add and move on?

Only if you’re sure the node was cleanly removed or is a fresh install that you intend to add. If the cluster still believes the node exists, joining without cleanup can create conflicts. Verify with pvecm nodes from a healthy node first.

3) What’s the safest first action when this happens?

Check quorum and Corosync status from a healthy node. If the cluster is quorate, your problem is localized. If it’s not, fix quorum before doing anything destructive.

4) Why does /etc/pve sometimes go read-only?

Because Proxmox refuses to accept writes when cluster membership isn’t trusted (often due to quorum loss). This prevents split brain and config divergence.

5) How do I handle this in a two-node cluster?

Either add a third voter (qdevice) or accept that losing one node likely loses quorum. Two-node clusters can work, but you must design for the quorum math rather than arguing with it during incidents.

6) Is it safe to copy corosync.conf from another node?

Usually no. If pmxcfs is healthy and the cluster is healthy, the config is already replicated. If pmxcfs is not healthy, copying files can worsen divergence. Use pvecm add for controlled distribution after cleaning state when necessary.

7) After rejoin, why are my storages missing in the UI?

Because storage definitions are in cluster config (/etc/pve/storage.cfg). If pmxcfs isn’t mounted or you joined incorrectly, the node may not have the replicated config. Verify /etc/pve mount and compare storage.cfg across nodes.

8) Does Ceph complicate rejoining a Proxmox node?

It complicates the blast radius, not the core rejoin mechanics. Proxmox cluster membership is one system; Ceph is another. You must ensure the node doesn’t cause double-mounts or OSD confusion, but you still fix Corosync/quorum first.

9) What if the node was renamed and I want to keep the new name?

Don’t rename in-place and hope the cluster follows. Remove the node from the cluster, clean its cluster state, set the new hostname consistently everywhere, then add it back as a new member. Plan the impact on VM configs and monitoring.

10) When is reinstalling actually the right move?

When the OS is compromised, irreparably misconfigured, or you can’t trust the node’s state—and you have a clean operational process to remove/re-add nodes. Reinstalling as a first response is usually impatience disguised as decisiveness.

Next steps you should actually do

If you’re holding an incident ticket right now, do these in order:

1. Establish quorum status from a healthy node with pvecm status. If not quorate, fix quorum before “rejoining.”
2. On the broken node: confirm hostname, name resolution, time sync, and that /etc/pve is mounted.
3. Read the Corosync logs. They are rarely poetic, but they are usually honest.
4. Fix the boring network issues: MTU mismatch, VLAN tagging, firewall drops, interface naming changes.
5. Choose the right recovery plan:
   • Plan A if identity is intact and the node is still listed.
   • Plan B if identity is stale (reinstall/renamed/new key).
   • Plan C if split brain is suspected—pick a source of truth and re-add carefully.
6. After recovery: validate storage, HA state, and replication jobs. Don’t declare victory until the boring checks pass.

Then do future-you a favor:

• Document the cluster network (interfaces, VLANs, MTU, firewall zones) and make it intentionally dull.
• If you run two nodes, add a quorum device or accept the operational constraints explicitly.
• Practice the remove/re-add procedure in a lab once. Incidents are a bad time to learn what pmxcfs is.