ZFS Missing Device Paths: Using by-id and WWN Like an Adult

Why /dev/sdX is a trap (and why it keeps happening)

/dev/sdX is not an identity. It’s a seat assignment. Linux hands out those letters in discovery order, and discovery order changes:
new HBA firmware, different PCIe slot training, a slow disk that answers later after a power event, multipath timing, enclosure resets, you name it.
Your disk didn’t become a different disk. The kernel just met them in a different order.

ZFS is perfectly happy to use whatever path you gave it. If you built the pool as mirror /dev/sda /dev/sdb,
ZFS will record those strings. Later, if /dev/sda now points to a different drive, you’ve got two problems:
ZFS can’t find the old one, and you might accidentally “fix” the wrong disk. The first one causes a degraded pool. The second one causes a résumé update.

The cure is to stop using ephemeral names and start using stable identifiers: /dev/disk/by-id and, specifically, WWN-based IDs when available.
These are designed to survive reboots, re-enumeration, and most hardware reshuffles.

Joke #1: Using /dev/sdX in production is like labeling servers “the one near the window.” It works until someone moves the desk.

Interesting facts and a little history

• Fact 1: ZFS originated at Sun Microsystems and was built around the idea that storage lies; it verifies data end-to-end with checksums.
• Fact 2: Linux device names like /dev/sda come from the SCSI disk subsystem, even for SATA and USB drives routed through SCSI emulation layers.
• Fact 3: WWN (World Wide Name) is a globally unique identifier concept from Fibre Channel/SAS ecosystems, later carried into SATA via standards and adapters.
• Fact 4: udev’s /dev/disk/by-* symlinks exist precisely because kernel device enumeration is not stable across boots.
• Fact 5: ZFS stores labels on disk (four labels per vdev member in typical implementations), which is why pools can often be imported even when paths change.
• Fact 6: “by-path” names encode the physical connection path (PCI bus, HBA port, enclosure slot) and are stable until you move cables or HBAs.
• Fact 7: In the early days of commodity storage, drive serial numbers were sometimes duplicated or padded weirdly by bridge chips; WWN generally behaves better.
• Fact 8: Multipath environments often expose both per-path device nodes and a consolidated mapper device; picking the wrong one can cause “flapping” devices in ZFS.

What you should use: by-id, WWN, and when by-path is acceptable

Pick a stable identity: WWN-based by-id symlinks

On most modern Linux systems you’ll find symlinks under /dev/disk/by-id. For SAS/SATA/NVMe, you typically get multiple flavors:
vendor/model/serial, and often a WWN-based one. The WWN-based entries commonly look like:
wwn-0x5000cca2... (SATA/SAS) or nvme-eui.000000... / nvme-uuid.... (NVMe).

For ZFS, prefer WWN/EUI/UUID forms because they’re intended to be globally unique and are less likely to be mangled by a USB-SATA bridge or a quirky HBA.
If you only have model+serial symlinks, that can still be okay, but you should validate uniqueness across the whole chassis.

When by-path is useful

/dev/disk/by-path tells you where the drive is plugged in. That’s operational gold when you’re doing “find the physical disk to pull”
in a noisy datacenter at 3 a.m. But it is not an identity if you plan to move the disk to another controller port. It’s an address.

I use by-path for troubleshooting and slot mapping, and by-id/WWN for pool membership. That split keeps humans sane and keeps pools stable.

What to avoid

• Avoid: /dev/sdX always. Even on “simple” servers. Especially on “simple” servers.
• Avoid: partition-less “whole disk” nodes if you’re in an environment where tools might write metadata (some installers, some RAID tools).
• Avoid: mixing mapper multipath devices and raw path devices inside the same pool.
• Avoid: relying on /dev/disk/by-label or filesystem labels for ZFS vdev identification; that’s not what they’re for.

How ZFS remembers devices (and what “missing device path” actually means)

ZFS doesn’t just trust a path string. It writes pool configuration and vdev labels to the disks themselves. That’s why you can move a pool to another host,
import it, and ZFS can often figure out what belongs where. But the path still matters because it’s what ZFS tries first, and what operators see in alerts.

A “missing device path” situation usually lands in one of these buckets:

• Path changed, disk present: The disk is fine, but the OS presented it under a different node. Classic /dev/sdX shuffle.
• Disk missing entirely: HBA issues, backplane problems, power, cabling, multipath misconfiguration, or an actual dead disk.
• Disk present but refused: Sector size mismatch, stale partition table, wrong replacement disk, or the wrong device node (e.g., a single path of a multipath device).
• Permissions/udev timing: Early boot import attempts, missing udev rules in initramfs, or encryption layers not opened yet.

Your job is to decide which bucket you’re in quickly. Then you can either relabel paths (harmless) or start treating the hardware as guilty (necessary).

One quote worth keeping on a sticky note (paraphrased idea): Hope is not a strategy — often attributed to engineering and operations culture.
In storage, it’s painfully literal.

Fast diagnosis playbook

This is the order I use when the pager says “ZFS DEGRADED” and someone is already asking if we need to fail over. The goal is to find the bottleneck fast:
is it naming, connectivity, or a failing device?

First: confirm what ZFS thinks is missing

• Run zpool status -P to force full paths and capture the exact “UNAVAIL” name.
• Check whether the missing entry is a raw disk, a partition, or a mapper device.

Second: confirm whether the OS can see the disk by stable IDs

• List /dev/disk/by-id and /dev/disk/by-path.
• Use lsblk -o NAME,SIZE,MODEL,SERIAL,WWN,HCTL,TYPE,MOUNTPOINTS to tie identity to topology.

Third: decide whether this is “path drift” or “hardware loss”

• If the WWN exists and matches your inventory/notes, it’s likely naming drift. Use zpool online or zpool replace with the correct by-id path.
• If the WWN is absent or the disk is missing from lsblk, check kernel logs (dmesg) and controller state.

Fourth: stop making it worse

• Do not start pulling disks based on /dev/sdX.
• Do not run zpool clear as a “fix.” That only clears errors; it doesn’t conjure disks back into existence.

Fifth: pick the smallest safe action

• For pure path drift: update ZFS to the stable path via replace (even replacing a device with itself, using the stable name).
• For actual device failure: replace with a new disk and resilver. If there are checksum errors elsewhere, treat the pool as contaminated and scrub.

Practical tasks: commands, outputs, and decisions (12+)

These are real things you run at the console. Each one includes what the output tells you and what decision it should trigger.
Use them as building blocks; don’t cargo-cult them into scripts without understanding the environment (multipath and encryption change the rules).

Task 1: Check pool health with full paths

cr0x@server:~$ sudo zpool status -P
  pool: tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
config:

        NAME                                      STATE     READ WRITE CKSUM
        tank                                      DEGRADED     0     0     0
          mirror-0                                DEGRADED     0     0     0
            /dev/sda2                             UNAVAIL      0     0     0  cannot open
            /dev/disk/by-id/wwn-0x5000c500a1b2c3d4-part2  ONLINE       0     0     0

errors: No known data errors

What it means: ZFS is missing /dev/sda2. The other mirror side is present. This strongly smells like path drift.
Decision: Do not replace hardware yet. First find what disk used to be /dev/sda and relink ZFS to a by-id path.

Task 2: Confirm the disk exists under by-id

cr0x@server:~$ ls -l /dev/disk/by-id/ | sed -n '1,12p'
total 0
lrwxrwxrwx 1 root root  9 Dec 26 02:10 ata-SAMSUNG_MZ7KM960HAHP-00005_S3Z1NX0K123456 -> ../../sdb
lrwxrwxrwx 1 root root 10 Dec 26 02:10 ata-SAMSUNG_MZ7KM960HAHP-00005_S3Z1NX0K123456-part2 -> ../../sdb2
lrwxrwxrwx 1 root root  9 Dec 26 02:10 wwn-0x5002538d4098abcd -> ../../sdb
lrwxrwxrwx 1 root root 10 Dec 26 02:10 wwn-0x5002538d4098abcd-part2 -> ../../sdb2
lrwxrwxrwx 1 root root  9 Dec 26 02:10 wwn-0x5000c500a1b2c3d4 -> ../../sdc
lrwxrwxrwx 1 root root 10 Dec 26 02:10 wwn-0x5000c500a1b2c3d4-part2 -> ../../sdc2

What it means: You have stable handles. The missing /dev/sda2 likely became /dev/sdb2 or /dev/sdc2.
Decision: Identify which WWN belongs to the missing vdev member (via ZFS labels or prior records), then reattach using that by-id path.

Task 3: Tie device nodes to WWN/serial/topology in one view

cr0x@server:~$ lsblk -o NAME,SIZE,MODEL,SERIAL,WWN,HCTL,TYPE,MOUNTPOINTS
NAME   SIZE MODEL              SERIAL          WWN                HCTL       TYPE MOUNTPOINTS
sdb  893.8G SAMSUNG MZ7KM960   S3Z1NX0K123456  0x5002538d4098abcd 1:0:0:0    disk
├─sdb1   1G                                      0x5002538d4098abcd 1:0:0:0  part
└─sdb2 892G                                      0x5002538d4098abcd 1:0:0:0  part
sdc  893.8G SAMSUNG MZ7KM960   S3Z1NX0K654321  0x5000c500a1b2c3d4  2:0:0:0    disk
├─sdc1   1G                                      0x5000c500a1b2c3d4 2:0:0:0  part
└─sdc2 892G                                      0x5000c500a1b2c3d4 2:0:0:0  part

What it means: You can now speak in identities (WWN/serial) and physical endpoints (HCTL).
Decision: If you have a runbook or asset notes mapping WWN to slot, this is where you stop guessing.

Task 4: Look for kernel-level storage errors fast

cr0x@server:~$ dmesg -T | egrep -i 'sd[a-z]|nvme|ata|sas|scsi|reset|I/O error|timed out' | tail -n 20
[Thu Dec 26 02:09:51 2025] sd 2:0:0:0: [sdc] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[Thu Dec 26 02:09:51 2025] sd 2:0:0:0: [sdc] tag#13 Sense Key : Not Ready [current]
[Thu Dec 26 02:09:51 2025] sd 2:0:0:0: [sdc] tag#13 Add. Sense: Logical unit not ready, cause not reportable
[Thu Dec 26 02:09:52 2025] sd 2:0:0:0: [sdc] tag#21 timing out command, waited 180s
[Thu Dec 26 02:10:04 2025] scsi 2:0:0:0: rejecting I/O to offline device

What it means: This is not “just naming.” The disk/controller path is sick: timeouts and “offline device.”
Decision: Treat as hardware/connectivity. Check cabling/backplane/HBA, and plan a replacement. Don’t just online it and hope.

Task 5: Query ZFS events to see recent device changes

cr0x@server:~$ sudo zpool events -v | tail -n 25
TIME                           CLASS
Dec 26 02:10:06.912726000      resource.fs.zfs.statechange
  pool = tank
  vdev_guid = 11801824156334312345
  vdev_path = /dev/sda2
  vdev_state = UNAVAIL
  vdev_ashift = 12
Dec 26 02:10:07.102938000      resource.fs.zfs.statechange
  pool = tank
  vdev_guid = 9938450195345123456
  vdev_path = /dev/disk/by-id/wwn-0x5000c500a1b2c3d4-part2
  vdev_state = ONLINE
  vdev_ashift = 12

What it means: ZFS believes /dev/sda2 was a real vdev path. You also learn ashift (important when replacing).
Decision: If you plan a replace, match ashift. Also use these events for incident timelines; they’re often clearer than syslog noise.

Task 6: Locate the missing vdev by GUID in on-disk labels

cr0x@server:~$ sudo zdb -l /dev/disk/by-id/wwn-0x5002538d4098abcd | sed -n '1,40p'
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'tank'
    state: 0
    txg: 1923481
    pool_guid: 8012456789012345678
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 11801824156334312345
        path: '/dev/sda2'
        phys_path: 'pci-0000:3b:00.0-sas-0x5000c500d00dbeef-lun-0'

What it means: You’ve found the disk that ZFS thinks is /dev/sda2. The label even records the old path.
Decision: This disk is your missing member. You can now reattach it under a stable path without guessing which sdX it is today.

Task 7: Replace a device with itself to update the path to by-id

cr0x@server:~$ sudo zpool replace tank /dev/sda2 /dev/disk/by-id/wwn-0x5002538d4098abcd-part2
invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/wwn-0x5002538d4098abcd-part2 is part of active pool 'tank'

What it means: ZFS is protecting you from doing something ambiguous. You’re “replacing” with a device already in the pool.
Decision: Use -f intentionally when doing the “replace-with-self” trick to rewrite the stored path to the stable symlink.

cr0x@server:~$ sudo zpool replace -f tank /dev/sda2 /dev/disk/by-id/wwn-0x5002538d4098abcd-part2

What it means: No news is good news. ZFS accepted the new path mapping.
Decision: Re-run zpool status -P and confirm the pool now references by-id entries everywhere.

Task 8: Bring a device online after transient loss

cr0x@server:~$ sudo zpool online tank /dev/disk/by-id/wwn-0x5000c500a1b2c3d4-part2
bringing device online

What it means: ZFS will try to resume I/O. If the underlying path is still failing, it will drop again.
Decision: If the disk repeatedly goes offline, stop treating it as transient. Escalate to hardware and plan replacement.

Task 9: Watch resilver progress like you mean it

cr0x@server:~$ sudo zpool status -P tank
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.
config:

        NAME                                              STATE     READ WRITE CKSUM
        tank                                              DEGRADED     0     0     0
          mirror-0                                        DEGRADED     0     0     0
            /dev/disk/by-id/wwn-0x5002538d4098abcd-part2  ONLINE       0     0     0  (resilvering)
            /dev/disk/by-id/wwn-0x5000c500a1b2c3d4-part2  ONLINE       0     0     0

scan: resilver in progress since Thu Dec 26 02:14:03 2025
        128G / 892G scanned at 1.12G/s, 6.5G / 892G issued at 58.1M/s, 0% done, 04:14:23 to go

What it means: Issued is what’s actually being written. If scanned is fast but issued is slow, you’re limited by writes, not reads.
Decision: If resilver time is huge, consider load shedding, checking for SMR disks, or investigating controller throttling.

Task 10: Confirm that every vdev now uses stable paths

cr0x@server:~$ sudo zpool status -P | egrep '/dev/(sd|vd|xvd)'

What it means: No output is the goal: it means you’re not referencing ephemeral nodes.
Decision: If you still see /dev/sd*, schedule a controlled maintenance to rewrite them via replace-with-self.

Task 11: Identify whether you’re accidentally using a single path in a multipath setup

cr0x@server:~$ lsblk -o NAME,TYPE,SIZE,WWN,PKNAME | egrep 'mpath|dm-|sd[a-z] '
sda  disk  1.8T 0x600508b1001c2d3e4f5a6b7c8d9e0001
dm-2 mpath 1.8T 0x600508b1001c2d3e4f5a6b7c8d9e0001
sdb  disk  1.8T 0x600508b1001c2d3e4f5a6b7c8d9e0001

What it means: Same WWN appears on multiple sd* nodes (different paths), and also on dm-2 as the multipath device.
Decision: For ZFS in multipath, you generally want the consolidated mapper device (e.g., /dev/mapper/mpath*) or a consistent DM path,
not one of the raw /dev/sd* legs. Mixing is how you get intermittent “device removed” chaos.

Task 12: Show udev properties for a specific disk (why did by-id change?)

cr0x@server:~$ udevadm info --query=property --name=/dev/sdb | egrep 'ID_SERIAL=|ID_WWN=|ID_MODEL=|ID_PATH='
ID_MODEL=SAMSUNG_MZ7KM960HAHP-00005
ID_SERIAL=SAMSUNG_MZ7KM960HAHP-00005_S3Z1NX0K123456
ID_WWN=0x5002538d4098abcd
ID_PATH=pci-0000:3b:00.0-sas-0x5000c500d00dbeef-lun-0

What it means: udev is telling you the WWN and the physical path. If ID_WWN is missing, you might be behind a bridge that doesn’t report it.
Decision: If WWN is missing, consider using the most stable available by-id entry, and verify uniqueness across all disks.

Task 13: Verify partition tables and ensure you’re using the correct partition

cr0x@server:~$ sudo sgdisk -p /dev/disk/by-id/wwn-0x5002538d4098abcd
Disk /dev/disk/by-id/wwn-0x5002538d4098abcd: 1875385008 sectors, 894.3 GiB
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 7A2B2C0D-9D2A-4B4A-8C31-2E1DA1A0C9B8
Partition table holds up to 128 entries
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         2099199   1024.0 MiB  EF00  EFI System
   2         2099200      1875384319   893.3 GiB   BF01  ZFS

What it means: You can confirm which partition is the ZFS member (here, partition code BF01).
Decision: If you accidentally point ZFS at the whole disk when your standard is partitions (or vice versa), stop and correct before replacing.

Task 14: Import a pool when paths changed (safe mode first)

cr0x@server:~$ sudo zpool import
   pool: tank
     id: 8012456789012345678
  state: DEGRADED
 action: The pool can be imported despite missing or damaged devices.
 config:

        tank                                      DEGRADED
          mirror-0                                DEGRADED
            11801824156334312345                  UNAVAIL  cannot open
            wwn-0x5000c500a1b2c3d4-part2           ONLINE

What it means: ZFS can see the pool by reading labels, even if one vdev is missing. Note it’s showing a GUID instead of a path for the missing one.
Decision: Import read-only if you’re unsure whether you’re looking at the right disks, especially in shared storage environments.

cr0x@server:~$ sudo zpool import -o readonly=on -d /dev/disk/by-id tank
cannot import 'tank': one or more devices is currently unavailable

What it means: Even with explicit by-id scanning, something is missing enough to block import (policy depends on vdev topology).
Decision: Don’t force import unless you understand the redundancy. Go find the missing device via cabling/HBA logs or confirm you have sufficient replicas.

Three corporate mini-stories from the trenches

Mini-story 1: The incident caused by a wrong assumption

A mid-sized company ran a ZFS-backed VM platform on a couple of storage servers. Nothing exotic: SAS shelves, mirrored vdevs, decent monitoring.
The pool was originally created during an “urgent migration weekend,” which is how all technical debt is born. The vdevs were added using /dev/sdX.

Months later they added a new HBA and moved a shelf cable to tidy the rack. The next reboot came with a degraded pool and one vdev member “missing.”
The on-call engineer saw /dev/sdc missing and assumed it was “the third disk in that chassis,” because humans love ordinal thinking.
They pulled what they believed was the right drive, replaced it, and ran zpool replace with the new /dev/sdc.

The system accepted the command. Of course it did: /dev/sdc existed. It just wasn’t the disk they thought. The “missing” disk was still present,
renamed to /dev/sdf after the controller change. Meanwhile, they had physically removed a perfectly healthy mirror side and replaced it with a blank drive.
For a brief period, resilvering kicked off while the remaining original disk carried the whole vdev alone.

The next twist was predictable and cruel: during resilver, the remaining original disk threw read errors. Old age, latent sector issues, whatever.
Now the only good copy of some blocks had been on the disk that got pulled. The result wasn’t total pool loss, but it was corrupted VM images and a week of cleanup.
The postmortem was blunt: the trigger wasn’t “disk failure,” it was “we treated /dev/sdX as identity.”

They rebuilt the operational practice around WWN naming and added a rule: no one touches a sled until the WWN on the console matches the WWN on the physical label.
It slowed replacements by five minutes. It saved them from themselves forever after.

Mini-story 2: The optimization that backfired

Another place had an engineer who liked tidy systems. They wanted ZFS vdev names to reflect physical slot numbers, so they standardized on /dev/disk/by-path.
It read beautifully in zpool status: you could point a flashlight at the rack and find the drive fast.

Then procurement swapped in a newer SAS HBA model because the old one was “end of life.” Same vendor, same cable type, same shelves. The hardware team did the upgrade.
Suddenly every by-path symlink changed because the PCI addresses and SAS expander enumeration changed. Nothing was wrong with the disks. The “addresses” changed.

ZFS import started failing intermittently in early boot because the initramfs didn’t have the updated udev rules and because those by-path names weren’t created yet.
Ops tried to band-aid it by adding sleeps to boot scripts. That’s not engineering; it’s superstition.

They eventually migrated to WWN-based by-id for pool membership and kept by-path only for human mapping tools. The “optimization” had been readability inside ZFS,
but it turned the pool config into a hostage of bus topology. Great for diagrams; terrible for reliability.

Mini-story 3: The boring but correct practice that saved the day

A large enterprise team ran ZFS pools for build artifacts and container images. The environment was intentionally boring: mirrored vdevs, hot spares,
and a written standard: all pools use WWN-based by-id names, partitions only, and every replacement includes a scrub after resilver.
No one loved it. It was process.

One weekend a power event bounced a rack. After reboot, one server came up with a pool degraded and two disks showing odd behavior in kernel logs.
The on-call could immediately identify which physical drives were involved because their runbook stored the WWNs and the enclosure slot mapping.
They didn’t guess. They didn’t “try a few commands.” They followed the checklist.

One disk was truly dead. The other was fine but had re-enumerated. Because the pool used stable WWN paths, ZFS didn’t get confused.
The only action needed was replacing the dead disk and letting resilver run. The system kept serving the entire time with predictable performance.

The quiet win was that nothing surprising happened. Management never heard about it. That’s the point of the boring practice: outages that never become meetings.

Migrating an existing pool to stable device paths (without drama)

If you built your pool with /dev/sdX, you can fix it without rebuilding. The technique is simple: replace each vdev member
with itself, but referenced via the stable by-id path. ZFS updates the stored path, and your future zpool status output stops lying.

Rules before you touch production

• Confirm uniqueness: Make sure each by-id name you plan to use maps to exactly one disk.
• Prefer partitions: If you standardize on -part2 (or similar), keep doing that. Consistency matters for replacement automation and safety.
• One disk at a time: Especially on RAIDZ where a mistake is harder to unwind, but also on mirrors because humans get confident.
• Capture state: Save zpool status -P output before and after.

A practical migration sequence

Start by listing the pool with full paths. Then for each member that’s a raw /dev/sd* path, find the corresponding by-id path and run a forced replace.
Example for a mirror member recorded as /dev/sda2:

cr0x@server:~$ sudo zpool status -P tank
  pool: tank
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            /dev/sda2  ONLINE     0     0     0
            /dev/sdb2  ONLINE     0     0     0

Now map /dev/sda2 to a stable ID:

cr0x@server:~$ readlink -f /dev/disk/by-id/wwn-0x5002538d4098abcd-part2
/dev/sda2

Then replace-with-self to rewrite the stored path:

cr0x@server:~$ sudo zpool replace -f tank /dev/sda2 /dev/disk/by-id/wwn-0x5002538d4098abcd-part2

Repeat for each remaining /dev/sd* member. When you’re done, zpool status -P should show only by-id paths.
If you still see a stray /dev/sdX, don’t ignore it. That’s how standards die: one exception at a time.

Joke #2: Storage outages are like dental work—avoidable with regular maintenance, but humans insist on learning the hard way.

Common mistakes: symptoms → root cause → fix

1) Pool shows UNAVAIL for /dev/sdX after reboot

Symptoms: zpool status shows a missing /dev/sdX path; disks appear “moved.”

Root cause: Kernel device renumbering; ZFS recorded unstable names.

Fix: Identify the correct disk via WWN/serial (lsblk, zdb -l) and rewrite paths using zpool replace -f with by-id.

2) Replacing a disk “works,” but the wrong disk starts resilvering

Symptoms: Resilver begins on a disk you didn’t intend; another disk becomes missing; panic.

Root cause: You used /dev/sdX and replaced the wrong device node, or you misidentified physical disk vs logical device.

Fix: Stop. Confirm WWNs and ZFS labels. Use zpool status -P, lsblk, and zdb -l. If necessary, offline the mistakenly targeted device and re-evaluate before proceeding.

3) Pool import fails in early boot but works manually later

Symptoms: System boots without pool; manual zpool import later succeeds.

Root cause: udev symlinks not present yet in initramfs; race between device discovery and import service; encryption layers not opened.

Fix: Ensure initramfs includes needed udev rules and modules; order services so block devices exist before import; for encrypted vdevs, ensure zfs-load-key/cryptsetup runs before import/mount.

4) Devices flap ONLINE/OFFLINE under load

Symptoms: Kernel logs show timeouts; ZFS marks vdev as offline; comes back briefly; repeats.

Root cause: Bad cable, backplane, HBA firmware issues, or using a single path device in a multipath environment.

Fix: Check dmesg, validate multipath configuration, replace suspect cable/backplane slot, update firmware if validated. Prefer stable multipath mapper devices if multipathing is in play.

5) by-id names change after firmware update

Symptoms: A disk’s by-id symlink looks different than before (serial formatting, vendor prefix).

Root cause: Firmware/bridge chips can change how inquiry strings are reported; udev rules derive IDs from these.

Fix: Prefer WWN/EUI identifiers; validate with udevadm info. If identifiers truly changed, update ZFS paths using replace-with-self to the new stable form.

6) You used by-path, then moved cables during maintenance

Symptoms: Pool comes up degraded; all “addresses” look different; operators insist “the disks are in the same slots.”

Root cause: by-path encodes connection topology; moving a cable or HBA changes it.

Fix: Use by-id/WWN for pool membership. Keep by-path for mapping/LED workflows, not as the authoritative identifier in ZFS.

7) Replacement disk joins but performs terribly during resilver

Symptoms: Resilver crawls; latency spikes; “issued” throughput low.

Root cause: SMR drive in a workload expecting CMR; controller write cache policy; mismatched sector sizes causing read-modify-write; heavy production load.

Fix: Verify drive model type, cache policy, and sector sizes. Consider using a known-good drive class for replacement and throttle workloads during resilver.

8) ZFS sees the disk but refuses to use it without -f

Symptoms: zpool replace complains the device is part of an active pool or has existing labels.

Root cause: You’re replacing with the same physical device (path rewrite), or the disk has stale ZFS labels from previous membership.

Fix: For path rewrites, use zpool replace -f deliberately. For stale labels on a truly new replacement, clear labels with zpool labelclear on the correct device before proceeding.

Checklists / step-by-step plan

Checklist A: Build a new pool the adult way

1. Identify disks by WWN/EUI/UUID, not /dev/sdX.
2. Use partitions consistently (e.g., -part2 for ZFS member).
3. Record a mapping of WWN → chassis slot/enclosure bay in your runbook.
4. Create the pool using /dev/disk/by-id/wwn-...-partN paths.
5. Immediately verify with zpool status -P that ZFS stored by-id paths.
6. Scrub after burn-in and periodically thereafter; treat scrubs as early warning, not optional hygiene.

Checklist B: After a reboot, pool is degraded with a missing path

1. Run zpool status -P and capture output.
2. Check lsblk -o NAME,SIZE,MODEL,SERIAL,WWN,HCTL for the missing disk identity.
3. Search dmesg for I/O errors/timeouts. If present, assume hardware until proven otherwise.
4. Use zdb -l on candidate by-id disks to match the vdev GUID/path.
5. If it’s path drift: zpool replace -f to the by-id path and confirm status.
6. If it’s hardware loss: schedule replacement; do not “clear” your way out of physics.

Checklist C: Controlled migration from sdX to by-id on an existing pool

1. Schedule a maintenance window if the pool is critical; this is safe, but safety includes humans having time.
2. Run zpool status -P and identify all /dev/sd* entries.
3. For each one, confirm the target by-id symlink resolves to the same partition (readlink -f).
4. Run zpool replace -f pool oldpath new-by-id-path one device at a time.
5. After each change, re-run zpool status -P and confirm the new path is stored.
6. When done, grep for remaining /dev/sd references and fix them.
7. Document the WWN list in your ops runbook and, if you have it, your CMDB.

Checklist D: Disk replacement without turning it into a thriller

1. Identify the failed disk by WWN in ZFS output, not by device letter.
2. Match WWN to a physical slot using your enclosure mapping.
3. Offline the vdev member (if still ONLINE but suspect) before pulling hardware.
4. Insert replacement and confirm its WWN and size match expectations.
5. Partition consistently (or replicate the partition table from a healthy sibling).
6. Run zpool replace using by-id paths, then monitor resilver to completion.
7. Run a scrub after resilver and review any checksum errors like you mean it.

FAQ

1) If ZFS stores labels on disk, why do paths matter at all?

Because paths are what ZFS tries first and what you operate on. Labels help ZFS find pools, but the path string is still operational reality:
it shows up in alerts, scripts, and human decision-making. Make it stable.

2) Should I use /dev/disk/by-id or /dev/disk/by-uuid?

For ZFS vdevs, use /dev/disk/by-id (WWN/EUI/UUID style). by-uuid usually refers to filesystem UUIDs; ZFS is not “a filesystem on a block device” in that way.

3) Is vendor+serial in by-id good enough?

Often yes, especially for enterprise SAS/SATA drives. But WWN/EUI is usually better. If you’re behind USB bridges or cheap controllers,
vendor+serial can be unreliable or duplicated. Validate uniqueness before trusting it.

4) What about NVMe naming? /dev/nvme0n1 also changes.

Correct. NVMe controller numbering can shift. Use by-id entries like nvme-eui.* or nvme-uuid.* for ZFS membership.
Treat nvme0n1 like sda: a temporary nickname.

5) Can I change vdev paths without resilvering?

Replacing a device with itself to update the path typically does not require copying data if it’s truly the same device.
ZFS may still perform some checks. The key is correctness: confirm identity first (WWN + zdb -l).

6) Should I use whole disks or partitions for ZFS vdevs?

Pick one standard and stick to it. Partitions give you clearer intent and reduce surprises with tools that scribble metadata.
Whole-disk vdevs can be fine in controlled environments. In mixed corporate reality, partitions are usually safer.

7) What does “cannot open” mean in zpool status?

It means ZFS attempted to open the path it has recorded and failed. That could be because the path no longer exists (renamed),
the device is gone, or permissions/availability are wrong during boot. Your next step is OS visibility checks (lsblk, dmesg, /dev/disk/by-id).

8) Is by-path ever the right choice for ZFS?

Rarely, and only if your operational model treats the physical connection as the stable identity (fixed backplane, no cable moves, consistent controllers),
and you accept that hardware maintenance can rewrite “identities.” Most teams don’t actually operate that way. Use WWN for identity; use by-path for location.

9) My by-id symlink points to a different sdX after reboot. Is that bad?

That’s normal and exactly why by-id exists. The symlink follows the disk’s identity even when the kernel’s letter assignment changes.
ZFS should be referencing the symlink, not the underlying sdX.

10) Can I import a pool by scanning /dev/disk/by-id only?

Yes, with zpool import -d /dev/disk/by-id. This can reduce ambiguity in hosts with lots of transient devices.
It doesn’t fix missing hardware, but it helps prevent importing the wrong pool from the wrong set of disks.

Conclusion: next steps you can do today

If you remember one thing: /dev/sdX is not a disk identity. It’s the kernel’s mood. ZFS will faithfully remember whatever you tell it,
which is comforting right up until it’s not.

Practical next steps:

1. Run zpool status -P on every host and look for /dev/sd* (or /dev/nvme*) in vdev paths.
2. For each unstable entry, map it to a WWN-based by-id symlink and perform a controlled replace-with-self using zpool replace -f.
3. Write down WWN → slot mappings and keep them where on-call can find them fast.
4. Update your build standards: new pools are created only with by-id/WWN paths, consistently partitioned.
5. When a pool degrades, follow the playbook: identify, verify, then act. No guessing. No “clear and pray.”

Do this once, properly, and the next time a device letter shuffle happens, you’ll barely notice. Which is the highest compliment production can offer.