Local-lvm at 100% is the Proxmox equivalent of a power steering failure: everything looks fine right up until the moment it isn’t. VMs stall, backups fail, and the web UI starts throwing errors like a stressed-out pager.
The tricky part isn’t that you’re out of space. It’s why you’re out of space—thin provisioning, snapshots, orphaned volumes, copy-on-write write amplification, a guest that never trims, or metadata that quietly hit the wall before data did. This is how you find what ate your thin pool, fix it without making it worse, and keep it from happening again.
What “local-lvm” really is (and why it fills differently)
On most Proxmox installs, you end up with two local storages by default:
- local: a directory (usually
/var/lib/vz) for ISOs, templates, backups. - local-lvm: an LVM-thin pool used for VM disks.
That second one is the troublemaker when it hits 100%. Not because it’s bad—LVM-thin is fast, simple, and good enough for a lot of clusters—but because it’s a different kind of “space accounting” than a classic filesystem.
Thin provisioning: promised space vs real blocks
LVM-thin lets you create virtual volumes (thin LVs) that claim a size—say 200G—but only consume physical blocks in the pool as writes happen. That’s thin provisioning. It makes it easy to overcommit storage. It also makes it easy to get surprised when reality catches up.
Snapshots are not free, they’re just delayed bills
LVM-thin snapshots are copy-on-write. The snapshot itself starts small, but every block that changes while the snapshot exists has to be copied and tracked. Heavy-write workloads plus “we’ll delete snapshots later” equals a thin pool that grows like an unchecked log file.
Two limits: data and metadata
Thin pools have data space and metadata space. Either one hitting 100% can ruin your day. Metadata is what tracks which blocks are allocated to which thin volumes. If metadata fills, allocations can fail even if data has room. If data fills, writes fail even if metadata has room. They fail differently and require different interventions.
Paraphrased idea, attributed to Werner Vogels (Amazon CTO): Everything fails eventually; resilience comes from designing for that reality, not wishing it away.
Fast diagnosis playbook (do this first)
You want fast signal, not a deep spiritual journey through LVM internals. Here’s the shortest path to “what’s full, why, and what do we do right now?”
First: confirm whether it’s data, metadata, or both
Run lvs with thin pool fields. If Data% is near 100, you need to free blocks or add space. If Meta% is near 100, you need metadata extension/repair and to stop the growth triggers (often snapshots).
Second: identify the biggest consumers and “dead” volumes
List thin LVs sorted by actual used blocks. Then cross-check what Proxmox thinks exists versus what LVM has. Orphaned disks are common after failed restores, interrupted migrations, or manual tinkering.
Third: stop the bleeding
Backups, restores, migrations, and snapshot-heavy jobs should be paused while you recover. Otherwise you’re bailing water with a hole in the boat.
Fourth: choose the least risky recovery action
- Delete unneeded snapshots (fast win).
- Delete truly orphaned volumes (high win, medium risk if you guess wrong).
- Enable/discourage TRIM depending on reality (longer win).
- Extend the thin pool (best if you have free PV space).
Interesting facts and short history (because it explains the weirdness)
- LVM has been in Linux since the 2.4 era, and LVM2 became the standard approach as distros matured and SANs became normal.
- LVM-thin arrived in the early 2010s as Linux needed thin provisioning and snapshots without pulling in a full CoW filesystem stack.
- Thin pool metadata is stored on-disk, not just in memory; if it’s corrupted or full, you can end up with read-only behavior or failed allocations.
- Proxmox default partitioning historically favored “local + local-lvm” because it matched common VM workflows: ISO storage on filesystem, disks on block storage.
- TRIM/Discard for thin provisioning was controversial for years due to performance and correctness concerns; it’s much better now, but still needs intentional enabling end-to-end.
- Snapshots are operationally expensive across almost every storage tech—LVM-thin, qcow2, SAN arrays—because they turn overwrites into allocate-and-copy operations.
- Metadata exhaustion often shows up “suddenly” because it’s not proportional to bytes written; it’s proportional to allocation mapping and snapshot activity.
- Overprovisioning is a business feature as much as a technical one: it improves utilization, but it’s fundamentally a bet that not everything grows at once.
Practical tasks: commands, meaning, and decisions (the meat)
Below are hands-on tasks you can run on a Proxmox node. Each task includes: a command, what the output means, and what decision you make from it. Assume your Proxmox host is Debian-based and you have root or sudo.
Task 1: Confirm Proxmox storage view vs reality
cr0x@server:~$ pvesm status
Name Type Status Total Used Available %
local dir active 98.00GB 12.40GB 80.60GB 12.65%
local-lvm lvmthin active 700.00GB 699.20GB 0.80GB 99.89%Meaning: Proxmox thinks local-lvm is basically full. That’s the symptom, not the diagnosis.
Decision: Move immediately to LVM view. Proxmox is reporting what LVM tells it; you need to see why it’s full (data vs metadata, biggest thin LVs, snapshots).
Task 2: See thin pool health (data and metadata)
cr0x@server:~$ lvs -o+lv_size,segtype,data_percent,metadata_percent,lv_attr,origin,pool_lv --units g
LV VG Attr LSize SegType Data% Meta% Origin Pool
data pve twi-aotz-- 650.00g thin-pool 99.85 12.40 data
root pve -wi-ao---- 60.00g linear
swap pve -wi-ao---- 8.00g linearMeaning: data is the thin pool. Data% is almost 100%, Meta% is fine. You are data-full, not metadata-full.
Decision: Your first recovery options are freeing blocks (delete snapshots/volumes) or extending the pool. Metadata repair isn’t your primary issue.
Task 3: If metadata is the problem, detect it explicitly
cr0x@server:~$ lvs -a -o lv_name,vg_name,lv_size,data_percent,metadata_percent,lv_attr --units g
LV VG LSize Data% Meta% Attr
data pve 650.00g 78.10 99.60 twi-aotz--
data_tmeta pve 4.00g ewi-ao----
data_tdata pve 650.00g ewi-ao----Meaning: Data usage is okay-ish, but metadata is near full. That’s “we’ll run out of mapping space soon,” which can be even worse than data-full because it can happen while you still have plenty of free blocks.
Decision: Stop snapshot churn, consider extending metadata, and plan for a repair if errors appear. Don’t “just keep writing.” That’s how thin pools get corrupted.
Task 4: Find which thin volumes are actually consuming blocks
cr0x@server:~$ lvs -o lv_name,lv_size,data_percent,metadata_percent,lv_attr --units g --sort -data_percent
LV LSize Data% Meta% Attr
vm-103-disk-0 200.00g 98.50 Vwi-aotz--
vm-101-disk-0 150.00g 92.10 Vwi-aotz--
vm-102-disk-0 80.00g 88.40 Vwi-aotz--
vm-120-disk-0 500.00g 10.20 Vwi-aotz--Meaning: Data% here is per thin LV’s allocated blocks relative to its virtual size. The ones near 100% are close to “fully allocated.” They are consuming real pool space.
Decision: Identify whether these are legitimate big disks doing legitimate work, or if something is off: old restores, unused VMs, forgotten snapshots.
Task 5: Map a volume back to a VM and config
cr0x@server:~$ qm config 103
boot: order=scsi0;ide2;net0
cores: 4
memory: 8192
name: app-prod-03
scsi0: local-lvm:vm-103-disk-0,size=200G
ide2: local:iso/debian-12.iso,media=cdrom
net0: virtio=DE:AD:BE:EF:10:03,bridge=vmbr0Meaning: vm-103-disk-0 is attached to VM 103. Now you can coordinate cleanup with application owners (or your future self).
Decision: If the VM is critical, you don’t delete it. You look for snapshots, guest cleanup, or expansion. If it’s a zombie VM, you clean aggressively.
Task 6: List snapshots for a VM (thin snapshots often hide here)
cr0x@server:~$ qm listsnapshot 103
`-> pre-upgrade-2024-11-15
`-> before-hotfix
`-> backup-tempMeaning: Snapshots exist. Each snapshot can pin old blocks and force CoW on writes, inflating pool usage.
Decision: If these snapshots are not actively needed, delete them. Keep only what you can justify in one sentence.
Task 7: Delete a snapshot safely
cr0x@server:~$ qm delsnapshot 103 backup-temp
Deleting snapshot 'backup-temp'...
TASK OKMeaning: Proxmox removed the snapshot. LVM space reclamation may not be immediate if blocks are still referenced elsewhere, but often you’ll see pool usage drop quickly.
Decision: Re-check lvs. If space doesn’t move, the pool is full for other reasons (or TRIM isn’t happening).
Task 8: Check for orphaned LVs (exist in LVM, not in Proxmox configs)
cr0x@server:~$ pvesm list local-lvm
Volid Format Type Size VMID
local-lvm:vm-101-disk-0 raw images 161061273600 101
local-lvm:vm-102-disk-0 raw images 85899345920 102
local-lvm:vm-103-disk-0 raw images 214748364800 103cr0x@server:~$ lvs --noheadings -o lv_name pve | sed 's/^[ \t]*//'
data
root
swap
vm-101-disk-0
vm-102-disk-0
vm-103-disk-0
vm-999-disk-0Meaning: vm-999-disk-0 exists in LVM but isn’t listed by Proxmox storage. That’s suspicious.
Decision: Investigate before deleting. It could be a leftover from a failed restore, or it could be referenced in an old config on another node if you did something creative.
Task 9: Verify if a suspicious LV is referenced anywhere
cr0x@server:~$ grep -R "vm-999-disk-0" /etc/pve/qemu-server /etc/pve/lxc 2>/dev/null || echo "no references found"
no references foundMeaning: Proxmox configs don’t mention it. That’s a strong signal it’s orphaned.
Decision: If you also confirm it’s not mounted/used by anything else, you can remove it to reclaim space. If unsure, snapshot/backup host metadata first or rename LV as a soft-delete.
Task 10: Soft-delete by renaming an LV (buy time)
cr0x@server:~$ lvrename pve vm-999-disk-0 vm-999-disk-0.ORPHANED
Renamed "vm-999-disk-0" to "vm-999-disk-0.ORPHANED" in volume group "pve"Meaning: You’ve removed the predictable name without destroying data. If something breaks, you can rename it back. If nothing screams, you can delete later.
Decision: If you’re under pressure, renaming is safer than deleting. Then schedule actual deletion during a calmer window.
Task 11: Hard-delete an orphaned LV to reclaim space
cr0x@server:~$ lvremove -y /dev/pve/vm-999-disk-0.ORPHANED
Logical volume "vm-999-disk-0.ORPHANED" successfully removed.Meaning: The thin LV is gone. That frees its referenced blocks in the thin pool.
Decision: Immediately re-check pool usage. If the pool drops significantly, you found the eater. If not, keep hunting.
Task 12: Confirm pool usage change
cr0x@server:~$ lvs -o lv_name,lv_size,data_percent,metadata_percent,lv_attr --units g
LV LSize Data% Meta% Attr
data 650.00g 92.30 12.45 twi-aotz--Meaning: You recovered data space: from ~99.8% to ~92.3%. Not perfect, but you’re out of “imminent crash” territory.
Decision: Don’t stop here. Thin pools don’t forgive complacency. Get to a stable free margin (at least 10–20% free for busy nodes).
Task 13: If you have free space in the VG, extend the thin pool (cleanest fix)
cr0x@server:~$ vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 5 0 wz--n- 930.00g 280.00gcr0x@server:~$ lvextend -L +200G /dev/pve/data
Size of logical volume pve/data changed from 650.00 GiB (166400 extents) to 850.00 GiB (217600 extents).
Logical volume pve/data successfully resized.Meaning: You had free extents in the volume group and used them to extend the thin pool.
Decision: Extending buys time and reduces operational stress. Still fix the root cause; otherwise you just made a bigger bucket for the same leak.
Task 14: Extend thin pool metadata if Meta% is high
cr0x@server:~$ lvextend --poolmetadatasize +2G /dev/pve/data
Size of logical volume pve/data_tmeta changed from 4.00 GiB (1024 extents) to 6.00 GiB (1536 extents).
Logical volume pve/data_tmeta successfully resized.Meaning: More metadata space for the thin pool. This is often needed when you’ve done lots of snapshotting or have many thin volumes.
Decision: If you were at Meta% > 90%, do this promptly. Then reduce snapshot churn so you don’t just fill the new metadata too.
Task 15: Check if discards are enabled at the thin pool level
cr0x@server:~$ lvs -o lv_name,lv_attr,discards,zero
LV Attr Discards Zero
data twi-aotz-- passdown yesMeaning: passdown means discards from thin LVs can be passed to the pool. This is good for reclamation, assuming the guest actually issues discards.
Decision: If discards are ignored and you expect TRIM, adjust. If you don’t want discards due to workload characteristics, accept that reclaim requires other methods and more headroom.
Task 16: Trigger fstrim inside a Linux guest (where the real reclaim happens)
cr0x@server:~$ qm guest exec 103 -- fstrim -av
/boot: 0 B (0 bytes) trimmed
/: 24.6 GiB (26411479040 bytes) trimmedMeaning: The guest informed the storage stack which blocks are no longer in use. If everything is wired correctly (virtio-scsi + discard enabled + thin pool passdown), thin pool usage can drop.
Decision: If trims reclaim meaningful space, schedule fstrim regularly (or enable continuous discard cautiously). If trims trim 0 bytes forever, you have a configuration gap.
Task 17: If you suspect the pool is stuck, check dmesg and thin errors
cr0x@server:~$ dmesg | tail -n 20
[12345.678901] device-mapper: thin: 253:2: reached low water mark of 32768 blocks
[12345.678950] device-mapper: thin: 253:2: no free space for data blockMeaning: Kernel thin target is complaining about low space / no free data blocks. If it says metadata instead, that’s a different emergency.
Decision: Stop writes (pause backups/migrations), free space or extend immediately. Continuing heavy IO here is how you graduate from “storage incident” to “data recovery project.”
Task 18: Identify the biggest writes from inside Proxmox (rough triage)
cr0x@server:~$ iotop -o -b -n 3 | head -n 20
Total DISK READ: 0.00 B/s | Total DISK WRITE: 48.32 M/s
PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
8421 be/4 root 0.00 B/s 22.10 M/s 0.00% 99.00% vzdump --compress zstd 101
9102 be/4 root 0.00 B/s 15.40 M/s 0.00% 97.00% kvm -id 103 -name app-prod-03
7766 be/4 root 0.00 B/s 9.90 M/s 0.00% 88.00% kvm -id 102 -name db-prod-02Meaning: Backups (vzdump) and specific VMs are generating writes. On a nearly-full thin pool, write bursts can push you over the edge fast.
Decision: Pause backup jobs temporarily. If a specific VM is runaway-writing (logs, temp files, compaction), fix it in-guest or throttle it.
Joke 1/2: Thin provisioning is like ordering a diet soda with a triple burger—technically possible, emotionally misleading.
Find what ate the pool: the usual suspects
Suspect A: snapshots that lived too long
Snapshots are meant to be short-lived. “Short-lived” means hours or days, not quarters. A busy VM with an old snapshot forces every changed block to be copied, and those blocks remain pinned until the snapshot is gone.
Practical tell: your pool grows steadily even though guest filesystems claim they’re stable. Another tell: qm listsnapshot returns a small tree that hides an expensive CoW tax.
Suspect B: backups/restores/migrations that left junk behind
Proxmox is generally good at cleaning up. But if a node crashes mid-restore, a storage hiccup interrupts a migration, or a human kills a task mid-flight, you can get orphaned thin volumes. These are “real” consumers and invisible in day-to-day UI views unless you compare pvesm list with lvs.
Operational reality: orphaned volumes are common enough that you should treat “reconcile LVM vs Proxmox configs” as a standard incident step.
Suspect C: guests that never discard (TRIM is not automatic)
A thin pool only learns about “freed” blocks if discards are issued and propagated. Deleting 100 GB of files inside a guest does not necessarily free 100 GB in the thin pool. Without discard/TRIM, the pool sees only writes, never forgets allocations, and slowly approaches 100%.
Windows guests need scheduled “Optimize Drives” (TRIM). Linux guests need periodic fstrim or mount options that support discard. And the virtual disk/controller must support it, plus Proxmox must not block it.
Suspect D: qcow2 in the wrong place (or raw with snapshots elsewhere)
On local-lvm (LVM-thin), Proxmox typically uses raw volumes, which is good. But if you store qcow2 on a directory storage (like local) and then snapshot at the qcow2 level, you get a different set of growth mechanics. Sometimes admins migrate disks between storages and end up with unexpected formats and snapshot layers.
Rule of thumb: don’t stack CoW-on-CoW unless you’re deliberately testing suffering.
Suspect E: a single VM doing “legitimate” but explosive writes
Common examples: databases doing compaction, log storms, runaway metrics retention, or a guest running a backup that writes a full image to its own disk. Thin provisioning makes this worse because you don’t feel the pressure until you do.
Suspect F: metadata pressure from many snapshots or many small allocations
Even if you’re not writing a ton of bytes, you can burn metadata with snapshot churn and allocation patterns. Metadata growth can appear unrelated to “used GB,” which is why people get blindsided.
Fix it now: safe recovery moves
1) Pause the churn
Stop scheduled backups, replication, and any automated snapshot jobs. Don’t do storage gymnastics while something is still writing hard.
cr0x@server:~$ systemctl stop pve-ha-lrm pve-ha-crm 2>/dev/null || true
cr0x@server:~$ systemctl stop cron 2>/dev/null || trueMeaning: This is a blunt instrument; use discretion. In clusters, consider the impact and coordinate. The goal is to reduce background tasks that create writes and snapshots.
Decision: If you can’t stop automation globally, at least stop the active backup/restore tasks and avoid starting new ones until space is stable.
2) Delete snapshots you don’t need
This is the highest ROI fix when snapshots are the cause. Delete from Proxmox so it cleans the whole chain correctly.
cr0x@server:~$ qm listsnapshot 101
`-> pre-upgrade
`-> weekly-safetycr0x@server:~$ qm delsnapshot 101 weekly-safety
Deleting snapshot 'weekly-safety'...
TASK OKMeaning: Snapshot removal can take time if the VM is active and there’s merge work. On LVM-thin it’s usually quicker than qcow2 merges, but still not instant if IO is heavy.
Decision: Re-check thin pool usage after each major deletion. Don’t delete everything blindly if you need a rollback point for an in-progress risky change.
3) Remove orphans (carefully)
If you have volumes not referenced by any VM config, reclaim them. Prefer rename-first if you’re unsure.
4) Extend the pool (if you can)
If your VG has free extents, extending /dev/pve/data is clean and low drama. If it doesn’t, you can add a new disk as a PV and extend the VG—just remember performance and failure domains.
cr0x@server:~$ pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.cr0x@server:~$ vgextend pve /dev/sdb
Volume group "pve" successfully extendedcr0x@server:~$ lvextend -L +500G /dev/pve/data
Size of logical volume pve/data changed from 850.00 GiB (217600 extents) to 1350.00 GiB (345600 extents).
Logical volume pve/data successfully resized.Meaning: You increased the thin pool capacity. Data gets breathing room immediately.
Decision: Extending is not a substitute for housekeeping. If you don’t fix the eater, you’ll just hit 100% again—now at 2 a.m. on a weekend, because storage has a sense of humor.
5) Address metadata emergencies properly
If metadata is full, the priority is to prevent further metadata consumption and regain metadata headroom. Extending metadata is often possible online, but if the pool is already in an error state you may need repair tooling and downtime planning.
Check for errors first:
cr0x@server:~$ lvs -o lv_name,lv_attr,seg_monitor,devices pve
LV Attr Cpy%Sync Monitor Devices
data twi-aotz-- monitored /dev/sda3(0)Meaning: Monitoring can help auto-extend in some setups, but don’t rely on it unless you configured it intentionally.
Decision: If Meta% is high, extend metadata and then reduce snapshot activity. If you see I/O errors, stop and plan a repair window.
6) Make TRIM real (end-to-end)
TRIM is only useful if it traverses the stack: guest filesystem → guest block layer → virtual disk → host block layer → thin pool.
On Proxmox, ensure the disk uses a controller that supports discard well (virtio-scsi is commonly used). In the VM config, you typically want discard enabled. Then schedule fstrim in guests.
Prevent the repeat: boring controls that work
Headroom is a policy, not a hope
Thin provisioning is fine. Running a thin pool at 90–95% on a busy node is not fine. Your safe operating envelope depends on write rate and recovery speed. In most production environments, I want at least 15–20% free data on thin pools that host write-heavy VMs.
Snapshot hygiene: time limits, not feelings
Put a TTL on snapshots. If a snapshot exists longer than the reason it was created, it’s not “safety,” it’s silent technical debt.
Regular reconciliation: LVM vs Proxmox
Once a month (or after any incident involving restores/migrations), compare thin volumes to VM configs. Catch orphans while they’re small and you’re not under the gun.
Trim schedule in guests
For Linux guests, a weekly fstrim.timer is usually enough. For Windows, schedule optimization (TRIM). For databases, coordinate with maintenance windows if you’re worried about IO spikes.
Alert on Data% and Meta% separately
Most alerting stacks watch only “storage usage.” That misses metadata. If you alert at 80/90% for data but never for metadata, you’re watching the wrong cliff.
Joke 2/2: A thin pool at 100% is nature’s way of reminding you that “unlimited” was always marketing.
Common mistakes: symptom → root cause → fix
1) Symptom: local-lvm shows 100%, but “df -h” looks fine
Root cause: local-lvm is block storage; df reports filesystem usage on mounted filesystems, not LVM-thin pool consumption.
Fix: Use lvs and pvesm status. Treat Data% and Meta% as first-class metrics.
2) Symptom: VM writes fail or filesystem goes read-only during backup
Root cause: thin pool data full; backup spikes writes (snapshots, temporary volumes, CoW overhead).
Fix: Stop backups, free space (snapshots/orphans), or extend pool. Then re-run backups with headroom.
3) Symptom: thin pool has space, but new allocations fail
Root cause: metadata full (Meta% high) or metadata errors.
Fix: Extend metadata (lvextend --poolmetadatasize) and reduce snapshot churn. Investigate for thin errors in dmesg.
4) Symptom: you deleted lots of files inside a VM, but pool usage didn’t drop
Root cause: no TRIM/discard propagation; blocks remain allocated in thin pool.
Fix: Enable discard end-to-end and run fstrim in the guest. Validate with reclaimed bytes and changing pool usage.
5) Symptom: pool usage keeps growing even when VMs are “idle”
Root cause: snapshots pin blocks; background jobs (log rotation, antivirus, indexing) rewrite data; backup tooling creates snapshot churn.
Fix: Remove old snapshots, reduce churn jobs, and stop storing “forever snapshots” on production thin pools.
6) Symptom: pool filled “overnight” after you resized a VM disk
Root cause: thin provisioning overcommit plus a guest filesystem expand/fill, or a restore wrote a full image into the thin volume.
Fix: Audit overcommit. Put caps on growth, increase headroom, and alert earlier.
Checklists / step-by-step plan
Step-by-step: recover from local-lvm at 100% (minimal risk)
- Confirm whether data or metadata is full (
lvs -o data_percent,metadata_percent). - Pause backup/restore/migration tasks creating churn.
- List per-VM disk usage (LVM thin LVs) and identify top consumers.
- Check snapshots on top consumers; delete nonessential snapshots first.
- Re-check pool usage after each deletion.
- Search for orphaned volumes by diffing
pvesm listandlvs. - Rename orphans as a safety step; delete after confirmation.
- If VG has free space, extend thin pool and/or metadata.
- Enable and validate TRIM end-to-end; run guest
fstrim. - Restart paused jobs only after you’ve restored headroom.
Operational checklist: prevent recurrence
- Alert on thin pool
Data%andMeta%separately. - Set a snapshot TTL policy and enforce it (automation helps).
- Monthly orphan scan: reconcile LVM volumes with Proxmox configs.
- Quarterly capacity review: thin overcommit ratio, growth trends, and “worst-case simultaneous growth” scenario.
- Guest trim schedule: verify it with real reclaimed bytes, not good intentions.
Three corporate mini-stories from the thin pool trenches
Mini-story 1: The incident caused by a wrong assumption
They had a small Proxmox cluster running internal services: build agents, artifact storage, a couple of database VMs, and “temporary” test environments that became very permanent. The team watched disk usage via a standard host metric: filesystem percent used on /. It stayed green.
Then one Monday morning, several VMs went read-only. The backup job failed. The web UI threw errors that looked like permissions issues. The on-call did what on-calls do: rebooted the noisiest VM first. It came back worse.
The wrong assumption was simple: “If the host filesystem has space, the VM storage has space.” But local-lvm was its own thin pool, separate from /. The thin pool had hit 100% late Sunday during a quiet backup window—quiet for humans, not for storage.
Recovery was straightforward once they looked in the right place: lvs showed data at 99.9%. A long-lived snapshot on a write-heavy database VM was pinning blocks. Deleting the snapshot freed enough space to stabilize, then they extended the pool and added alerts for Data% and Meta%.
What changed culturally was better: they stopped treating thin provisioning as “magic extra storage” and started treating it as a capacity planning tool with explicit risk. The incident wasn’t caused by LVM-thin. It was caused by an assumption no one had written down, so no one had questioned.
Mini-story 2: The optimization that backfired
A different shop wanted faster backups. Someone noticed that snapshot-based backups were “instant” at the snapshot step, and reasoned they could take snapshots more frequently to reduce recovery point objectives. They automated a snapshot every hour for key VMs and kept them for two weeks. It looked responsible on paper.
The first week was fine. The second week, performance got weird: latency spikes, occasional IO stalls, and growing storage usage that didn’t match application growth. The thin pool wasn’t just filling; it was filling in a pattern that correlated with write-heavy batch jobs.
Hourly snapshots meant constant CoW overhead. The “optimization” increased write amplification, burned metadata faster, and pinned old blocks for two weeks—exactly long enough to turn normal churn into persistent allocation. They hadn’t reduced backups; they’d added a second backup-like retention mechanism inside the storage layer.
They rolled back to a saner approach: one short-lived pre-change snapshot during maintenance windows, plus real backups with tested restores. Snapshot frequency dropped, performance stabilized, and the thin pool stopped behaving like a slowly inflating balloon.
The takeaway: snapshots are not a free backup system. They’re a sharp tool for short-term rollback. Use them like a scalpel, not like a filing cabinet.
Mini-story 3: The boring but correct practice that saved the day
A finance-adjacent team (the kind that makes auditors happy and engineers mildly annoyed) ran Proxmox for a few dozen VMs. Their approach was unsexy: capacity thresholds, headroom rules, and scheduled housekeeping. Every VM had a documented owner. Every snapshot had an expiration.
One afternoon a restore test went sideways. A large VM restore was interrupted mid-stream due to a network hiccup. The operator retried, and the second attempt succeeded. Nobody noticed that the first attempt left an orphaned thin LV behind.
Two weeks later, the monthly “storage reconciliation” job flagged an LV present in LVM but not referenced by any Proxmox config. The report didn’t delete anything automatically; it just created a ticket with the LV name, size, and creation time. Someone verified it wasn’t referenced and removed it. Space returned.
Nothing broke. No outage. No emergency cleanup. No weekend ruined.
The practice wasn’t clever. It was the opposite: routine auditing with a bias toward safe actions (rename-first, then delete). That boredom is what prevented a future 100% thin pool incident from happening on a day when the team was already busy with other disasters.
FAQ
Why does local-lvm hit 100% when my VMs still show free space inside?
Because thin pool usage tracks allocated blocks, not guest “free space.” Deleting files in-guest may not return blocks without TRIM/discard. Snapshots can also pin old blocks.
Is local-lvm the same as LVM on the host root disk?
No. It’s typically a thin pool LV inside a VG (often pve), separate from your root filesystem LV. Different accounting, different failure modes.
What’s more dangerous: Data% at 99% or Meta% at 99%?
Both are bad. Metadata at 99% can be nastier because allocations can fail unexpectedly and recovery may be more delicate. Treat either as an incident.
Can I just delete random “unused disks” from the GUI?
You can delete unused disks that Proxmox knows about. The dangerous part is volumes Proxmox doesn’t list (orphans). For those, reconcile configs first; rename-before-delete is a good safety pattern.
Why didn’t deleting a snapshot free space immediately?
If other snapshots still reference blocks, or if the workload rewrote blocks heavily, space may not drop much. Also, you might be data-full due to other volumes, not snapshots.
Does enabling discard hurt performance?
It can, depending on workload and underlying storage. Many environments do fine with periodic fstrim rather than continuous discard. Measure on your hardware.
How do I know which VM is the biggest space consumer?
Use lvs sorted by per-LV Data% and size, and map volumes back to VMs with qm config <vmid>. Also check snapshots and backups for write spikes.
Can I shrink a thin LV to reclaim pool space?
Not safely as a first-line fix. Shrinking virtual disk sizes requires in-guest filesystem shrink and careful block-level operations. Prefer deleting snapshots/orphans, trimming, or extending the pool.
Should I move VM disks off local-lvm to something else?
If you need predictable reclamation, replication, and visibility, consider storage with stronger management primitives (or separate dedicated storage per tier). But local-lvm is fine when you enforce headroom, snapshot TTLs, and trimming.
What’s the fastest “I need space now” move?
Delete unneeded snapshots and remove confirmed orphaned volumes. If you have VG free space, extending the thin pool is also fast and low risk.
Conclusion: next steps you should actually do
When Proxmox local-lvm hits 100%, the fix is rarely mystical. It’s usually one of four things: snapshots that overstayed, orphaned volumes, lack of TRIM, or plain old capacity shortfall. The win comes from diagnosing which one before you start deleting things you’ll regret.
Do these next:
- Run
lvsand recordData%andMeta%. That’s your ground truth. - Delete snapshots you don’t need, starting with the busiest VMs.
- Reconcile
pvesm listvslvsand remove confirmed orphans (rename first if you’re cautious). - Extend the thin pool if you have VG space; extend metadata if Meta% is high.
- Make TRIM real: enable it end-to-end and verify reclaimed bytes with
fstrim. - Set alerts and a snapshot TTL policy so you don’t do this again under pressure.