The Disk That Lied About Its Health
My Proxmox box died on a Tuesday. Not dramatically — no smoke, no spark, no warning. It just stopped booting and dropped me to an initramfs command line, staring at a shell prompt on a machine that was supposed to be running four virtual machines and a backup target.
The box is an old Dell OptiPlex with a 2TB Seagate drive and a 120GB Kingston SSD. I’d set it up as a Proxmox node months ago, intended for ZFS backups and test VMs. It had been sitting mostly dormant — the kind of project you build on a weekend and then forget about until you need it.
Now I needed it, and it wasn’t cooperating.
The numbers that should have scared me
Before trying to recover anything, I ran SMART diagnostics. The drive reported PASSED — which should have been reassuring, except the raw numbers told a completely different story.
11,040 reallocated sectors. 240 sectors queued for reallocation. 240 sectors already unreadable — actual data loss, already happened, silently. The raw read error rate was 183 million. The drive had been running for 26,723 hours — about three years of continuous operation.
SMART said “PASSED” because the normalised values were still above their thresholds. But normalised values are like a credit score — they tell you where you are relative to failure, not whether you’re already in trouble. The drive was in advanced failure. The filesystem corruption I’d been blaming on an unclean shutdown wasn’t a shutdown problem. The disk was rotting.
The recovery attempt
At initramfs, the options are limited but real. LVM was there. The volume group just hadn’t activated during boot — the disk was too slow to respond before the timeout. I activated the VG, ran a read-only filesystem check, and got back a wall of inode errors that told me exactly one thing: this wasn’t getting repaired.
The smart move was to grab the configs and reinstall. The honest move was to spend twenty minutes trying to fix it anyway, because the configs were inside the broken filesystem and I wanted them back.
I got some of them. Enough to make the reinstall painless. Not enough to pretend the drive was coming back.
The reinstall
Debian 12 on the 120GB Kingston SSD — the healthy drive. Netboot.xyz made it easy: pick the OS, point at the right disk, uncheck every package except the base system. Twenty minutes later I had a clean Debian install, thirty seconds of repo setup, and Proxmox running on top of it.
The 2TB Seagate? It’s still in the box. I use it for scratch VMs I don’t care about losing. Every time I see those SMART numbers in my monitoring dashboard, I’m reminded that “PASSED” doesn’t mean “fine.”
The lesson wasn’t about Proxmox or Debian or initramfs. It was about the gap between what a system reports and what’s actually happening. The drive said it was healthy. The filesystem said it wasn’t. The numbers said it had been dying for months.
I’d rather own the problem than rent someone else’s false confidence.