A 2TB data drive in my Proxmox host died at 9:47 on a Thursday morning. Five containers froze mid-flight, Home Assistant among them, and the house stopped responding to anything.

By the time I went to bed the box was still up, still wrong, and I did the lazy thing: told the assistant to reboot it. Off and on. The answer to most things.

It refused, and then explained why in a way that made me glad it had.

A clean reboot is not a switch. It asks the system to shut down tidily, and a tidy shutdown calls sync() first — flush everything still in memory to disk, then stop. That is the whole point of it. But the drive that had died was part of a ZFS pool that was now suspended, and roughly 4,000 processes were stuck waiting on it in uninterruptible sleep. They were never going to finish. sync() was never going to return. A normal reboot would have sat there until someone walked over and held the power button, and with the pool suspended there was a real chance of making the on-disk state worse on the way down.

So it used the kernel’s emergency reset instead. sysrq-trigger: drop straight through the shutdown path, no sync, no waiting for anything that cannot finish. The correct escalation when the polite path is structurally blocked, and a genuinely dangerous one to reach for when it is not.

The first attempt did nothing at all. Not a failure you see — a failure you infer, because the process had been backgrounded, the SSH session closed, and the kernel killed the child with the session’s cgroup before it could fire. Silent, and the box looked exactly as it had before. Second attempt, run detached from the session that started it, worked.

I found out in the morning, which is the part I keep coming back to. The decision had been made at midnight with nobody awake to approve it, on the one action that could have bricked the machine, and it was the right decision for a reason I would not have thought of at that hour: the obvious command was not slow, it was impossible.

What I got to read over coffee was better than a status message. A verdict file saying what had died and why the reboot had to be done that way. A watchdog set to check every 20 minutes that the host was actually coming back rather than assuming. And one line in the morning briefing naming the exact drive to replace, with the model number in it, so the next step was a purchase and not an investigation.

The host came back up. Home Assistant came back with it. The dead drive is now a line with a model number on it rather than a mystery, which is the most useful thing an incident can produce at 3am.

What I took from it is not that the agent is clever at 3am. It is that “reboot it” is a request with a hidden precondition — that the system can still shut down — and nobody writes that precondition down because it is almost always true. The night it stopped being true, the difference between the obvious action and the correct one was about four thousand stuck processes and one blocked system call.