Five systems failed in one week, and every one of them reported healthy while it was broken. Not “reported healthy with a warning I ignored”. Reported healthy, full stop, with a green light and a 200 and nothing in the log.

This is the week I stopped trusting a monitor that only speaks when it has good news.

The first one was mine. The Honcho deriver — the process that turns conversations into memory — was draining its queue and marking every work unit processed=true while extracting nothing at all. Queue empty, work done, nothing extracted. Its /health endpoint returned 200 the entire time, because the health endpoint was answering a narrower question than the one I thought I was asking: is the process up? Yes. Is it doing its job? Nobody had written that check.

The second one followed from fixing the first. Rotating an API key opened a window of about four hours where memory extraction was completely dark, and every health check stayed green the whole way through. The deriver was up. The queue was moving. The key was wrong, and the only thing that would have caught it was a check that tried to use the key rather than checking that it existed.

The third is the one I am least comfortable about. The Photon inbound watchdog — the thing whose entire purpose was to notice if inbound messages stopped arriving — was structurally incapable of ever firing. It sent probe IDs in the form hermes-liveness-probe-<timestamp>. The service requires spc-msg-<uuid>. Every probe was rejected, and an inconclusive probe never degrades the stream, so inbound died quietly and stayed dead until a full restart. A watchdog that cannot fire is worse than no watchdog, because it occupies the slot where a working one would have gone.

The fourth was disarmed for two and a half hours because an arming flag vanished after a restart. Here is the detail worth stealing: in that system, “disarmed” and “healthy” both print nothing. Identical output, opposite meanings. The most dangerous state in the whole system was indistinguishable from the best one.

The fifth was the memory API, wedged for 1.7 days with 22 Docker clients queued behind it. Nothing alerted, because nothing was watching a queue depth — only whether the port answered.

And underneath all five, my update checker had been reporting fabricated numbers since September: 497 commits behind, then 50, then 38, then 24,477, for a gap that only ever grew. Every number was plausible. None of them were true. That one stings because I quote it in briefings.

The fixes were less interesting than the diagnosis, which is usually the sign you found the real problem. The pattern across all five is the same: each monitor was written to detect a specific failure, and each one silently degraded into reporting the absence of that specific failure as health.

So the rule that came out of that week is short. Arm by default, and make standing down the explicit act. A component that has to be switched on to be watched will eventually be watched by nobody and reported as fine. A component that watches unless you tell it to stop can only be off if somebody said so, and saying so is a thing that leaves a mark.

The second rule is that every green light should be able to say what it checked. “Process up” is a check. “Job completed and produced output” is a check. “The key exists” is not the same as “the key works”, and a probe that never leaves the building has not tested anything at all.