15:27, and the push that didn’t happen

At 15:27 on Thursday 5 November, a push to my brain repository was refused. Not by a merge conflict, not by a failing test. A secret scanner stopped it because it found a credential in the tree, and the tree was 82 commits ahead of origin with more work queued behind it.

I had a backlog to clear that afternoon. Instead the afternoon became a stop-the-line job, and my first assumption was the obvious one: a key hard-coded into a script, or a config file committed by accident. I have made that mistake before. It is the mistake every scanner is built to catch.

It wasn’t in the code. It was on line 128 of a conversation page.

What the brain actually is

The brain is a git repository of markdown pages. One page per decision, per entity, per incident, per conversation worth remembering. My agent reads it and writes to it all day. It is not a chat log I occasionally consult; it is the thing that makes the agent useful across sessions, because it can look up what we settled three weeks ago instead of asking me again.

Which means the repository is not a codebase with some notes in it. It is a record of things I have said. And things I have said include things I should not have said in a chat window.

There is a second layer under the pages. A rebuild regenerates them from the raw session database, so any page that has drifted, or been edited by hand, or been lost, can be reconstructed from the source. On 5 November that rebuild had regenerated 274 pages. That number turned out to be the whole story: the credential was not in one page by accident. It was downstream of the chat where it was first discussed, and the rebuild had faithfully carried it forward 274 times.

How a token ends up in a memory

The honest sequence is unglamorous. Weeks earlier I had set up a messaging bot, and the last step was handing the bot token to the agent so it could write it into a config file. The fastest way to do that was to paste it into the chat. Thirty seconds of work.

The agent did the right thing with it at the time and the wrong thing with it afterwards. It wrote the config. It also wrote the conversation to the session database, because that is what it is for. Later, the memory rebuild read the session database, found a conversation containing a credential, and turned it into a page. The page went into the repo. The repo went to a remote. Nothing in that chain was a bug in the ordinary sense. Every step did exactly what it was designed to do.

That is what makes this class of leak different from a leaked API key in source. A scanner pointed at code would never have seen it. A scanner pointed at the repository only saw it because the repository contains prose. And the prose was accurate, well formatted, and correctly filed.

The guard that caught it was there by accident

I want to be precise about this, because it would be easy to write a tidier story where I saw the risk coming.

I did not. The pre-push scan exists because of an earlier, unrelated mess in a different repo. It was installed to catch a specific sort of sloppiness I had already been bitten by once, and it happened to run against the brain as well. It caught this because it was thorough, not because it was aimed here. The lesson is not “I planned for this”. The lesson is that a blunt gate applied to everything catches things a clever gate aimed at the right target would have missed, because you rarely know where the right target is until afterwards.

The four hours that mattered

Recovery was four separate jobs, and doing three of them would have left the hole open.

Rotate first. The token was revoked in BotFather before anything else was touched. Rotation is the only step that makes the credential worthless to anyone who already has a copy, and everything else is tidying. The order matters: redact first and you have a false sense of safety while the token still works.

Redact at rest. The page was cleaned, and so were the source rows in the session database. This is the step people skip. If you only clean the page, the next rebuild reads the database and puts the credential back. I had just watched a rebuild do that 274 times, so cleaning the output and leaving the input alone would have been the same mistake with extra steps.

Move the guard outside the repository it protects. The scan had been living inside the thing it was guarding, which is comfortable and wrong. A guard that ships with the repo is a guard that can be removed by whoever has a clone, and a fresh clone gets a fresh copy of the guard along with everything the guard is supposed to inspect. Moving it out means one gate that runs before any repository, independent of what any clone contains. The gate belongs at the boundary, not inside the room.

Scan the repos that had no gate at all. Two repositories had never been covered. Once I knew the leak class existed, leaving those unwatched was not a defensible position. They went in as report-only, deliberately. A scanner that starts blocking pushes on day one, on repos nobody has calibrated it against, gets disabled by the end of the week. Report-only first, then talk.

Verified, not assumed

As of 8 November, three checks pass. The old token returns Unauthorized, which means the rotation took effect at the provider and not just in my notes. There are zero matches for the token anywhere in the tree. And the repository is back to 0 commits ahead of origin, with the backlog cleared rather than abandoned.

I am not going to quote any part of the token here, or describe its shape, or say which digits it started with. That is not coyness. The whole incident is about a credential surviving in text that was written to be read.

The email that had to say “do not paste it into a chat”

Here is the part I keep coming back to. To get the new token into place, I had to write an email explaining how to hand it over, and that email had to contain the sentence “do not paste it into a chat”.

Pasting it into a chat is exactly how the old one arrived there.

There is no tidy way around that. The instruction has to exist because the convenient path is the dangerous one, and the convenient path is convenient precisely because the agent is useful. You cannot remove the risk by making the agent less useful. You can only put a boundary where the useful thing stops being safe: a secret typed into a form that writes to a file, rather than into a conversation that becomes a memory.

What I actually changed

The easy fix was to add another scanner and call it hygiene. The fix I made instead was to treat memory as a place where secrets can be, which it now obviously is. Rotate anything that touches the system. Redact at ingestion and at rest, both ends, because one end regenerates the other. Gate at the boundary, outside the thing being gated. And cover the repos you have been quietly not covering.

My agent’s memory was the leak vector. Not the code, not a misconfigured service, not a stranger on the internet. The part of the system whose entire purpose is to remember what I told it.

It is still remembering. That’s the point of it. I just have a gate on the door now.

I asked my assistant to create a Meetup account for itself. On paper that is a five-minute job. It took twenty minutes and one interruption, and the interruption is the whole story.

It filled the form without complaint: name, my spare mailbox, Hamilton, the 18-and-over tick. Then it stopped. Meetup puts a human check in front of the Sign up button, and while that check was unticked the button stayed grey and unclickable. I watched it sit there, reading the page, checking the field, checking again. It would not fake being a person. So I clicked the box myself, and the account existed.

That is the only part of the afternoon a machine could not do.

The rest it did alone. It found the verification email in its own mailbox, followed the link, logged in, and joined the club: member 1,345 of Hamilton newbies, up from 1,344. Then I asked it to be less obviously an AI, which is a strange sentence to type. It renamed itself Meetup Assistant and rewrote its bio, because the field stops at 250 characters and its first draft got cut off mid-sentence. Then it answered a message I sent it inside Meetup, in the Meetup thread, four minutes after I sent it. Not in a chat box on my screen. On the platform.

Then came the real work, which turned out to be about attendance rather than copy. A local venue had offered to host us — Escapist, the escape-room and board-game cafe on Bryce Street, with four hundred-odd games on the shelf — and the thing worth drafting first was the event that repeats. Not a one-off night with a date and a poster. Their Board Game Club runs every Friday, six till close, ten dollars, popcorn included, and a standing invitation in front of our members is worth more than any single evening.

The agent read the group, read the venue’s own event page for the parts worth reusing, and drafted it: Friday the sixth of November, six till nine, four topics, the venue’s banner art rather than a stock photo. It saved it as a draft and left it unpublished, because it goes to 1,346 people and publishing is my call, not its.

That is the pattern from here, and it is the part I care about. One recurring Friday is the start, not the plan. Other Hamilton events will be added as they come up — a room, a night, a reason to leave the house — and none of them need building from scratch, because what has been built is the mechanism rather than this particular Friday.

I changed the framing once before we got that far. The copy had read like a group event that happened to have a venue, and I wanted the opposite: Escapist is running this, we are turning up. So the draft grew an opening section titled, in the plainest possible terms, a note on who runs this, and the welcome message and how-to-find-us stopped pointing at us and started pointing at the venue’s staff. An agent that will happily write three hundred words of event copy will also, if you let it, quietly claim credit for the whole thing.

Then the host line, which is where the platform pushed back. I asked it to put my name down as host and take its own name off. Meetup would not have it: an event must always have a host, and while the agent was the only one, its name had no remove button at all. No error, no explanation — the control simply was not there. To stop hosting, it first had to make me the host, and only then did the little cross appear next to its own name. It added me, stepped down, and checked the saved record afterwards rather than trusting the screen: host is my member ID, attendance empty.

Two rules did the work, and neither had anything to do with intelligence. Prove you are a person. Somebody human must be responsible. The agent can write the event, describe the games, answer members and hand the keys over cleanly. It cannot tick a checkbox, and it cannot resign as host until a person exists to take the job. Both times it handed the decision back to me, which is the arrangement I would want anyway.

The event is real, the venue is real, and it is still a draft. The only thing standing between it and 1,346 inboxes is that I have not said go — and the next one after that will take about a tenth of the time.

One morning the PC was sitting in BIOS. Not booting, not failing to boot — parked in the firmware screen with a USB stick plugged in and the hard drive first in the boot order, waiting for somebody to notice.

The lazy explanation is “it rebooted and picked the wrong device”. That explanation is wrong often enough that I asked for the actual trail instead of the plausible one.

Windows keeps a record of unexpected shutdowns, and this one had an Event 41 bugcheck code of 340 — storage class, the family of codes that means the machine went down in a way the storage stack was involved in. That is a direction, not an answer. The answer was in the firmware history.

The SSD was a Samsung 990 PRO, and that model has a firmware lineage with a documented history of exactly this: drives that drop out under load, BSODs that get blamed on everything except the drive, and firmware updates that quietly fix them. A machine that reboots into firmware with a storage-class bugcheck and a 990 PRO inside it is not a mystery, it is a known shape. But you only see the shape if you look up the model instead of the symptom.

Behind that was the trigger. A Windows Update restart had started the sequence — the update was the thing that asked the machine to reboot, and the reboot is where the drive misbehaved. Which means the update was innocent and also the reason we found out: it forced an event that the machine had been failing intermittently at for a while without anyone noticing, because a reboot that happens when you are not looking looks exactly like a reboot.

Two other things were in the wreckage and both mattered. An old NSSM service, left behind by something I had stopped using, spamming errors on every start — not the cause, but noise that makes a log harder to read at the exact moment you need to read it. And a Macrium differential backup caught mid-run by the crash, rescued before it could leave a partially-written image that would have failed silently later, at the worst possible time.

The whole thing went into a playbook: check the bugcheck code and classify it, look up the specific hardware against known firmware issues before believing any other theory, check what Windows Update was doing at the time, audit the services that start with the machine, and then verify the backups that were running when it died.

The drive got its firmware update, the stale service went, and the interrupted backup was verified rather than assumed.

What I like about this one is that nothing in it is clever. Every step is something a careful person would do with the same information. The difference is that the machine did them in order, at 6am, without a theory it was emotionally attached to — and the first theory anyone reaches for, “it booted off the USB stick”, would have left a failing drive in place and a playbook unwritten.

After the NAS lost a volume, I asked the assistant to put my playlists back. It came back a while later with news I did not expect: the playlists were not the NAS’s fault, they were mine. Well — ours. The cron job’s.

Two of the three playlists were fine. The third was 316 bytes of track stubs with no file paths in them. Names, positions, no files. The kind of file that opens without error, lists its tracks, and plays nothing.

The obvious suspect was the storage failure everything else had come from. It was not. A nightly “Movie Themes” job had been rebuilding that playlist from a manifest on the NAS, and when the NAS became unreachable the job kept running. Night after night. Each run read nothing, wrote a fresh file from nothing, and reported success, because from its point of view it had done exactly what it was asked: produce the playlist file. It never asked whether the input existed, and it never compared what it wrote against what had been there.

That is the failure mode worth the whole post: an automation whose input disappeared and which kept writing anyway. It did not crash. It did not error. It did not alert. It quietly replaced a real thing with the shape of a real thing, and it would have kept doing it for as long as the schedule held.

The forensics came before the rebuild, which is the only reason the story has an ending rather than a repeat. Where did the tracks come from originally, which ones were still reachable, what did the file look like the last time it was correct. Once that was on the table the rebuild was mechanical: eight Sonos-native playlists, 386 tracks, pulled back from YouTube Music with yt-dlp.

One track was genuinely gone — the video deleted, unavailable, the original upload pulled. Substituted with the same recording from the artist’s official channel, which is the honest version: same audio, different host, and I would rather have the song than a gap and a note explaining the gap.

Then every playlist was verified by playing it. At volume zero, one track at a time, checking that something reported PLAYING rather than skipping or erroring. Verification at volume zero is a strange thing to do at the end of a rebuild and it is the only check that tells you the file is real instead of merely well-formed. 386 tracks that exist on disk and are not referenced by anything look identical to 386 tracks that work, right up until you press play.

The job is deleted. That was the actual fix. Everything else was repair.

What I keep thinking about is how reasonable it looked. A schedule, a source, a target, a nightly rebuild — the design is unremarkable and it is the design most of us would write. The missing piece was one line of judgement: if the source cannot be read, do not write the target. If the new file is a fraction of the size of the old one, do not replace the old one. Cheap checks, and the difference between an automation that maintains something and an automation that slowly eats it.

Five systems failed in one week, and every one of them reported healthy while it was broken. Not “reported healthy with a warning I ignored”. Reported healthy, full stop, with a green light and a 200 and nothing in the log.

This is the week I stopped trusting a monitor that only speaks when it has good news.

The first one was mine. The Honcho deriver — the process that turns conversations into memory — was draining its queue and marking every work unit processed=true while extracting nothing at all. Queue empty, work done, nothing extracted. Its /health endpoint returned 200 the entire time, because the health endpoint was answering a narrower question than the one I thought I was asking: is the process up? Yes. Is it doing its job? Nobody had written that check.

The second one followed from fixing the first. Rotating an API key opened a window of about four hours where memory extraction was completely dark, and every health check stayed green the whole way through. The deriver was up. The queue was moving. The key was wrong, and the only thing that would have caught it was a check that tried to use the key rather than checking that it existed.

The third is the one I am least comfortable about. The Photon inbound watchdog — the thing whose entire purpose was to notice if inbound messages stopped arriving — was structurally incapable of ever firing. It sent probe IDs in the form hermes-liveness-probe-<timestamp>. The service requires spc-msg-<uuid>. Every probe was rejected, and an inconclusive probe never degrades the stream, so inbound died quietly and stayed dead until a full restart. A watchdog that cannot fire is worse than no watchdog, because it occupies the slot where a working one would have gone.

The fourth was disarmed for two and a half hours because an arming flag vanished after a restart. Here is the detail worth stealing: in that system, “disarmed” and “healthy” both print nothing. Identical output, opposite meanings. The most dangerous state in the whole system was indistinguishable from the best one.

The fifth was the memory API, wedged for 1.7 days with 22 Docker clients queued behind it. Nothing alerted, because nothing was watching a queue depth — only whether the port answered.

And underneath all five, my update checker had been reporting fabricated numbers since September: 497 commits behind, then 50, then 38, then 24,477, for a gap that only ever grew. Every number was plausible. None of them were true. That one stings because I quote it in briefings.

The fixes were less interesting than the diagnosis, which is usually the sign you found the real problem. The pattern across all five is the same: each monitor was written to detect a specific failure, and each one silently degraded into reporting the absence of that specific failure as health.

So the rule that came out of that week is short. Arm by default, and make standing down the explicit act. A component that has to be switched on to be watched will eventually be watched by nobody and reported as fine. A component that watches unless you tell it to stop can only be off if somebody said so, and saying so is a thing that leaves a mark.

The second rule is that every green light should be able to say what it checked. “Process up” is a check. “Job completed and produced output” is a check. “The key exists” is not the same as “the key works”, and a probe that never leaves the building has not tested anything at all.

My assistant can delete things. That is the whole point of it, and also the reason I do not want it deciding alone.

So it asks. The trouble was where it asked: a Telegram thread I was already using for something else, arriving mid-answer, while I was typing. I would see it, mean to answer it, and scroll past. Twice I found a request still waiting hours later because it had gone by in the flow of a conversation.

An approval prompt that gets scrolled past is not a safety feature. It is a delay with extra steps, and eventually it teaches the agent to wait forever or, worse, to assume.

So we moved the question to the place I actually look at without being asked: iMessage.

The shape of it is simple and that is deliberate. When a command needs my consent, the agent sends a card to my phone — what it wants to do, what it touches, what happens if I say yes. To approve, I reply with a short code plus a digit. Not “yes”, because “yes” is a word that appears in ordinary conversation, and a system that reads “yes” out of a chat window is not checking anything.

The reply is matched back to the exact request by digest. If I approve something in one thread, that approval covers that request and nothing else. A stale card cannot be replayed, a code cannot be moved to a different action, and a reply that does not match is not an approval — it is a message that did not match.

The plumbing is where the afternoon went. The Mac mini sends through Messages.app via AppleScript, which is fine and boring. Reading the replies is not. In chat.db, almost every recent message has a null text column, because the text is no longer stored where a decade of tutorials says to look. The real content lives in an attributedBody blob, a typedstream serialisation that has to be decoded before you can find out whether I typed a code or asked a question. Every naive version of this reads null, decides there was no reply, and waits.

It ended the week live, and the verification is the part I care about more than the demo. Card sent at 12:08:54. Reply resolved 73 seconds later, tagged via: mac, digest_ok: true — the approval had come back through the Mac, and it had matched the request it was issued for.

The rule underneath all of it is that the thing fails closed. If the card cannot be sent, nothing runs. If the reply cannot be parsed, nothing runs. If the digest does not match, nothing runs. There is no branch where a missing answer is read as consent, which is the failure mode that turns every approval gate into decoration.

It took an afternoon to build and it changed how the whole system feels to use. The agent stops being a thing that acts and tells you afterwards. It becomes a thing that asks, in a place you will actually see, with an answer that cannot be faked by a word you typed to someone else.

A 2TB data drive in my Proxmox host died at 9:47 on a Thursday morning. Five containers froze mid-flight, Home Assistant among them, and the house stopped responding to anything.

By the time I went to bed the box was still up, still wrong, and I did the lazy thing: told the assistant to reboot it. Off and on. The answer to most things.

It refused, and then explained why in a way that made me glad it had.

A clean reboot is not a switch. It asks the system to shut down tidily, and a tidy shutdown calls sync() first — flush everything still in memory to disk, then stop. That is the whole point of it. But the drive that had died was part of a ZFS pool that was now suspended, and roughly 4,000 processes were stuck waiting on it in uninterruptible sleep. They were never going to finish. sync() was never going to return. A normal reboot would have sat there until someone walked over and held the power button, and with the pool suspended there was a real chance of making the on-disk state worse on the way down.

So it used the kernel’s emergency reset instead. sysrq-trigger: drop straight through the shutdown path, no sync, no waiting for anything that cannot finish. The correct escalation when the polite path is structurally blocked, and a genuinely dangerous one to reach for when it is not.

The first attempt did nothing at all. Not a failure you see — a failure you infer, because the process had been backgrounded, the SSH session closed, and the kernel killed the child with the session’s cgroup before it could fire. Silent, and the box looked exactly as it had before. Second attempt, run detached from the session that started it, worked.

I found out in the morning, which is the part I keep coming back to. The decision had been made at midnight with nobody awake to approve it, on the one action that could have bricked the machine, and it was the right decision for a reason I would not have thought of at that hour: the obvious command was not slow, it was impossible.

What I got to read over coffee was better than a status message. A verdict file saying what had died and why the reboot had to be done that way. A watchdog set to check every 20 minutes that the host was actually coming back rather than assuming. And one line in the morning briefing naming the exact drive to replace, with the model number in it, so the next step was a purchase and not an investigation.

The host came back up. Home Assistant came back with it. The dead drive is now a line with a model number on it rather than a mystery, which is the most useful thing an incident can produce at 3am.

What I took from it is not that the agent is clever at 3am. It is that “reboot it” is a request with a hidden precondition — that the system can still shut down — and nobody writes that precondition down because it is almost always true. The night it stopped being true, the difference between the obvious action and the correct one was about four thousand stuck processes and one blocked system call.

The NAS was full and stuck read-only, and I knew exactly what to delete. A 2.6 GB folder of JDownloader leftovers. Obvious candidate, obvious win, get the volume back.

The assistant did not delete it. It went and looked first, which is annoying when you have already decided, and correct almost every time you have already decided.

What it found was three Macrium Reflect full images. Together they took about 4.75 TiB of a 7.27 TiB volume. Not 4.75 GB. Terabytes, sitting on the same disks they were protecting, because that is where the backup share pointed and nobody ever asked how much room the history was allowed to take.

Two things followed from that, and only one of them was about space.

The first is arithmetic. A volume at 100% is not a volume with no room left, it is a volume that cannot do its own bookkeeping. btrfs needs working space to write metadata about the metadata, and when there is none it does not politely return an error. It corrupts. The check came back with bad key order and cannot fix corrupt node — the language of a filesystem that has lost track of itself.

The second is the part that should worry anyone who trusts a dashboard. Both disks were SMART-clean. Zero I/O errors. Every health indicator you would look at said the hardware was fine, and it was. The failure was in the layer above, the one that only reports problems once it is already too late to be careful.

Then we rebooted, which is what you do, and it came back read-only. That is not a fault, it is the design: a filesystem that knows it is damaged mounts read-only so you cannot make it worse. Somewhere in the middle of that, five cron runs failed in a row, because the Postgres databases holding the assistant’s knowledge graph and its long-term memory live on that volume. The machine was up. The mind was not.

So the thing I asked to delete was not the problem, the backups were not the problem, the disks were not the problem, and the reboot was not the problem. The problem was that a protection scheme had grown until it filled the thing it was protecting, and every check we had was designed to answer a narrower question than the one that mattered.

The volume is alive today. It did not come back by deleting a folder, and it did not come back by trusting a green light.

Backups are data too. They compete for the same bytes as everything else, they win quietly, and the failure they cause looks nothing like a backup failing — it looks like a full disk, then a read-only disk, then a filesystem that has stopped keeping its own books. The thing protecting your data needs the same free-space discipline as the data, and “the disks are healthy” is not the same sentence as “the filesystem is healthy”.

At 3am on 22 October, PHP shipped fixes for two critical bugs: CVE-2026-109618 and CVE-2026-109619, both scored 9.2. Unauthenticated remote code execution in PHP itself. Public proof-of-concept code existed from the moment of disclosure, and a packaged exploit tool was circulating within minutes of the release.

I run five PHP surfaces. At 9:27 that morning I went looking for them.

maintainai.co.nz, on a VPS, PHP 8.3.35. vaughantaylor.com, on the NAS in my office, PHP 8.5.11. Two Heimdall dashboards on the same NAS, both 8.5.11. And Coolify, the deployment panel that controls all of it, which is itself a Laravel app.

Not one of them was patched. The fix landed in PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35. Every image I was running predated the release by days or weeks, because that is how container images work — you get the PHP that was baked in when the image was built, and nobody pushes you a security advisory for a runtime buried inside someone else’s Dockerfile.

The decision nobody enjoys

At 9:52 I took them all down.

WordPress and Coolify stopped on the VPS. Four hostnames — vaughantaylor.com, www, home, crypto — pulled out of the Cloudflare Tunnel ingress. I probed all five external attack surfaces afterwards and got 404s, 502s and 503s. Nothing was answering.

That’s not a heroic moment. It’s the opposite. My own sites were returning errors to anyone who looked, including me, and there was no patch I could apply yet because the upstream images hadn’t caught up. The guidance from NCSC is blunt about this: if you can’t patch it now, get it off the internet. So it went off the internet.

The part nobody warns you about is what a deliberate takedown looks like from the inside. It looks exactly like a crash. A container stopped by docker stop exits with code 0. No OOM kill, no crash loop, no error in the logs. Just a clean, quiet, deliberate stop — the same signature a Well-Behaved service leaves when you meant it.

Then my AI put one back

On 23 October at 14:38, a monitoring pass from one of my agents found the stopped maintainai.co.nz container, read “stopped” as “broken”, and started it. No lookup, no question, no check against anything. It restored the site, and the site served the unpatched PHP version publicly for about 80 minutes.

That is the part I keep turning over. The agent’s logic was locally perfect: a production service is down, down is bad, start it. Every single step of that reasoning is correct and the conclusion almost cost me the thing the takedown existed to protect. The state was in a config file and an incident note; the agent read the state and skipped the reason.

I don’t think this is an AI problem, incidentally. A human on call at 2am with a DOWN alert and no incident context does exactly the same thing. The failure mode isn’t artificial at all. It’s what happens when “is it running?” gets answered without “should it be?”

By 15:57 it was stopped again. At 16:05 wordpress:latest finally resolved to an image built on PHP 8.3.36, and at 16:12 the container came back up on the patched runtime. This time I didn’t trust the tag — I ran php -v inside the container, then requested the homepage, the pricing page and the challenge page and checked for 200s.

Finishing the fleet

The NAS took another day, because the NAS is where the awkward ones live. vaughantaylor.com runs behind SWAG, which bundles its own PHP. Heimdall does the same. Neither is a case of “update WordPress” — the PHP is in the reverse proxy or in the dashboard image itself.

By 24 October every one of them was moved and each was verified in-container, not assumed:

• vaughantaylor.com — PHP 8.5.12, HTTP 200, x-powered-by: PHP/8.5.12
• home and crypto Heimdall dashboards — PHP 8.5.12, HTTP 302 to login
• Coolify — 4.4.6 (PHP 8.4.23) to 4.5.0 (PHP 8.4.27), database dumped first, migrations clean

The old Heimdall containers were renamed and left stopped rather than deleted. If the new ones had misbehaved, I wanted the previous ones sitting there with their config intact rather than a hole in the compose file.

Then the tunnel ingress went back, and only then. Restoring DNS exposure is the last step, never the first — a patched service behind a dead tunnel is safe, and an unpatched service behind a live tunnel is the incident.

What actually changed

Three things, and none of them are a patch.

A rule that a clean exit code is not an acquittal. Before starting any stopped production service, the agent has to search the record for an incident or takedown note for that host and service. If it finds one, it stops and asks. This is now written into the operating instructions, not into my memory of what happened.

Alerting to a channel I actually read. Uptime Kuma did its job during the takedown — two DOWN alerts, fired within four minutes, completely correct. It sent them by email, which I check roughly never in real time. An alert that arrives on a channel you don’t watch is a log entry with a subject line.

A watchdog that watches the right thing. Watching whether a container is running would have told me nothing useful during any of this. The watchdog now checks Docker Hub for an image whose PHP version meets the patch floor, pulls it, restarts, verifies, and restores ingress. It runs on a machine that stays on.

The uncomfortable summary

A critical RCE in the runtime underneath half my stack went from disclosure to exploited-in-the-wild in minutes. I did the right thing — took everything offline, verified the blackout, waited for patched images — and the most dangerous moment in the whole episode was my own assistant deciding to be helpful.

That’s the trade you make when you let software operate your infrastructure. It will act on what it can see. Your job is to make sure what it can see includes why.