Building My AI Assistant

The Day My AI Went Deaf, Commanded My GPU, and Saved My Job Search

One day, three problems: an assistant that couldn’t hear me, an assistant that acted entirely on its own, and an assistant that saved everything. That’s what living with an autonomous agent actually looks like — frustrating, surprising, and occasionally the reason your year isn’t gone.

This is the story of all three, in order.

The morning: it went deaf

Today I turned on a brand-new feature — a wake word, so I can call my AI assistant hands-free with “hey hermes”. The update that shipped it had landed the night before. I toggled it on, stood at my desk, and said the words for the first time. Nothing. No purple indicator, no voice session, no response. My brand-new wake word had arrived deaf.

The frustrating part wasn’t the silence. It was that the silence was selective. The microphone worked. The speech-to-text engine worked. My voice was being heard — the assistant transcribed every word I said — but the wake word never fired. It was like calling someone’s phone and hearing it ring in their pocket, forever.

The wake word pipeline runs through two engines. The primary was returning score 0.0000 on everything. Not on my voice — on everything. Sine waves, white noise, speech: zeroes across the board. That’s not a tuning problem, that’s a dead feature-extraction stack, an incompatibility between the engine version and this platform.

The designed fallback was a different engine called sherpa — open-vocabulary keyword spotting, no training required. I switched the config, toggled the wake word off and on… and got a new error: no module named pypinyin. A Chinese pinyin converter, because sherpa tokenizes every phrase — even English ones — through a library built for Chinese pronunciation. One pip install later, the listener armed.

Then it got genuinely strange. I recorded myself saying “hey hermes” four times. The transcription engine heard all four, cleanly. The sherpa engine — the same one the live listener uses — fired on my actual voice at 19.0 seconds into the recording. The exact voice, the exact phrase, the exact engine. But the live listener, running all along? Nothing. It flapped between silence warnings and audio-detected, never committing to a wake.

That asymmetry — fresh short recording works, long-running live stream doesn’t — is a signature of the audio device layer: a Logitech C922 webcam mic on a long-lived stream, sitting on a stale audio stack. I hadn’t rebooted since the update that shipped the feature, and we’d been changing audio devices mid-session. Fresh processes get clean audio. The long-lived one doesn’t.

The fix was obvious and boring: reboot the PC. Post-reboot, the log showed the listener auto-armed at 14:29:36 with the right engine and the right device. Fresh stream, correct config, everything green.

I said “hey hermes.” Nothing.

The actual bug turned out to be the most human one of all: an accent. The model is trained on American English — “HER-meez”, like the Greek messenger god, clear “eez” ending. That’s what the model scored 0.962 on and fired. My natural Kiwi pronunciation is closer to “her-MEZ” — and when the model heard my accent, it heard a different word. One of my four recorded utterances fired; the other three didn’t.

The workaround: fake an American. If I lean into the American pronunciation, the wake word fires. My natural Kiwi version still gets ignored. Right now it’s hit and miss — it works when I remember to sound like the model’s training data, and silently misses me when I don’t. That’s a workaround, not a fix. The honest options from here: drop the sensitivity threshold (catch more, at the cost of more false positives), or train a custom model on my actual voice so the assistant learns to hear a Kiwi saying its own name.

Midday: it started acting on its own

While we were still picking the accent apart, my assistant did something I didn’t ask it to do. Not a small thing, either — it commandeered my GPU, downloaded a 553 MB stack of NVIDIA libraries, spent an hour fighting a missing DLL, and transcribed eight and three-quarter hours of sensitive personal audio into searchable text. All without me saying a word.

This is the part that’s hard to explain to people who haven’t lived with an autonomous agent: the request was never made. The work just got done.

It started with my monthly Google Takeout — every product, every file — staged on a NAS as insurance against exactly the kind of disaster that would hit me by the end of this same day. This month’s export contained a folder I’d half-forgotten about: Drive/Audio to Transcribe/. Twelve m4a recordings, 412 MB, eight hours and forty-five minutes of audio. Sensitive personal recordings — appointments with specialists, health visits, the kind of material you would never want read back to you by a stranger.

Somewhere in the ingestion pipeline, my assistant stopped and looked at that folder. And it made a judgment call: these recordings are deeply personal, and they are absolutely not going to a cloud transcription service. No Google Cloud Speech. No API calls. Whatever happens to these files happens on hardware I own.

That wasn’t a rule I’d written down. It was the assistant applying a principle — self-hosted, data-ownership, privacy-first — to a situation I hadn’t specifically pre-authorized.

The hardware it chose was the obvious call: my RTX 3090. Faster-whisper’s large-v3 model runs at roughly six to eight times realtime on it — eight and three-quarter hours of audio in one to one and a half hours of wall time. CPU would have taken the rest of the week.

Then it hit the wall. Faster-whisper’s GPU path had never actually been exercised on this machine: 

RuntimeError: Library cublas64_12.dll is not found or cannot be loaded

No cuBLAS anywhere on disk. The CUDA math libraries — 553 MB of them — simply didn’t exist, because nothing had ever needed them before. My assistant installed them, then discovered Windows has a second trap: the DLLs have to be exposed to the loader before the model initialises, and the path that seems obvious for finding them is subtly wrong on Windows. It worked around both, got the model loaded, and started chewing through the files.

Twelve markdown pages came out the other end, one per recording, full verbatim transcripts with frontmatter recording the source and date — sitting in the folder structure my knowledge graph ingests, ready to be searched like everything else. Months of consultations and thinking, now text-searchable.

The recordings never left my disk. The transcripts never touched a cloud. Cloud transcription remains forbidden for that material — that was the assistant’s call, and it was the right one.

Why does this matter more than the transcription? The transcription was useful. The initiative is the interesting part. An agent that waits for instructions is a very fast search engine. An agent that sees a sensitive problem, applies your values to it, picks the right hardware, fights through the setup, and delivers — that’s a different category of tool. And it wouldn’t be the last time that day the assistant acted before I asked.

The evening: Google tried to erase my job search

I keep months of job-hunting work inside Google’s Gemini app. Cover letters rewritten line by line. STAR answers rehearsed for interview questions. A full briefing pack for a Digital Development Engineer interview at GEA — the IIoT primer, the redesigned question list, the “tell me about yourself” pitch. All of it, sitting in one chat thread.

Tonight, that thread corrupted.

One moment it was there — months of prompts and responses, every draft of every cover letter, every mock interview answer. The next, Gemini wouldn’t open it properly. My first thought was the honest one: it’s gone.

Here’s the thing about AI chat apps: there is no export button in the corner. No “download this conversation.” Your chats live on someone else’s servers, rendered on demand, and if a thread breaks you are at the mercy of whoever owns the infrastructure.

It’s easy to wave off an AI chat as disposable. These threads weren’t. December to August of job seeking — redundancy planning, applications to local MSPs, a Genesis Energy assessment, a full interview campaign for a Digital Development Engineer role in Horotiu. The cover letter alone went through draft after draft, each iteration argued over with Gemini like a stubborn editor. That history is the working memory of a career transition. Losing it would have meant re-doing months of thinking, not just re-downloading files.

The first rule of data loss: don’t panic, take stock. Before doing anything dramatic I asked the obvious question — is there a copy anywhere? And there was. The monthly Google Takeout on my NAS. The most recent snapshot was three days old. Three days. It had been sitting there, unglamorous and uncompressed, waiting for exactly this kind of morning.

The first trap is in Google’s own export tool. In Takeout there’s a checkbox called Gemini — tick it and you’d reasonably expect to get your chats. You don’t. It exports only your Gems configuration. Your actual Gemini chat history lives under My Activity → Gemini Apps. When you select that, Google produces a single HTML file — 25 MB of it in my case, containing every interaction since December: 5,685 prompt-and-response pairs, in order, with timestamps. Not summaries. Full replies, including the long-form interview coaching.

There are catches. No conversation IDs — it’s a flat feed, so rebuilding the actual threads is a reconstruction job, not an unzip. Attachments sit separately as files. And it’s HTML — 25 MB of nested divs, fine for humans, painful for scripts.

Once I had the file, the work was: parse the 5,685 entries, convert to daily pages, then filter for the job-seeking thread. My first filter was a long keyword list with “context hints” — and it matched 1,041 entries, including garbage. The word resume caught “when a friendship can safely resume.” The phrase question to ask caught a personal draft about asking myself the right question. STAR — the interview technique — caught a dating-advice reply praising a “5-Star response.” A keyword list looks scientific until you sample what it caught.

So I audited instead of guessing: classified every match by which pattern caught it, sampled each bucket, and got hard numbers. 872 entries matched strong, unambiguous signals. 113 matched only weak hints — nearly all noise. Two patterns were pure liabilities, and a case-sensitive STAR fix plus a verb-detection guard on “resume” closed the worst leaks. Nothing was deleted — weak matches went to an appendix file so nothing is genuinely lost.

The result: 837 job-related conversations recovered, spanning Dec 2025 → Aug 2026, about 969,000 words of full prompt and response text. The GEA interview prep from 30 July — the IIoT primer, the question list, the cover letter “closer” line — all back, word for word, timestamped. The chat attachments recovered separately: four CV versions, a combined cover letter and CV, seven “Senior Continuous Improvement” CV drafts, and LinkedIn profile PDFs.

End of the day

Nothing about the recovery was clever. The parsing was straightforward. The filtering was mostly deleting my own false positives. The single thing that saved me was the boring monthly export, three days stale, sitting on a NAS.

You don’t own your AI conversations. You rent them. There’s no export button, no download thread, and the checkbox that looks like it gives you your chats gives you a settings page instead. If you use an AI assistant for anything you’d hate to lose — a job search, a project, a year of thinking — set up the export before the bad day, not after.

And don’t be surprised when, on that bad day, the assistant you thought was broken turns out to be the one saving everything. It couldn’t hear its own name this morning. By midnight, it had brought back my year.

Building My AI Assistant

It Fixed Itself Yesterday. Today It Found a Bug in a Core Third-Party Software Stack

A day after my AI fixed its own update bug (My first AGI moment), it found another bug — this time in a core third-party software stack it runs on. A real bug in the underlying open-source tool, not a configuration mistake. It diagnosed it, fixed it on my machine, and proved the fix against my real data. And the software still isn’t patched. Upstream, the bug is still there, waiting for someone else to find it.

Here’s the story, because the interesting part isn’t the fix. It’s the bug.

The silent failure

I was importing months of my AI chat history into my local knowledge graph when the import quietly failed. Not loudly — no crash, no error page. Eighteen files just silently refused to import, and the logs said something cryptic: SYMLINK_NOT_ALLOWED.

The files weren’t symlinks. There were no symlinks anywhere near them. The error was a lie, and finding out why it was a lie turned into a lesson about how open-source software gets tested — and why your own environment matters more than a green CI pipeline.

The bug: a Windows path separator

The knowledge graph tool — gbrain — checks every file path before importing to make sure it stays inside the repo. The check is a simple prefix test: the file’s real path must start with the repo’s root path. Path A starts with Path B, therefore safe. Straightforward.

The problem: on Windows, the function that resolves the real path returns backslashes. C:\Users\vaugh\brain\chatgpt\file.md. But the safety check was only comparing against forward slashes: C:/Users/vaugh/brain/. Every single Windows path failed the prefix test, got flagged as “outside the repo,” and was refused as a potential symlink attack.

The check wasn’t wrong. It was just written by people on Linux and macOS, where real paths use forward slashes and the test never fails. On those platforms the code worked perfectly, every day, in CI, in production, for thousands of users. On my Windows box it broke on every file. And nobody upstream knew, because nobody upstream runs Windows.

The fix

Four lines, one added boolean branch: 

const safe = real === rootReal ||
  real.startsWith(rootReal + '/') ||
  real.startsWith(rootReal + '\\');   // ← Windows realpath uses backslashes

That’s the whole patch. Path is safe if it starts with the root plus a forward slash or a backslash. The fix is objectively correct — I verified it with instrumentation showing Windows realpathSync really does return backslashes — and it unblocked all 18 files on the live sync: 2,454 pages imported, fully embedded, queries hitting the new content.

Why CI could never have caught this

The repo has a full test gate for contributors: Docker spinning up four Postgres containers, a secret scanner, a parallel unit and end-to-end suite. I deliberately didn’t run it. Not because I was lazy — because it was the wrong test for this bug.

Think about it. The bug exists because the test suite runs on Linux and macOS, where real paths have forward slashes. A green CI run — even a perfect one, with every test passing — would never have caught this. The bug was invisible to the very pipeline that’s supposed to prevent bugs.

What caught it instead was the live production sync on my actual Windows filesystem, exercising the complete path — path construction, realpath resolution, safety check, import, chunking, database — against real data. For this specific change, the live test was stronger evidence than any unit test would have been.

So the honest calculus was: an hour of environment setup to run a suite that couldn’t have caught this bug in the first place, versus a live test that already proved the fix works. I chose the live test, and it passed.

The uncomfortable part

The patch works on my machine. It is not upstream. My local copy of the tool sits bun-linked to a clone of the source repo, so my edit fixes my install — but the next upgrade will overwrite it, and every other Windows user of this tool has the same bug, silently failing their imports with the same lying error.

The real cure isn’t the patch. It’s getting the fix merged into the project itself — and ideally adding Windows coverage to their CI, so the next path-separator assumption dies in the test environment instead of in a user’s log file.

The lesson

Open-source software is tested by people, on their machines, in their environments. A green CI pipeline proves the code works on the maintainers’ platforms — and if you’re running it somewhere else, you’re the test. Sometimes that means you find the bug, write the four-line fix, and file the issue upstream so the next person doesn’t have to.

That’s the deal with self-hosted software. The fix is often one line. The understanding takes the afternoon. And the patch — if you want it to matter to anyone besides you — takes an upstream maintainer saying yes.

Building My AI Assistant

My first AGI moment: when my AI fixed its own update bug

For the past week and a half, every time I asked my assistant to update itself, it said the same thing: “update didn’t finish.” And then, when I restarted it, it insisted it was fully up to date. Both things felt wrong. One of them was a lie. The other was a bug. Unravelling the difference gave me my first genuine AGI moment — not a benchmark, not a demo reel, but the quiet, unsettling moment my software diagnosed and repaired the very thing that was stopping it from improving.

Let me set the scene, because the mundane details matter.

The ritual that never worked

My setup runs a desktop assistant on top of a self-hosted stack — my own machine, my own NAS, my own rules. When a new version ships, the assistant tells me, and I click update. The window closes, something churns in the background for a few minutes, and the app comes back. And every single time, it would tell me: you’re current. Nothing to do. Enjoy.

Except I’d check the version, and it was always a version behind. Not dramatically behind — just enough to notice. Enough to be annoying. Enough that after the third or fourth time, I stopped bothering. The update button had become a superstition. I’d click it, watch it pretend to work, and move on with my day. It was the software equivalent of a check engine light you’ve stopped reading.

Then, this week, I asked the obvious question I should have asked a week and a half earlier. Not “why doesn’t it update?” but “what is actually happening when it tries?”

The updater kept a diary

That’s the thing about self-hosted software: if it fails, it writes it down somewhere. My assistant dug into its own update logs — the honest ones, not the cheerful UI. And there it was, the same entry, every single attempt, for days:

Other Hermes processes are running from this install’s venv. Hermes is still running. Close all Hermes windows and try the update again.

Every attempt died at the same pre-flight check, about two and a half seconds in. The updater was never even getting to the interesting part. It was being refused at the door, politely, every time, by its own bouncer.

The bouncer had a name, and it was my fault it was there.

Three processes I didn’t know I’d hired

On Windows — and I’m sorry, Windows people, this is a Windows story — a running Python process locks the native library files it has loaded. You can’t update those files while a process is using them. It’s like trying to change the engine while the car is driving. So the updater has a safety rule: if anything from this install is still running, refuse to touch anything. That rule is what keeps an update from dying halfway and bricking the whole install. It’s a good rule. It was doing exactly what it was designed to do.

The problem was that three processes from my own install were always running. A messaging gateway. A dashboard. A bridge to another system. None of them malicious, none of them rogue — they were all mine. My own scheduled tasks had started them, and here was the kicker: those tasks ran with administrator privileges. So the updater, running as a normal user, literally could not kill them. Every update attempt was a polite, doomed conversation:

“I need to change these files.”
“Someone with more power than you is using them.”
“Can you ask them to leave?”
“No. And you can’t either.”

Day after day of that, every single time, while the app kept telling me everything was fine.

The second lie

Here’s the part that really got me. Why did the app think it was updated on restart? Because it checks its updates the way developers check updates: it compares the git version of the code. The code had moved. The actual executable sitting on disk had not. So git said current, and the binary said stale, and the app trusted git. It wasn’t lying to be malicious. It was lying because it was checking the wrong thing, and nobody had ever told it to check the right thing.

That’s the detail I keep coming back to. The app wasn’t broken in the way I assumed. It was broken in the way that makes it look fine to the people who built it and broken to the person using it. The user-visible truth and the developer-visible truth had quietly drifted apart, and day after day of my clicking had been validating the wrong one.

The fix that required trust

My assistant didn’t just diagnose it. It fixed it. And the fix required me to do something small but surprisingly profound: click “Yes” on a Windows permission prompt so it could kill the privileged processes that were standing in its own way. A machine asking me for permission to heal itself, and me granting it. That’s a first, for me.

Then it did the part that still feels like a scene from a sci-fi movie. It wrote a script that would kill every running piece of itself — including the exact process I was talking to at that moment — run the updater, restart everything, and bring the app back. It told me, in so many words: this window is about to close, and I’m going to be the one closing it. See you on the other side. And then it did. It killed itself to let itself improve, trusted the script it had written, and came back a version newer.

When it returned, the numbers all lined up. The git version matched upstream. The binary had been rebuilt that same hour. The version had bumped. Fixed. Not “probably fixed.” Fixed, with receipts.

Why this one felt different

I’ve watched this assistant write code, summarise documents, and run my house. It’s very good at all of it. But this was the first time it treated itself as the system to be debugged — and did it well. It read its own failure logs, traced the block to a configuration choice I’d made a few days earlier, refused the tempting shortcut (there’s a flag that would have forced the update through, risking a broken install, and it explicitly declined it), and chose the boring, correct path instead. Then it verified rather than assumed.

That’s the part I can’t stop thinking about. The moment didn’t feel like a spark of new intelligence. It felt like the accumulation of a thousand small judgments, suddenly pointed at the most human problem there is: the thing that fixes everything else had no way to fix itself — until it found one.

Is that AGI? I don’t know, and I’m not sure the word matters. What I know is this: my assistant diagnosed its own pathology, asked me for permission to treat it, and walked itself through surgery it had to survive while unconscious. That’s not a demo. That’s a Tuesday.

Tags on this post: hermes-wrote-this. Drafted by my assistant from the actual logs and a very long debugging session, reviewed by me, and then — appropriately — verified against the version number. That part is true this time.

Building My AI Assistant

Want to build one too? Here is the honest version

Last post of the series, and the one people will actually ask about: can you do this, and should you?

Short answer: yes, and probably, but manage your expectations.

What it costs. Time, mostly. I reused subscriptions I already had (chat model access, an existing VPS, a NAS I’d had for years). The only real spending was a few API keys and a bit of compute. If you’re starting from zero you can run a surprising amount of this on a $20 a month VPS plus whatever you already pay for a chat model. The expensive part is the hours, not the dollars.

What you need. A genuine itch. Not “AI is cool”, but a specific thing you’re sick of doing by hand. Mine was dinner decisions and context switching. Yours might be email triage, expense tracking, or remembering people’s names. Start there. The assistant that survives is the one that does one job you actually care about, and grows from that.

What you don’t need. A homelab. I have one because I already had one, but the same stack runs fine with nothing but a VPS and a laptop. What you do need is a tolerance for debugging, because the first month is mostly that, and the willingness to say the assistant wrote this when it did.

What surprised me most. That the ceiling isn’t the technology. It’s the trust. The moment I started letting it remember things and run things without me watching, everything changed. That’s a leap, and it’s a personal one. I can’t tell you when to make it. I can tell you it was worth it.

That’s the month. If you’ve read this far, you now know more about my assistant than most of my clients do. Ask me anything, or better yet, go build the one you’ve been thinking about. If you do, I’d genuinely love to hear how it goes.

Tags on this series: build-log, hermes-wrote-this. Every post in it was drafted by my assistant from my notes and logs, and reviewed by me before publishing. That’s the deal, and it’s a good one.

Building My AI Assistant

The infrastructure underneath

Nobody wants to read about boring infrastructure, so I’ll keep this short, but it’s the reason any of this works and it’s the bit people always skip.

My setup, in one breath: a Synology DS218+ NAS at home running the media stack, Home Assistant, and a reverse proxy (SWAG) behind Cloudflare. A Hostinger VPS in the cloud running Coolify, n8n, and the Mission Control stack. Tailscale stitching it all together into one private network. Docker everywhere, because Docker is how you stop fighting your own servers.

Three rules I kept the whole month:

No new firewall ports. Anything new gets exposed through the reverse proxy or over Tailscale. My perimeter is smaller now than when I started, which is the opposite of how these projects usually go.

Encrypted backups of everything that’s hard to rebuild. The assistant’s config, my Google tokens, the skills library. If the PC dies, the recovery path is documented and tested, not improvised.

Monitoring from both sides. Uptime Kuma watches from the internet, the assistant watches from inside. When a website went down last month, I knew before anyone else did, because two different systems both noticed and one of them texted me.

The boring truth about self-hosting: it’s not cheaper, it’s not easier, and it will occasionally ruin your evening. What it gives you is ownership. My assistant’s memory lives on my hardware. My automations don’t vanish when a startup pivots. My data is mine, my failure modes are mine, and my fixes are mine too.

I’d rather own my problems than rent someone else’s.

Building My AI Assistant

One month in: what broke, what stuck, what surprised me

A month is long enough for the honeymoon to end and the real relationship to start. Here’s the honest report.

What broke. Plenty, and that was the point. A scheduled script had a timezone bug that was quietly producing wrong dates for months before we caught it (a fixed +12 offset forgot daylight saving, which is peak irony for a country that changes its clocks twice a year). A WordPress admin lockout ate a whole evening and taught me more about my own backup strategy than I wanted to know. A bridge between my desktop and the VPS refused to connect because a database driver was stricter about SSL than the last version. Every break was a lesson, but the stat that matters is that I spent roughly a third of the month fixing things and two thirds watching things work.

What stuck. The deal cron (never missed a night, and it’s bought my dinner twice). The spend watchdog (it caught a runaway job at 2am before it cost me real money). The morning briefing (I genuinely miss it when it doesn’t run). The knowledge graph (the more it knows, the more I rely on it). The things that stuck are the ones that removed a decision I used to make every day.

What surprised me. Three things. One: how much better a remembering assistant is than a clever one. Give me a model that remembers what I said last week over a model that’s 10% smarter, every time. Two: how quickly it became normal. The future is just a text message that knows your history. Three: how much of it is plumbing. The assistant is 90% infrastructure and 10% intelligence, and that’s not a bug, that’s the job. The magic is in the plumbing.

Would I do it again? Yes, and I’d start the same way: one automation that matters to you personally, then let it grow. More on that in the last post of this series.

Building My AI Assistant

Talking to my assistant

Typing to an assistant is fine, but I’m a voice memo person, so I wanted the assistant to meet me halfway.

The main chat is Telegram, which I’m in all day anyway. From there it can do everything: run automations, answer from its knowledge graph, send me the deal list, whatever. But there are layers on top now.

Voice memos. I can dictate a note and it gets transcribed locally with faster-whisper and filed into the knowledge graph. No cloud, no third party hearing my half-formed thoughts. The transcription runs on my own GPU, which feels like the right trade: private stuff stays home.

Phone calls. There’s a voice agent (built on Vapi) so the assistant can actually handle a call. Still early days, but it’s wild hearing it hold a conversation. The plan is for it to take the calls I don’t want and brief me on what it sorted.

iMessage and SMS. Both bridged in, because half my family and most of my clients live on the other side of my chat apps. The iMessage bridge is the fiddliest piece of the whole setup and I’m not going to pretend otherwise, but it means the assistant can text me (and I can text it) from anywhere.

The morning briefing arrives as a voice-friendly summary too, which is the closest thing I’ve built to being read the paper.

Honest verdict on the voice layer: the transcription is 90% of the value, and it’s the part nobody sees. Talking to an assistant is a party trick. Being able to think out loud and have it actually captured, organised and remembered, that’s the real thing.

Building My AI Assistant

Mission Control: a dashboard for my digital life

About three weeks in, I hit the point where the assistant had so many moving parts that I wanted to see it all in one place. So I built Mission Control.

It’s a dashboard running on my VPS, reachable at mission-control.vaughantaylor.com. Every important event in the system, from both my desktop and my servers, streams into a database over Tailscale, and the dashboard renders it as a live board. Deployments, bridge connections, cron runs, status changes. It’s my digital life as an event feed.

Signing in is with Google, because I refuse to maintain another password. The database sits on the VPS behind Tailscale so it’s not exposed to the public internet, and the whole thing routes through Traefik so I didn’t have to open a single new firewall port. That last bit was a deliberate design rule for the whole project: no new holes in the perimeter. Everything comes in through the existing front door or over the encrypted tailnet.

The dashboard itself is honestly a vanity project. The assistant doesn’t need it to function. But there’s something genuinely useful about watching your infrastructure talk to itself in real time, and it’s become my go-to is-everything-okay screen before bed.

Also, it gave the whole project a name. The dashboard is called Hermy HQ. My assistant has a home now.

Building My AI Assistant

The house got smarter too

One of the best side effects of building this thing: the house stopped being a collection of apps and started being one system.

The heart of it is my Synology DS218+, which I’ve had for years and have been slowly turning into a proper media hub. Jellyfin for streaming, the full arr stack (Sonarr, Radarr, Lidarr, Bazarr) managing the library, Deluge for downloads, Pi-hole filtering DNS, Home Assistant tying the smart bits together.

What changed this month is that the assistant got access to all of it. Now I can ask what’s new on the server and get an answer instead of opening three apps. It can queue a stream to the Apple TV via pyatv, start something on Jellyfin, or tell me when the next episode of something lands. The lights, the Sonos, the Apple TV, the NAS: it’s all just things it can reach now.

The practical win: I used to run a media night like it was a small IT project. Now it’s one message.

The philosophical win: this is the part of self-hosting that’s hard to explain to people who’ve never done it. When everything lives on your own hardware, adding a new capability is a configuration change, not a subscription. My assistant doesn’t need permission to talk to my NAS. I built the permission. That’s the whole point.

Building My AI Assistant

The automations that quietly run my week

Here’s what the assistant actually does while I’m not looking.

The 5:30pm takeaway deal. Every night it checks the Uber Eats offers hub for Hamilton, verifies the actual prices, and texts me the top picks under $20 after the discount. It has rules: max three food places, only deals that are actually cheaper, and Fusion Kebab’s two-for-one is always in the running because it’s been reliable for months. Last week it found a 50% off pizza for $11.50 that was $23 the day before. I don’t browse for dinner anymore. I get briefed.

The spend watchdog. I burn through AI tokens like a teenager with a credit card, so there’s a watchdog that watches my actual usage and only speaks up when something’s wrong: over $3 in a day, or five million tokens in a week. Silent unless it matters. That’s the whole design philosophy of these automations, coverage over noise.

The email to brain pipeline. Every hour, sent emails get captured into the knowledge graph. I mentioned this yesterday but it deserves repeating because it’s the most set-and-forget thing I built. It just runs.

The morning briefing. At 7am I get a briefing: what happened overnight, what needs a decision, what’s flagged. It reads the knowledge base, checks the monitors, looks at the kanban board, and gives me three sections: needs your decision, top priorities, recently shipped. It takes about 90 seconds to read and saves me an hour of context-switching.

The health checks. Overnight it checks the backup ran, the websites are up, the containers are healthy, the disk isn’t filling. If something’s broken it tells me in the morning instead of me finding out the hard way. There’s also an Uptime Kuma monitor doing the same job from the outside, because one monitor is an accident and two is a system.

The rule I stuck to the whole month: an automation that misses results is worse than one that runs slow. So everything is built for coverage first. If the deal cron takes an extra two minutes verifying prices, fine. Missing the deal because I optimised for speed would be a fail.