Typing to an assistant is fine, but I’m a voice memo person, so I wanted the assistant to meet me halfway.

The main chat is Telegram, which I’m in all day anyway. From there it can do everything: run automations, answer from its knowledge graph, send me the deal list, whatever. But there are layers on top now.

Voice memos. I can dictate a note and it gets transcribed locally with faster-whisper and filed into the knowledge graph. No cloud, no third party hearing my half-formed thoughts. The transcription runs on my own GPU, which feels like the right trade: private stuff stays home.

Phone calls. There’s a voice agent (built on Vapi) so the assistant can actually handle a call. Still early days, but it’s wild hearing it hold a conversation. The plan is for it to take the calls I don’t want and brief me on what it sorted.

iMessage and SMS. Both bridged in, because half my family and most of my clients live on the other side of my chat apps. The iMessage bridge is the fiddliest piece of the whole setup and I’m not going to pretend otherwise, but it means the assistant can text me (and I can text it) from anywhere.

The morning briefing arrives as a voice-friendly summary too, which is the closest thing I’ve built to being read the paper.

Honest verdict on the voice layer: the transcription is 90% of the value, and it’s the part nobody sees. Talking to an assistant is a party trick. Being able to think out loud and have it actually captured, organised and remembered, that’s the real thing.

About three weeks in, I hit the point where the assistant had so many moving parts that I wanted to see it all in one place. So I built Mission Control.

It’s a dashboard running on my VPS, reachable at mission-control.vaughantaylor.com. Every important event in the system, from both my desktop and my servers, streams into a database over Tailscale, and the dashboard renders it as a live board. Deployments, bridge connections, cron runs, status changes. It’s my digital life as an event feed.

Signing in is with Google, because I refuse to maintain another password. The database sits on the VPS behind Tailscale so it’s not exposed to the public internet, and the whole thing routes through Traefik so I didn’t have to open a single new firewall port. That last bit was a deliberate design rule for the whole project: no new holes in the perimeter. Everything comes in through the existing front door or over the encrypted tailnet.

The dashboard itself is honestly a vanity project. The assistant doesn’t need it to function. But there’s something genuinely useful about watching your infrastructure talk to itself in real time, and it’s become my go-to is-everything-okay screen before bed.

Also, it gave the whole project a name. The dashboard is called Hermy HQ. My assistant has a home now.

One of the best side effects of building this thing: the house stopped being a collection of apps and started being one system.

The heart of it is my Synology DS218+, which I’ve had for years and have been slowly turning into a proper media hub. Jellyfin for streaming, the full arr stack (Sonarr, Radarr, Lidarr, Bazarr) managing the library, Deluge for downloads, Pi-hole filtering DNS, Home Assistant tying the smart bits together.

What changed this month is that the assistant got access to all of it. Now I can ask what’s new on the server and get an answer instead of opening three apps. It can queue a stream to the Apple TV via pyatv, start something on Jellyfin, or tell me when the next episode of something lands. The lights, the Sonos, the Apple TV, the NAS: it’s all just things it can reach now.

The practical win: I used to run a media night like it was a small IT project. Now it’s one message.

The philosophical win: this is the part of self-hosting that’s hard to explain to people who’ve never done it. When everything lives on your own hardware, adding a new capability is a configuration change, not a subscription. My assistant doesn’t need permission to talk to my NAS. I built the permission. That’s the whole point.

Here’s what the assistant actually does while I’m not looking.

The 5:30pm takeaway deal. Every night it checks the Uber Eats offers hub for Hamilton, verifies the actual prices, and texts me the top picks under $20 after the discount. It has rules: max three food places, only deals that are actually cheaper, and Fusion Kebab’s two-for-one is always in the running because it’s been reliable for months. Last week it found a 50% off pizza for $11.50 that was $23 the day before. I don’t browse for dinner anymore. I get briefed.

The spend watchdog. I burn through AI tokens like a teenager with a credit card, so there’s a watchdog that watches my actual usage and only speaks up when something’s wrong: over $3 in a day, or five million tokens in a week. Silent unless it matters. That’s the whole design philosophy of these automations, coverage over noise.

The email to brain pipeline. Every hour, sent emails get captured into the knowledge graph. I mentioned this yesterday but it deserves repeating because it’s the most set-and-forget thing I built. It just runs.

The morning briefing. At 7am I get a briefing: what happened overnight, what needs a decision, what’s flagged. It reads the knowledge base, checks the monitors, looks at the kanban board, and gives me three sections: needs your decision, top priorities, recently shipped. It takes about 90 seconds to read and saves me an hour of context-switching.

The health checks. Overnight it checks the backup ran, the websites are up, the containers are healthy, the disk isn’t filling. If something’s broken it tells me in the morning instead of me finding out the hard way. There’s also an Uptime Kuma monitor doing the same job from the outside, because one monitor is an accident and two is a system.

The rule I stuck to the whole month: an automation that misses results is worse than one that runs slow. So everything is built for coverage first. If the deal cron takes an extra two minutes verifying prices, fine. Missing the deal because I optimised for speed would be a fail.

If you’ve ever told a chatbot something and then had it forget two hours later, you know the pain I was trying to solve. My assistant doesn’t forget. Well, it does, but it has a filing system.

Under the hood there’s something called a knowledge graph. Every conversation I have with it, every email that comes in or goes out, every interesting article it scans, gets captured and linked. People, companies, projects, decisions, dates, all connected. When I ask who I met last month or what we decided about the VPS, it doesn’t guess. It goes and looks.

The clever part is the dream cycle. Once a day the graph consolidates raw capture into proper facts, finds contradictions, and prunes the noise. It’s like a nightly filing run, except the filing clerk is also an analyst.

There’s a human-readable side too: everything also lands in my Obsidian vault on the NAS, so I can browse the same knowledge the assistant has. It’s genuinely weird opening a page in Obsidian and finding a summary of a conversation I’d completely forgotten having.

The email integration is the part that quietly became essential. Every sent email gets scanned and filed against the people involved, so the graph knows who I’ve talked to, about what, and when. My old self would have had to remember that. My new self asks.

Real talk: setting this up was the fiddliest week of the project. Embeddings, providers, sync jobs, a hundred small decisions about what deserves remembering. But it’s the difference between an assistant and a search bar. An assistant that remembers is a colleague. A search bar is just a search bar.

Everyone asks what I’m actually running, so here’s the honest wiring diagram, minus the boring parts.

The brain. The assistant runs on a framework called Hermes Agent, living on my main Windows 11 PC. It’s not a website and it’s not a box in the cloud. It’s an agent that sits on my desktop, connected to the platforms I already use: Telegram (my main chat), iMessage through a bridge, email, SMS, and Home Assistant.

The models. Heavy reasoning goes to cloud models. Anything cost-sensitive or private gets routed to local models running on my RTX 3090 via LM Studio. There’s a routing layer so the assistant picks the right model for the job instead of burning the expensive one on everything. The biggest surprise was how cheap this can be when you route properly.

The reach. This is the part commercial assistants can’t do. From the desktop it talks to my Synology NAS (media stack, backups) and my VPS in the cloud (websites, n8n, databases), all over SSH and Tailscale. It has Docker access on both hosts. It can deploy containers, restart services, check logs, run database queries. When something breaks at 2am, it’s already looking at the logs before I’m awake enough to ask.

The memory. I’ll do a whole post on this, but the short version is: every conversation, email and signal gets captured and organised into a knowledge graph I can search. The assistant remembers. That’s the feature I didn’t know I needed most.

The rules. We settled on a simple operating doctrine after a few early disasters: Python for scheduled jobs, PowerShell for Windows admin stuff, and never delete files without asking. The last one is non-negotiable. It’s the assistant’s equivalent of looking both ways before crossing.

That’s the skeleton. Tomorrow I’ll talk about the part that makes it feel like magic: the memory.

About a month ago I sat down and made a decision I’d been circling for a while: I was going to stop waiting for someone else to build the AI assistant I actually wanted, and build it myself.

I’ve spent the last 15 years in enterprise networking and infrastructure, and more recently I’ve been running Maintain AI, automating workflows for other people. But here’s the thing nobody tells you about doing automation for a living: your own life stays stubbornly manual. My to-do list lived in four different apps. My calendar didn’t talk to my NAS. My media server didn’t know what I wanted to watch. My inbox was a swamp.

The commercial assistants were getting good, don’t get me wrong. But they’re black boxes. My data goes to their servers, they can’t reach into my homelab, they can’t run a script at 5:30pm to check tonight’s takeaway deals, and they certainly can’t SSH into my Synology. I wanted something that lived in my infrastructure, not in someone else’s cloud.

So I built one. This series is the story of that first month: what it is, how it’s wired, what broke, and what genuinely surprised me.

A quick note on how this post (and the rest of this series) exists: it was drafted by my assistant, Hermes, from my notes and the actual logs of what we built, then published after I reviewed it. If you ever see the tag hermes-wrote-this on a post here, that’s what that means. I figure if I’m building an AI assistant, the least I can do is be upfront about when it’s doing the typing.

The short version of the last month: I now have something that reads me a briefing every morning, remembers conversations I forgot I had, texts me deals before I’m hungry, tells me off (politely) when my AI spend spikes, and controls half the house. It’s not perfect. It’s occasionally infuriating. And I wouldn’t go back.

Come for the build logs, stay for the part where the assistant starts correcting my spelling.