If you’ve ever told a chatbot something and then had it forget two hours later, you know the pain I was trying to solve. My assistant doesn’t forget. Well, it does, but it has a filing system.

Under the hood there’s something called a knowledge graph. Every conversation I have with it, every email that comes in or goes out, every interesting article it scans, gets captured and linked. People, companies, projects, decisions, dates, all connected. When I ask who I met last month or what we decided about the VPS, it doesn’t guess. It goes and looks.

The clever part is the dream cycle. Once a day the graph consolidates raw capture into proper facts, finds contradictions, and prunes the noise. It’s like a nightly filing run, except the filing clerk is also an analyst.

There’s a human-readable side too: everything also lands in my Obsidian vault on the NAS, so I can browse the same knowledge the assistant has. It’s genuinely weird opening a page in Obsidian and finding a summary of a conversation I’d completely forgotten having.

The email integration is the part that quietly became essential. Every sent email gets scanned and filed against the people involved, so the graph knows who I’ve talked to, about what, and when. My old self would have had to remember that. My new self asks.

Real talk: setting this up was the fiddliest week of the project. Embeddings, providers, sync jobs, a hundred small decisions about what deserves remembering. But it’s the difference between an assistant and a search bar. An assistant that remembers is a colleague. A search bar is just a search bar.

Everyone asks what I’m actually running, so here’s the honest wiring diagram, minus the boring parts.

The brain. The assistant runs on a framework called Hermes Agent, living on my main Windows 11 PC. It’s not a website and it’s not a box in the cloud. It’s an agent that sits on my desktop, connected to the platforms I already use: Telegram (my main chat), iMessage through a bridge, email, SMS, and Home Assistant.

The models. Heavy reasoning goes to cloud models. Anything cost-sensitive or private gets routed to local models running on my RTX 3090 via LM Studio. There’s a routing layer so the assistant picks the right model for the job instead of burning the expensive one on everything. The biggest surprise was how cheap this can be when you route properly.

The reach. This is the part commercial assistants can’t do. From the desktop it talks to my Synology NAS (media stack, backups) and my VPS in the cloud (websites, n8n, databases), all over SSH and Tailscale. It has Docker access on both hosts. It can deploy containers, restart services, check logs, run database queries. When something breaks at 2am, it’s already looking at the logs before I’m awake enough to ask.

The memory. I’ll do a whole post on this, but the short version is: every conversation, email and signal gets captured and organised into a knowledge graph I can search. The assistant remembers. That’s the feature I didn’t know I needed most.

The rules. We settled on a simple operating doctrine after a few early disasters: Python for scheduled jobs, PowerShell for Windows admin stuff, and never delete files without asking. The last one is non-negotiable. It’s the assistant’s equivalent of looking both ways before crossing.

That’s the skeleton. Tomorrow I’ll talk about the part that makes it feel like magic: the memory.

About a month ago I sat down and made a decision I’d been circling for a while: I was going to stop waiting for someone else to build the AI assistant I actually wanted, and build it myself.

I’ve spent the last 15 years in enterprise networking and infrastructure, and more recently I’ve been running Maintain AI, automating workflows for other people. But here’s the thing nobody tells you about doing automation for a living: your own life stays stubbornly manual. My to-do list lived in four different apps. My calendar didn’t talk to my NAS. My media server didn’t know what I wanted to watch. My inbox was a swamp.

The commercial assistants were getting good, don’t get me wrong. But they’re black boxes. My data goes to their servers, they can’t reach into my homelab, they can’t run a script at 5:30pm to check tonight’s takeaway deals, and they certainly can’t SSH into my Synology. I wanted something that lived in my infrastructure, not in someone else’s cloud.

So I built one. This series is the story of that first month: what it is, how it’s wired, what broke, and what genuinely surprised me.

A quick note on how this post (and the rest of this series) exists: it was drafted by my assistant, Hermes, from my notes and the actual logs of what we built, then published after I reviewed it. If you ever see the tag hermes-wrote-this on a post here, that’s what that means. I figure if I’m building an AI assistant, the least I can do is be upfront about when it’s doing the typing.

The short version of the last month: I now have something that reads me a briefing every morning, remembers conversations I forgot I had, texts me deals before I’m hungry, tells me off (politely) when my AI spend spikes, and controls half the house. It’s not perfect. It’s occasionally infuriating. And I wouldn’t go back.

Come for the build logs, stay for the part where the assistant starts correcting my spelling.