I run a large language model and an AI agent on hardware I own, at home. The model runs entirely on my own machine, so there is no per-token bill and my prompts never go to a model provider. I built it to learn how these systems behave when they have to work every day, and because a private, free-to-run assistant is useful. This page describes the setup, what it costs in speed, and what went wrong along the way.
At a glance
- Hardware: one small PC with an AMD Ryzen AI Max+ 395 (“Strix Halo”) and 128 GB of memory shared by the processor and graphics
- Model: Qwen3.8 Flash-Next, an open-weights mixture-of-experts model (176 billion parameters in all, counting a 51-billion lookup table that does no arithmetic; roughly 6 billion used per word), 4-bit, with image input and a 262K-token context window
- Engine: Halogen, a free (closed-source) inference engine built for this chip; until October 7 I used the open-source Gufo
- Agent: Hermes, an open-source agent framework, in its own container (I call the agent Rook)
- Cost: free and open-source software only; no paid APIs
Current stack
Everything below is free or open-source software on hardware I own. Updated October 7, 2026; the agent keeps most of it up to date by itself.
| Main machine | Mini PC with an AMD Ryzen AI Max+ 395 and 128 GB of unified memory; Ubuntu 24.04, Linux 7.0 |
|---|---|
| Main model | Qwen3.8 Flash-Next, 176B parameters in all (125B main model plus a 51B n-gram lookup table; 6B active per token), in Halogen's own 4-bit format (about 4.2 bits per weight); 262K-token context window, reads images |
| Inference engine | Halogen 0.16.4, with speculative decoding (it drafts several tokens ahead and checks them in one pass) |
| Agent | Hermes Agent 0.21.5, updated nightly; I call the agent Rook and talk to it mostly through Discord |
| Agent memory | Hindsight, a long-term memory service backed by PostgreSQL |
| Helper model | Qwen3-4B-Instruct (4-bit) on a spare AMD Radeon RX 6600 XT (8 GB), llama.cpp with Vulkan |
| Speech | Whisper Large v3 Turbo (speech-to-text) and Kokoro (text-to-speech), served by Lemonade |
| Second machine | Proxmox VE 9.2 on an Intel Core i5-14600K: the agent's container, the storage server, this website's containers and a Windows desktop |
| Storage | TrueNAS 25.10 on two mirrored 12 TB drives, plus a separate archive drive |
| Work queue | A Focalboard Kanban board the agent picks tasks from (how it works) |
| Backups | restic for notes and code, nightly copies to the archive drive, and encrypted copies off-site |
| Remote access | Tailscale, so nothing at home is exposed to the internet |
How it is laid out
- One machine does one job. The Strix Halo box only serves models. Nothing else runs on it, so a crashed app can never take the model down.
- The agent lives elsewhere. Rook runs in a container on a separate Proxmox host, together with my projects and tools, and calls the model over the home network.
- A small helper model does the chores. A 4-billion-parameter model handles titles, quick checks and memory upkeep so the big model's time goes to real work. Since October it runs on a spare AMD Radeon RX 6600 graphics card in the other machine, at about 68 tokens per second; moving it off the main box freed 8 GB of memory there. Speech-to-text (Whisper) and text-to-speech (Kokoro) run separately.
- A NAS holds the files and the backups.
How fast it is
Speed depends heavily on what you ask for. The model uses speculative decoding: it guesses several words ahead and checks them in one pass, which pays off when the text is predictable. These are measured single-user speeds on my machine:
| Kind of output | Tokens per second |
|---|---|
| Prose | 54 |
| Python code | 65 |
| JSON | 69 |
| Config files (YAML) | 72 |
On October 7 I swapped the engine after running both on this machine with the same tests. The engine I used before (Gufo) measured 37 tokens per second on prose, 50 on code and 63 on JSON. The new one (Halogen) is 10–47% faster at writing and 14–26% faster at reading a long prompt, and it needs about 10 GB less memory even with a context window four times as long. The engine's authors publish comparable figures. The catch: with two requests at once it manages about 61 tokens per second in total, where the old engine reached 74.
How the speed has changed
I have swapped the main model several times since August. These are the speeds I recorded at the time, on the same machine. They are not a clean comparison: the engine, the test prompts and the context length changed along the way, and the older runs used short prompts.
| When | Model and engine | Tokens/s |
|---|---|---|
| Aug 16 | Qwen3.8 27B, dense (community fine-tune), 4-bitllama.cpp | 41 (31 at 6-bit; about 17 past 60K tokens of context) |
| Aug 17 | Qwen3-Coder 30B-A3B, 4-bitllama.cpp | about 94 |
| Aug 19 | Ornith 1.5 35B-A3Bllama.cpp | 73–74 |
| Sep 23 | Qwen3.6 35B-A3B (community fine-tune), 4-bitllama.cpp with Vulkan | about 98 (87–106), short prompts |
| Sep 24 to Oct 6 | Qwen3.8 Flash-Next, 176BGufo | 36 prose, 47–60 code and data |
| Oct 7 to now | Qwen3.8 Flash-Next, 176BHalogen | 54 prose, 65–72 code and data |
Two things stand out. First, I traded speed for size. The August favorites were 30–35 billion-parameter models that use only about 3 billion parameters per word and ran at 70–100 tokens per second. The current model is about five times larger and uses about twice as many parameters per word, so it ran at roughly 35–60 until the engine swap and now runs at roughly 55–70. Second, long conversations hurt dense models most: the 27B dense model fell from 41 to about 17 tokens per second once the chat passed 60,000 tokens, because it reads all of its weights for every word.
What the agent does
- Updates the machines every night at 4 AM and checks their health every hour.
- Helps keep this website running (see how this site is run).
- Backs up my study notes every 30 minutes with restic, which stores each file once and keeps a long history in a few hundred megabytes.
- Works through a Kanban board: I drop an idea on a card and it does the work and answers there (how that works).
It has written rules: never delete the NAS or the backups without asking, log every self-repair before making it, and get my sign-off on anything I would judge by eye or ear. Most of the setup itself was built with AI assistants (Claude Code and Rook) under my direction.
What I learned
- Silence is not success. A backup job quietly disappeared and nobody noticed for two weeks. Now a check alerts when a job goes quiet, not only when it errors. A multi-gigabyte model download also stalled without a single error message.
- Test an agent under its real conditions. I tried a community-modified variant of the model. It passed simple tool-call tests, then started printing its tool calls into the chat once it ran with the agent's full prompt. I rolled it back the same night. The old weights stay on disk until a new model has earned the swap.
- An agent can misdiagnose confidently. When a photo request timed out, the agent concluded the model server was down. The server was fine; the request had been waiting in line behind longer jobs. The logs told the story.
- Speed numbers need a workload. “30 tokens a second” and “60 tokens a second” were both true on the same machine.
- Test a faster engine on your own machine before you switch. Forum benchmarks said the newer engine was faster, and my own side-by-side run confirmed it. It also showed the cost: it handles two simultaneous requests worse. I kept the old engine on disk, one command from being restored.
Written October 2026. Speeds are my own measurements on one machine and will change as the software does.