Running AI at Home

I run a large language model and an AI agent on hardware I own, at home. The model runs entirely on my own machine, so there is no per-token bill and my prompts never go to a model provider. I built it to learn how these systems behave when they have to work every day, and because a private, free-to-run assistant is useful. This page describes the setup, what it costs in speed, and what went wrong along the way.

At a glance

Current stack

Everything below is free or open-source software on hardware I own. Updated October 7, 2026; the agent keeps most of it up to date by itself.

Main machineMini PC with an AMD Ryzen AI Max+ 395 and 128 GB of unified memory; Ubuntu 24.04, Linux 7.0
Main modelQwen3.8 Flash-Next, 176B parameters in all (125B main model plus a 51B n-gram lookup table; 6B active per token), in Halogen's own 4-bit format (about 4.2 bits per weight); 262K-token context window, reads images
Inference engineHalogen 0.16.4, with speculative decoding (it drafts several tokens ahead and checks them in one pass)
AgentHermes Agent 0.21.5, updated nightly; I call the agent Rook and talk to it mostly through Discord
Agent memoryHindsight, a long-term memory service backed by PostgreSQL
Helper modelQwen3-4B-Instruct (4-bit) on a spare AMD Radeon RX 6600 XT (8 GB), llama.cpp with Vulkan
SpeechWhisper Large v3 Turbo (speech-to-text) and Kokoro (text-to-speech), served by Lemonade
Second machineProxmox VE 9.2 on an Intel Core i5-14600K: the agent's container, the storage server, this website's containers and a Windows desktop
StorageTrueNAS 25.10 on two mirrored 12 TB drives, plus a separate archive drive
Work queueA Focalboard Kanban board the agent picks tasks from (how it works)
Backupsrestic for notes and code, nightly copies to the archive drive, and encrypted copies off-site
Remote accessTailscale, so nothing at home is exposed to the internet

How it is laid out

How fast it is

Speed depends heavily on what you ask for. The model uses speculative decoding: it guesses several words ahead and checks them in one pass, which pays off when the text is predictable. These are measured single-user speeds on my machine:

Kind of outputTokens per second
Prose54
Python code65
JSON69
Config files (YAML)72

On October 7 I swapped the engine after running both on this machine with the same tests. The engine I used before (Gufo) measured 37 tokens per second on prose, 50 on code and 63 on JSON. The new one (Halogen) is 10–47% faster at writing and 14–26% faster at reading a long prompt, and it needs about 10 GB less memory even with a context window four times as long. The engine's authors publish comparable figures. The catch: with two requests at once it manages about 61 tokens per second in total, where the old engine reached 74.

How the speed has changed

I have swapped the main model several times since August. These are the speeds I recorded at the time, on the same machine. They are not a clean comparison: the engine, the test prompts and the context length changed along the way, and the older runs used short prompts.

WhenModel and engineTokens/s
Aug 16Qwen3.8 27B, dense (community fine-tune), 4-bitllama.cpp41 (31 at 6-bit; about 17 past 60K tokens of context)
Aug 17Qwen3-Coder 30B-A3B, 4-bitllama.cppabout 94
Aug 19Ornith 1.5 35B-A3Bllama.cpp73–74
Sep 23Qwen3.6 35B-A3B (community fine-tune), 4-bitllama.cpp with Vulkanabout 98 (87–106), short prompts
Sep 24 to Oct 6Qwen3.8 Flash-Next, 176BGufo36 prose, 47–60 code and data
Oct 7 to nowQwen3.8 Flash-Next, 176BHalogen54 prose, 65–72 code and data

Two things stand out. First, I traded speed for size. The August favorites were 30–35 billion-parameter models that use only about 3 billion parameters per word and ran at 70–100 tokens per second. The current model is about five times larger and uses about twice as many parameters per word, so it ran at roughly 35–60 until the engine swap and now runs at roughly 55–70. Second, long conversations hurt dense models most: the 27B dense model fell from 41 to about 17 tokens per second once the chat passed 60,000 tokens, because it reads all of its weights for every word.

What the agent does

It has written rules: never delete the NAS or the backups without asking, log every self-repair before making it, and get my sign-off on anything I would judge by eye or ear. Most of the setup itself was built with AI assistants (Claude Code and Rook) under my direction.

What I learned

Written October 2026. Speeds are my own measurements on one machine and will change as the software does.