Project
I build software with an AI agent, which raises an obvious question: who tests the software? Not me — I have a day job. So I built the thing that tests it by itself: an agent that learns my web application the way a curious human would, by clicking around, trying things, and remembering what happened. Then it reports bugs.
At a glance
- Target: my wargame web app (Django + SQLite) — the same Domain Maneuver project
- Brains: a local LLM (see Running AI at Home), no cloud APIs
- Memory: a persistent world model in SQLite that grows every night
- Proof: every mission is checked by a machine against the database — the agent can't grade its own homework
- Data on this page: regenerated automatically after each nightly run · last learned 2026-10-20
The idea
Most automated testing is brittle: scripts that know where a button is, and break when the page changes. I wanted something that behaves like a new employee on their first day — it doesn't know where anything is, it explores, it notices when something breaks, and it gets better every day it works here. Concretely, each night the agent spawns two copies of my app in a sandbox, and plays episodes: it looks at the page, picks an action, acts, and observes what changed.
What makes the learning stick is the memory. Every page it visits becomes a state; every action that changes the page becomes an edge. After a few nights it has a map of the whole application — which pages exist, what each button does, which paths lead somewhere new. Exploration is steered by novelty: pages it has already mapped score zero, so its attention naturally drifts toward the parts of the app it doesn't know yet.
What it has learned so far
Live numbers
- 1550 distinct page states mapped
- 6121 action-to-effect edges learned
- 739 interactive elements catalogued
- 340 episodes played (334 by the LLM policy)
- 23 lessons distilled into its playbook
- 180 bug events logged — two root causes so far, write-ups below
Separate from exploration, it gets missions with machine-checkable success conditions — for example “create a unit on the map” is verified by querying the database for the resulting row. An LLM that wants a reward will happily believe it succeeded; the postcondition check is what keeps it honest.
Mission scoreboard
| Task | Scripted policy | LLM policy | |
|---|---|---|---|
a | Stand up a unit and place it on the map | done (scripted + agent) | done (agent) |
b | Resolve turn state and submit the other side's move | done (scripted) | open |
c | Flip a hidden feature toggle | done (scripted) | open (digest fix pending) |
d | Create an article, then correct it | done (scripted + agent) | done (agent) |
e | Snapshot the game state | done (scripted + agent) | done (agent) |
Bugs it found on its own
This is the part I didn't expect to work. Nobody told it what to look for — it just notices when reality disagrees with expectation:
- A blank map, ninety-six times. Every page that renders the tactical map produced a grid with no terrain. The agent logged the identical console error ninety-six times, and the report trace nailed the cause: the map images are routed by my web-server config, so pages served without it — development setups — ask the app server for tiles it doesn't serve. Production is fine; a fresh clone isn't.
- Developer pages in production clothing. Wrong URLs returned a debug page that helpfully lists the site's internal address structure. Development configuration on a public listener — exactly the kind of sloppy leftover a real attacker would try first, and exactly what a fuzzer finds. Fix is a one-line settings change, queued behind the review that owns it.
What it taught me
- Verification beats enthusiasm. The model claimed success on tasks it failed (the scoreboard is honest because the database is not). Machine-checkable postconditions are the whole difference between a demo and a test suite.
- Seeing is a feature. The most interesting open failure isn't the model being dumb — it's that my page-summary format didn't tell the model whether a checkbox was checked. It cannot reason about state it cannot perceive.
- Small and consistent beats big and vague. Plain-language playbook lessons, injected into every episode prompt, took a task from failed to passed between two runs. No retraining needed yet.
What's next
- The hardest mission — walking the turn loop my game exists for — is still open for the LLM policy. That is the interesting failure mode: pages that exist but never surface in the links it follows.
- It runs nightly on my home server and updates this page from its own database as it goes. The numbers above are as of the latest campaign; coverage of the app's URL space is roughly 4% and climbing.
Built October 2026. The app under test is my own; the agent runs only in sandboxes on my local network, and this page shows only what I chose to publish.