The architecture page shows every layer and backstop we built. This one shows which of them we've proven earns its latency and token cost — and which we've measured, found wanting, and switched off. First result below: the Trajectory Supervisor.
A harness is a governance and reliability control plane wrapped around a model. Every layer it adds costs latency and tokens on every turn. That cost is only worth paying if the layer actually changes outcomes — catches a contradiction the free lexical check missed, recovers a run that would otherwise stall, stops a wrong answer going out.
The honest way to know is a controlled benchmark that is allowed to come back negative. So we built one, and we run it against our own features. When a feature doesn't earn its cost, we say so here and switch it off — the code stays, flag-gated, so a later test on a different task shape can flip it back if the evidence changes.
Almost every candidate has the same shape: an expensive per-turn LLM call layered on top of a free lexical or mechanical check, added on the bet that the cheap check misses real cases often enough to pay for itself. That bet is exactly what the benchmark tests.
Modelled on how ML systems are evaluated: a fixed dataset, a mechanical grader, an isolated independent variable, and a confidence interval on the result.
Write, in one sentence, the belief the feature encodes — the thing that would have to be true for it to be worth the cost.
A benchmark arm that flips this one feature and nothing else. Everything else — model, corpus, grader — is held fixed.
A tagged set of tasks built so the feature should show its value here if it shows it anywhere. No delta on its own stress slice means no value.
Run the matrix at least three times. Report Δ task success, recovery, overconfident-wrong answers, cost, latency and tokens — each with a 95% confidence interval.
Keep: positive delta on the stress slice, no regression elsewhere, cost within budget. Cut: negative, or neutral with real cost. Expand: underpowered — grow the corpus and re-run.
Not just the number. Where the idea came from, what we expected, what actually happened, and why — as an entry on this page, plus a row in the benchmark doc and the machine report JSON in the repo. Negative results included.
One row today. This table grows as each audit lands — positive and negative alike.
| Feature | The bet it encodes | Verdict | Evidence |
|---|---|---|---|
| Trajectory Supervisor HARNESS_TRAJECTORY_SUPERVISOR · Layers 7 & 10 | A stalled run recovers better with a meta-controller that reads the failure trajectory and issues a targeted directive than by walking the deterministic strategy ladder. | Cut | 3-seed, 18-task, judge-on. Task success −3.7% (CI ±3.6%); latency +34%, cost +32%, tokens +48%; zero extra stall recoveries. Full case study ↓ · Read the actual runs → |
A slow-loop meta-controller that wakes up only when the run has measurably stopped making progress. It reads a digest of the failure trajectory and returns one directive — redirect the strategy, reframe the plan, spawn a bounded read-only investigation, ask the user a targeted question, or abort. Between those moments it stays out of the way: the main agent keeps every tactical decision.
In August 2026 NVIDIA published AVO, an agent architecture that scored a perfect 100.00 across all 25 ARC-AGI-3 environments and ran a seven-day GPU-kernel optimisation that beat cuDNN and FlashAttention-4. Their writeup credits two components for sustaining that kind of long-horizon autonomy: a supervisor that "monitors the broader trajectory for stagnation or repeated unproductive cycles and can redirect the main agent toward alternative strategies," and a persistent trajectory memory that lets the agent "resume from the current state rather than repeatedly reconstructing the search."
That shape mapped almost one-to-one onto our own recovery path — except ours was a blind ladder: on a stall it just advanced to the next strategy archetype without reading why it was stuck. Making that branch intelligent looked like free upside.
One caveat we wrote down before starting: AVO's report contains no controlled ablation of the supervisor — the authors say plainly that their experiment "does not isolate its individual contribution." A perfect score on a seven-day task is a strong signal for the architecture as a whole, not evidence that this one part carries its weight. Which is exactly what our benchmark is for.
A stalled run recovers more often, or more cheaply, with a targeted directive than by blindly advancing to the next strategy archetype in a fixed ladder.
The arm is the assistant with the supervisor flag on versus off — nothing else changes. The stress corpus is six lookup and pivot tasks that inject a persistent tool failure, so the run is forced to the stall edge where the supervisor is actually consulted. Everything is graded mechanically and re-run across three independent seeds with an LLM judge on.
Task success moved +2.8% (CI ±10.9%) — noise. Worse, the supervisor couldn't have helped: its two commonest directives can't re-queue a single-node task graph, so on a one-shot assistant turn there was nothing for them to act on. The measurement apparatus, not the idea, was the blocker.
Four changes: the recovery path now re-queues a failed leaf task after a redirect or an investigation; the investigation sub-agent can now read workspace files (it was limited to web search); its findings are spliced into the model's context; and the failure-injection harness allows a real second attempt over the new evidence. Now the supervisor runs a full recover cycle on every stall.
With the supervisor able to act: task success 72.2% → 68.5% (Δ −3.7%, CI ±3.6% — just outside the band). Latency +34%, cost +32%, tokens +48%, all far outside their intervals. And zero additional stalls recovered — the assistant already recovers one of the six on its own, and routing that recovery through the supervisor's extra model call converts none.
eval/reports/2026-09-07T08-54-28-789Z.multiseed.json. Per-seed task
success — off: 72.2 / 72.2 / 72.2; on: 72.2 / 66.7 / 66.7. No change in hallucination or unauthorized
actions on either arm. The transcript-backed 3-seed re-run is published below —
read every run, turn by turn →
Every turn of every arm × task × seed is published as a browsable transcript — the model turns, the tool calls and their results, the supervisor's directive text where it fired, and the grader checks — so the −3.7% is a number you can inspect, not one you have to trust. Open the transcript index → Each run's page names the model it executed on.
The idea didn't transfer. AVO's supervisor earns its keep across a seven-day search with thousands of actions and long plateaus to detect. An assistant turn resolves in one or two iterations — there is almost nothing for a stagnation monitor to monitor, and when a stall is forced, the deterministic recovery already handles the one case in six that's recoverable at all. Adding a model call on top of that bought nothing and cost a third more per turn.
Verdict: cut. In the single-turn assistant shape, the Trajectory Supervisor does not
earn its cost. HARNESS_TRAJECTORY_SUPERVISOR stays off by default. The code stays merged and
flag-gated. One open question remains — a decomposed, multi-step planning path, closer to AVO's own
setting, where a redirect or a re-framed plan has downstream tasks to affect — and that needs its own
benchmark slice before the idea is closed for good.
Ordered by how much new scaffolding a verdict needs. Batch A is a single run against a corpus that already exists. The five features below with a runs → link are queued in the audit matrix; the scheduler grinds through them one seed at a time and the link fills in with the full turn-by-turn transcripts as each feature finalizes.
| Feature | Per-turn cost | Batch |
|---|---|---|
| The 11-layer harness itselfvs. a bare ReAct loop · runs → | Staging, control state, verification, reviewer pass, memory — the whole control plane. | Batch A — next |
| One-loop proposerASSISTANT_ONE_LOOP · runs → | The harness drives every tool call in-loop, instead of reviewing an already-finished reply. | Batch A — next |
| Semantic contradiction backstopLayer 4 · runs → | +1 LLM call per new batch of beliefs, when the lexical negation-pair check can't classify them. | Batch B |
| LLM injection detection on tool outputevery fetched page / shell result · runs → | +1 LLM call on every web page and command output, scanning for prompt-injection. | Batch B |
| Semantic failure-mode backstopLayer 10 · runs → | +1 LLM call whenever the exact-string match against the failure library finds nothing. | Batch B |
| The reviewer passLayer 11 · 3 lenses | Consistency, adversarial and abstraction-fit review on every harness turn. | Batch C |
| The verification layerLayer 9 · 9 sub-checks | Always-on mechanical verification of every result before it's accepted. | Batch C |
| Up-front decomposition + task reframingLayer 5 | Breaking a request into a task graph and rewriting each task to lead with its subject. | Batch C |
| Model-inferred factsnever auto-promoted to memory | The classifier's free-form read of what the user implied — recorded for the session, never kept. | Batch C |
| Semantic criterion coverageLayer 11 | +1 LLM call per plan success-criterion the reviewer's substring check couldn't match. | Batch C |
| Semantic review-checkerLayer 11 | +1 LLM call per proposed change, checking for a real conflict with a held belief. | Batch C |
| AnswerClaim attributionevidence / confidence line | Cheap to compute — a UX surface. Its value is a trust question, not a benchmark one. | Open — not benchmark-shaped |
The task corpus, the mechanical grader, the arm definitions and the multi-seed runner all
live in the product repo under packages/personal-assistant/eval/. Every result on this page
links to its machine report JSON, and each run records the model it executed on. One command re-runs the
supervisor matrix:
npx tsx scripts/run-harness-benchmark.ts \
--arms=flagOn,supervisorOn \
--slice=supervisor_lookup,supervisor_pivot \
--seeds=3Because "11 layers, 27 node types, thousands of tests" proves we built something, not that it works. A harness is only worth its latency and token cost if it changes outcomes. The honest way to know is a benchmark that can come back negative — and when it does, we say so and cut the feature.
No. A cut feature stays merged and flag-gated off. The negative result is recorded, not overridden. If a later test on a different shape of task shows value, the flag can flip back on — but the default reflects the evidence we actually have.
The internal gate every behaviour-changing change must clear: a measured delta on a fixed task corpus, multi-seed with a reported confidence interval, before it becomes a default. A feature that shows no positive delta on its own stress slice does not ship on by default.
Each run writes the model id it actually executed on — and the LLM judge's model — into its machine report, and every transcript page names both, so a published number can always be traced back to the model that produced it.
The architecture page is the map of everything the harness contains, read straight out of the source. This page is the running record of which of those parts we've put on the scale — and what the scale said.
Aielia ships the harness — and the benchmark that keeps it honest. Every layer is a drawable canvas node if you want to build and evaluate your own.