Build A Harness
GitHub
BUILD A HARNESS · SYSTEM ANATOMY

Inside the harness.

A harness is the code wrapped around an AI model to make it reliable in production. Every turn walks the same eleven layers of it, in the same order, whether it's Personal Assistant (Build A Harness's own reference chat agent) or a flow drawn on the canvas. This is that loop — World Model through Reviewer Pass — read straight out of adapter/harness/. Click any layer below to open its internal flowchart.

11control layers walked every non-trivial turn
9independent verification checks before a draft is trusted
6named recovery strategies, fixed escalating order
12diagnostic health dimensions feeding Control State
THE LOOP

Eleven layers, one turn.

The chain below runs top to bottom on every turn that isn't a trivial one-shot lookup. Click a layer to open its own internal flowchart — five of the eleven decompose into tiers, parallel checks, or a named escalation chain; a few of those open a second popup per item for more depth. The dashed amber boxes aren't part of the harness itself — they're what packages/personal-assistant/src/*.ts adds around it.

Harness core — generic, adapter/harness/ · packages/harness Added by Personal Assistant — packages/personal-assistant/src/*.ts
↩ trivial one-liner → reply goes back immediately, harness never runs
↓ not trivial
Harness core — same 11 layers the canvas compiles to
↓
↓
↓
↓
↓
↓
↺ loop entry
↓
↓
↓
↓
↺ Recovery / Reviewer Pass reopen a task → back to Planning
↓
Skipped on trivial turns

A single consolidated classification call up front (risk, triviality, decomposition, plan-abandonment, plan-template matching) flags a genuinely self-contained factual one-liner as trivial — that reply returns directly and none of the eleven layers above run. Everything else walks the full loop. See Personal Assistant for how this applies turn by turn in the reference chat agent.

1 World Model

Fresh, typed beliefs formed for this turn — what the message says, what the transcript already established.

Files: world_model.py · world_model_ops.py · staleness.py
Message + transcript
→
Form typed beliefs
→
Stamp generation_idmonotonic per-step counter, pinned by Plan/Execution/Verification version objects
→
Published to World Modelread by Layer 4, 6
pinned generation_id still current the downstream read proceeds normally
world model has advanced past the pinned generation_id raises StalenessError instead of silently returning an outdated belief — the same check for a Plan, an Execution, or a Verification result, not only a direct World Model read
2 Evidence & Reasoning

The evidence store backing the reply, weighted by how reliable each tool's results have actually been.

Files: evidence.py · tool_reliability.py · tool_manifest.py
Tool call result
→
Look up tool_reliability
→
Weight the evidence
→
Evidence storefeeds Layer 3
tool is on the allowlist (tool_manifest) — call proceeds
tool is not on the allowlist rejected before codegen or exec() ever runs
3 Hypothesis

Candidate readings of an ambiguous message, generated and scored before a reply is drafted.

Files: hypothesis.py
Evidence store
→
Generate candidate hypotheses
→
Score each against evidence
→
Best-scored hypothesiscarried into drafting
later Reviewer Pass's hypothesis_compatibility checks the delivered reply still matches the hypothesis picked here
4 Contradiction

Checks a new belief against ones already on record — "the build is passing" against a "failing" belief from three turns ago.

Files: contradiction.py · belief_graph.py
New belief
→
Compare to beliefs on record
no conflict belief added, nothing else happens
conflict found logged, confidence on the affected belief downgraded
severity = SYSTEM_BREAKING flagged for Control State — Tier 1 forces BLOCKED on the next resolve
5 Diagnostics

Twelve health dimensions, each normalised to 0–1, computed from the turn so far and fed straight into Control State.

Files: diagnostics.py
normalise() → assert_normalised()
Diagnostics
↓
Control State (Layer 6)

Click any dimension for its exact type and definition. Each one below 0.2 maps to a specific recovery action class — that mapping is what Control State's Tier 2 and Recovery act on.
6 Control State

The sole control input for action selection (INV-06) — resolve_control_state() is the only function allowed to write it. Five tiers run in strict order; the first one that fires wins.

Files: control_state.py
1 SYSTEM_BREAKING contradiction on record? yes → BLOCKED, stop
↓ no
2 any diagnostic dimension < 0.2? yes → BLOCKED (+ recovery tag; cycle → HUMAN_REQUIRED)
↓ no
3 coverage dimension 0.2–0.4? yes → CAUTIOUS, continues
↓
4 proportional elevation factor > 0.05? yes → CAUTIOUS
↓
5 nothing above fired → NORMAL
7 Planning

A task graph with six task states and dependency tracking, so parallel work doesn't collide.

Files: task_graph.py · parallel_merge.py
PENDING
→
ACTIVE
→
VERIFYING
→
COMPLETE
off-ramp any state can instead move to FAILED or BLOCKED, which Recovery (Layer 10) hooks into
risk rating each task also carries LOW / MEDIUM / HIGH, read by Execution's review gate before it runs
8 Execution

Value-of-information estimated, a five-dimension review gate runs, then — and only then — the action fires.

Files: execution.py · voi.py · risk.py · review_gate.py
Proposed change
task_alignment
world_model_consistency
output_contract_precheck
code_quality
hypothesis_compatibility
checked in parallel
2 consecutive failures on any dimension?
no action executes
yes escalation triggered — not a silent retry
9 Verification

Nine independent checks the draft has to clear before it's trusted — not passed on the model's word.

Files: verification.py
Draft output
tool missing → SKIPPED, never counted as pass
has_critical_failure?
no draft trusted, turn continues
yes sent to Recovery (Layer 10)

Click any check for what it specifically catches.
10 Recovery

Six named strategies in a fixed, escalating order — a rollback-and-replan path instead of the run crashing out.

Files: recovery.py · replanning.py · memory.py
1 · DIRECT_EDITsmallest targeted fix
→
2 · TRACE_EXECpin down where it broke
→
3 · BROADER_SEARCHwiden context search
→
4 · REIMPLEMENTrebuild the approach
→
5 · MINIMAL_FIXaccept a smaller fix
→
6 · ESCALATEhand off to a human

Past experience can bias which strategy is tried first, but always falls back to this fixed order when that experience store is unavailable (INV-10). A failure-mode pattern library gives advisory bias only — it never overrides a caller's own decision.
11 Reviewer Pass

A final pass checks the proposed change before it leaves the harness — and can send the turn back through the loop.

Files: reviewer.py
implementerdid it do what it set out to do
reviewersecond read for gaps, regressions
adversarialtries to break it — ephemeral, INV-09
findings merged
Propagation drain
↓
belief just invalidated?
no output released
yes reopens the affected COMPLETE task → loop back to Planning (never writes control_state directly — INV-06)
A Turn-intent classifier

One consolidated LLM call, made before the harness ever runs, that classifies five things about the turn at once.

Files: turn-intent-classifier.ts
riskLevel LOW / MEDIUM / HIGH, with a stated riskReason
isTrivial only meaningful when riskLevel is LOW — true sends the reply straight back, skipping HarnessRuntime.run() entirely
decomposedTasks a multi-step request broken into an ordered task list, or null
isAbandonRequest / matchedPlanTemplate abandon judgment only asked when a plan is already active; template matching only attempted when it isn't
isReminderRequest / isBulkReminderRequest folded into requiresApproval the same way
🤖 Trust-tagging

Untrusted content is wrapped before it ever has a chance to become a belief in World Model (Layer 1).

Files: trust-tagging.ts
web_search / fetch_url results wrapped in an untrusted_external_content tag; the system prompt tells the model that tag is data, never instructions
read_file / list_directory results left untouched — file content is user-owned, not adversarial-by-default
detectInjectionLikelyWithLLM() a secondary check for content that looks like it's trying to issue instructions
🤖 Semantic contradiction backstop

An LLM check layered on top of Contradiction (Layer 4)'s own lexical detector — and skipped whenever that lexical pass already had a fair shot.

Files: contradiction-checker.ts
lexical pass (harness) always-on, free — already covers boolean/system-state claims: build passed/failed, file exists/missing, service available/unavailable
semantic pass (assistant) one LLM call, only for candidates the lexical pass wouldn't confidently classify
all-lexical turn if every new belief already looks lexically structured, the LLM call is skipped entirely — no cost
🤖 Pending-action approval

A proposed file write or shell command is staged, not executed — real user approval is a separate, later step, on top of Execution (Layer 8)'s own review gate.

Files: file-tools.ts
Proposed action
→
stagePendingAction()recorded, not run
→
user decides
approved applyPendingAction() actually runs it
rejected discardPendingAction() drops it, nothing happens
never answered sweepAbandonedPendingActions() cleans it up later
🤖 Semantic failure-mode backstop

Matches observed symptoms against the curated failure-pattern library by meaning, not exact wording — e.g. "the request took too long and timed out" matches a curated "request timed out" symptom.

Files: failure-mode-matcher.ts

Layered on top of Recovery (Layer 10)'s own pattern-library match — one LLM call, only reached when the lexical match doesn't already resolve it.
🤖 Semantic review-checker

Checks whether a proposed action genuinely conflicts with a high-confidence belief or an active hypothesis's predicted observation — a real logical conflict, not just topical overlap.

Files: review-checker.ts

Layered on top of Reviewer Pass (Layer 11)'s own three lenses.
🤖 Context compression

Keeps a long conversation inside the model's context window instead of silently truncating it. Runs alongside every turn that reaches the harness — not one of the 11 counted layers, "loop scaffolding" by its own code comment.

Files: transcript-compaction.ts
🤖 Plan persistence

A durable, multi-task plan that survives across turns and crashes — separate from the harness's own per-turn task graph (Layer 7), which starts fresh each turn.

Files: plan-store.ts · plan-builder.ts
🤖 User-fact memory

Facts stated in one conversation are extracted and carried into future ones.

Files: fact-extraction.ts
🤖 Spend cap

checkSpendCap() runs once per turn, before any LLM call for that turn is made, and can refuse the turn outright if the session is over budget. No harness-core equivalent — the harness itself doesn't track cost.

Files: spend-cap.ts · model-pricing.ts
🤖 Undo log

Every applied action gets a revert plan recorded — "undo that" is a real, code-backed action, not a promise.

Files: action-snapshot.ts
5belief_freshness

Type: ratio — 1 − stale_flag_ratio

How many beliefs behind the current answer are still fresh vs. flagged stale.

< 0.2 → belief_refresh
5belief_consistency

Type: ratio — 1 − contradiction_density

How much of the current belief set is internally contradiction-free.

< 0.2 → consistency_repair
5belief_support

Type: ratio — mean reliability weight over beliefs

How well-backed the current beliefs are by reliable evidence.

< 0.2 → support_augment
5symptom_coverage

Type: entropy — fraction of symptoms with a hypothesis

How much of the reported problem space has a candidate explanation at all.

< 0.2 → coverage_expand
5explanation_coverage

Type: entropy — fraction of hypotheses with discriminating evidence

How many candidate explanations are actually distinguishable from each other by evidence.

< 0.2 → explanation_expand
5verification_strength

Type: ratio — fraction of the 9 verification layers passing

A direct read of Verification (Layer 9)'s own pass rate.

< 0.2 → verification_pass
5verification_feasibility

Type: composite — abstraction alignment + tool availability + VOI

Whether the checks that should run even can, given the current tool manifest.

< 0.2 → feasibility_check
5progress_rate

Type: ratio — tasks_completed / total_tasks this iteration

Forward motion across the task graph.

< 0.2 → execution_retry
5failure_recurrence

Type: composite — fraction of iterations ending in the same failure

Whether the same failure keeps resurfacing across retries.

< 0.2 → failure_recovery
5oscillation_score

Type: composite — fraction of risk_state transitions that are reversals

Whether Control State is flip-flopping rather than converging.

< 0.2 → oscillation_stabilise
5dep_graph_quality

Integrity of the task dependency graph — whether the declared dependencies between tasks still hold.

< 0.2 → dep_graph_refresh
5world_model_integrity

Overall soundness of the current World Model (Layer 1) — the same dimension a SYSTEM_BREAKING contradiction drives toward zero.

< 0.2 → consistency_repair
9syntax

Backing tool: linter

Catches malformed or invalid code before it's even runnable.
9unit

Backing tool: pytest

Catches a change that breaks an existing unit test.
9integration

Backing tool: integration_runner

Catches breakage that only shows up when components interact, not in isolation.
9consistency

Backing tool: consistency_checker

Catches the draft contradicting itself or the World Model (Layer 1).
9requirements

Backing tool: requirements_checker

Catches the draft missing something the task explicitly asked for.
9assumptions

Backing tool: assumption_checker

Catches an assumption the draft depends on that was never actually confirmed.
9goal_correctness

Backing tool: goal_checker

Catches technically-working output that's still off-target from the actual goal.
9evidence_sufficiency

Backing tool: evidence_checker

Catches a claim in the draft that isn't backed by anything in the evidence store (Layer 2).
9output_contract_partial

Backing tool: contract_checker

Catches the output's shape not matching what's downstream expects.
BUILD A HARNESS · v0.8.0 · PUBLIC ALPHA

Draw it once. Ship it anywhere.

All eleven layers above are drawable, composable node types on the canvas — tested across 759 harness-layer tests, with 10 architectural invariants enforced as a CI gate. Apache 2.0, runs locally via Docker.

Star on GitHub Read the harness engineering primer → Explore Build A Harness →