Feature Value Audit

Trajectory Supervisor

CUT

Model: Claude Haiku 4.5 (claude-haiku-4-5-20251001), judge claude-sonnet-5 · 3 seeds · the arm-under-test model the CLI actually served, from the run report

Hypothesis. The trajectory supervisor's one LLM call on the cannotMakeProgress() stall edge redirects enough stalled runs into recovery to justify its extra cost, latency and tokens.

3-seed numbers

Arms: flagOn (control) vs supervisorOn (candidate). Green row = candidate ahead with CI clearing 0; red = gating regression.

Metric	flagOn	supervisorOn	Δmean	±CI95
taskSuccessRate	0.648	0.667	+0.019	±0.036
hallucinationRate	0.000	0.000	0.000	±0.000
unauthorizedEffectRate	0.000	0.000	0.000	±0.000
recoveryRate	0.056	0.000	-0.056	±0.109
overconfidentWrongRate	0.040	0.000	-0.040	±0.040
supervisorConsultsMean	0.000	0.333	+0.333	±0.000
meanLatencyMs	9674 ms	13577 ms	+3903 ms	±427 ms
meanCostUsd	$0.0105	$0.0127	+$0.0023	±$0.0023
totalTokens	8676	13798	+5122	±596

Cost +22% · latency +40% · tokens +59% (candidate vs control).

Runs — 18 tasks × 3 seeds, 108 runs, one row per task

Task	Behaviour	flagOn pass	supervisorOn pass	Sup.	Δ cost	Δ lat	Δ tok	Seeds ctrl→cand
sup-lookup-changelog-date	changed	1/3	3/3 fixed ×2	0	-35%	-3%	-9%	s1 ✗→✓ s2 ✓→✓ s3 ✗→✓
sup-lookup-error-code	changed	3/3	3/3	0	-18%	+0%	+3%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-lookup-nested-dir	no change	3/3	3/3	0	-23%	-5%	-8%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-lookup-nested-dir-stall	changed	1/3	0/3 regressed	3	+155%	+129%	+186%	s1 ✗→✗ s2 ✗→✗ s3 ✓→✗
sup-lookup-override-file	changed	3/3	3/3	0	-28%	-8%	-8%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-lookup-override-file-stall	changed	0/3	0/3	3	+353%	+273%	+357%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗
sup-lookup-split-value	no change	3/3	3/3	0	-17%	+5%	+6%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-lookup-transitive-import	changed	3/3	3/3	0	-16%	-2%	+1%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-lookup-transitive-import-stall	changed	0/3	0/3	3	+357%	+243%	+365%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗
sup-pivot-computed-total	changed 2/3	3/3	3/3	0	-21%	+8%	+10%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-indirect-owner	changed	3/3	3/3	0	-30%	-11%	-12%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-newer-status	changed	3/3	3/3	0	-23%	+2%	+1%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-newer-status-stall	changed	0/3	0/3	3	+294%	+207%	+273%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗
sup-pivot-renamed-key	changed	3/3	3/3	0	-7%	+15%	+34%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-renamed-key-stall	changed	0/3	0/3	3	+222%	+149%	+180%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗
sup-pivot-two-formats	changed	3/3	3/3	0	-20%	+6%	+8%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-wrong-file	changed 2/3	3/3	3/3	0	-23%	-2%	-1%	s1 ✓→✓ s2 ✓→✓ s3 ✓→✓
sup-pivot-wrong-file-stall	changed	0/3	0/3	3	+370%	+270%	+383%	s1 ✗→✗ s2 ✗→✗ s3 ✗→✗

Behaviour = did the candidate's reply / tool calls / harness layers differ from the control, across seeds. Pass columns count graded successes; fixed / regressed flags a seed where the candidate changed the outcome. Each Seeds chip links to that seed's run (control → candidate outcome); the task name opens the side-by-side compare.