Your Name
b993f25c53
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the exercise; deterministic gate = solution test + approved file. Key results: mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both); weak model fails only under the mandate (A+Q), passes when free or optional. Drift documented with code: scorer.updateSelection left inside the extracted method (A+Q, C+Q). Harness in tools/rehearse; summary in rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
1.7 KiB
1.7 KiB
rehearsal-runs/ — curated per-run evidence (2026-09-27)
One directory per sub-agent run. Generated by tools/rehearse/run-rehearsal.sh;
full raw sessions (pi jsonl transcripts, complete workspaces) live in
/tmp/rehearsal/runs/<run>/ and are intentionally not committed.
Per run:
| File | Content |
|---|---|
agents-md.txt |
The exact AGENTS.md the agent worked against (shared design instructions + variant tool rules) |
prompt.txt |
The user prompt given to pi -p |
measure.txt |
Tokens (sum + per-usage breakdown), turn/tool-call counts, session span, wall clock, tool-name histogram (from measure_run.sh) |
verify.txt |
Deterministic gate result: diff size vs pristine, mvn test with the solution's test + approved file, structural checks (see verify_outcome.sh) |
diff.patch |
Final diff of the run workspace vs the pristine exercise (excludes test artifacts) |
pi-stdout.txt |
The agent's final report (summary + final diff as it saw it) |
time.txt |
/usr/bin/time -v output (wall clock, max RSS, exit status) |
mvn-verify-tail.log |
Tail of the mvn output for the gate run — only present for failures |
Run → variant/model map:
| Dir | Variant | Model |
|---|---|---|
A |
CLI mandated | openrouter z-ai/glm-5.3-flash |
B |
plain pi tools | openrouter z-ai/glm-5.3-flash |
C |
CLI optional + intro | openrouter z-ai/glm-5.3-flash |
A+Q |
CLI mandated | entrim Qwen/Qwen3.6-35B-A3B |
B+Q |
plain pi tools | entrim Qwen/Qwen3.6-35B-A3B |
C+Q |
CLI optional + intro | entrim Qwen/Qwen3.6-35B-A3B |
Narrative, drift documentation, and conclusions: see
../rehearsal-findings.md.