learning-hours/learning-hour-maker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add agent rehearsal harness, findings, and per-run evidence for split-loop-with-lambda

Browse Source 
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the
exercise; deterministic gate = solution test + approved file. Key results:
mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both);
weak model fails only under the mandate (A+Q), passes when free or optional.
Drift documented with code: scorer.updateSelection left inside the extracted
method (A+Q, C+Q). Harness in tools/rehearse; summary in
rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in
rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
This commit is contained in:
Your Name
2026-09-27 17:36:19 +00:00
parent a27e32688a
commit b993f25c53
51 changed files with 2281 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ideas/split-loop-with-lambda/exercise/rehearsal-runs/README.md
+32
View File
@@ -0,0 +1,32 @@
# rehearsal-runs/ — curated per-run evidence (2026-09-27)
One directory per sub-agent run. Generated by `tools/rehearse/run-rehearsal.sh`;
full raw sessions (pi jsonl transcripts, complete workspaces) live in
`/tmp/rehearsal/runs/<run>/` and are intentionally not committed.
Per run:
| File | Content |
|------|---------|
| `agents-md.txt` | The exact AGENTS.md the agent worked against (shared design instructions + variant tool rules) |
| `prompt.txt` | The user prompt given to `pi -p` |
| `measure.txt` | Tokens (sum + per-usage breakdown), turn/tool-call counts, session span, wall clock, tool-name histogram (from `measure_run.sh`) |
| `verify.txt` | Deterministic gate result: diff size vs pristine, `mvn test` with the solution's test + approved file, structural checks (see `verify_outcome.sh`) |
| `diff.patch` | Final diff of the run workspace vs the pristine exercise (excludes test artifacts) |
| `pi-stdout.txt` | The agent's final report (summary + final diff as it saw it) |
| `time.txt` | `/usr/bin/time -v` output (wall clock, max RSS, exit status) |
| `mvn-verify-tail.log` | Tail of the mvn output for the gate run — only present for failures |
Run → variant/model map:
| Dir | Variant | Model |
|-----|---------|-------|
| `A` | CLI mandated | openrouter z-ai/glm-5.3-flash |
| `B` | plain pi tools | openrouter z-ai/glm-5.3-flash |
| `C` | CLI optional + intro| openrouter z-ai/glm-5.3-flash |
| `A+Q` | CLI mandated | entrim Qwen/Qwen3.6-35B-A3B |
| `B+Q` | plain pi tools | entrim Qwen/Qwen3.6-35B-A3B |
| `C+Q` | CLI optional + intro| entrim Qwen/Qwen3.6-35B-A3B |
Narrative, drift documentation, and conclusions: see
[`../rehearsal-findings.md`](../rehearsal-findings.md).