# Rehearsal of the exercise with coding agents — full summary **Date:** 2026-09-27 · **Harness:** `tools/rehearse/` · **Raw per-run artifacts:** [`rehearsal-runs/`](./) · **Exercise:** Slice with Lambda (IceCreamForecasting kata, `LongRangeForecasts`) Six scripted sub-agent runs did the refactoring under different tool rules and two models. Each run got a fresh workspace, a shared AGENTS.md (identical design instructions), a 10-minute wall-clock cap, then measurement from the pi session log and a deterministic gate: the reference solution's test + approved file (`{Strawberry:12, Chocolate:12, Vanilla:17, }` with an `alwaysSunny` lambda) dropped in and `mvn test` executed. Variants: - **A** — `java-refactor` CLI mandated (text edits forbidden except the introduce-lambda step and a fallback for step 3) - **B** — plain pi tools (read/bash/edit), CLI absent from PATH - **C** — CLI available, use *optional*, with a 6-line intro to the relevant subcommands - **+Q** — same rules on entrim's Qwen/Qwen3.6-35B-A3B instead of openrouter's z-ai/glm-5.3-flash ## Results | Run | Model | Rules | Wall clock | Turns | Tool calls | Tokens (sum) | Peak context | Gate | |-----|-------|-------|-----------:|------:|-----------:|-------------:|-------------:|------| | A | glm-5.3-flash | CLI mandated | 4:06 | 27 | 27 (19 bash, 4 read, 4 edit) | 418,718 | 23.0 K | **PASS** | | B | glm-5.3-flash | plain tools | 0:39 | 10 | 10 (5 bash, 1 read, 4 edit) | 62,196 | 7.6 K | **PASS** | | C | glm-5.3-flash | CLI optional | 2:37 | 23 | 23 (15 bash, 4 read, 4 edit) | 150,805 | 13.5 K | **PASS** | | A+Q | Qwen3.6-35B | CLI mandated | 2:07 | 27 | 28 (17 bash, 8 read, 3 edit) | 352,812 | 22.6 K | **FAIL** | | B+Q | Qwen3.6-35B | plain tools | 1:13 | 17 | 17 (7 bash, 5 read, 5 edit) | 260,052 | 29.8 K | **PASS** | | C+Q | Qwen3.6-35B | CLI optional | 1:12 | 17 | 17 (7 bash, 5 read, 5 edit) | 149,663 | 16.1 K | **PASS** | Matrix (n=1 per cell — indicative, not statistical): | | glm-5.3-flash (strong) | Qwen3.6-35B (weak) | |---|---|---| | **CLI mandated (A)** | PASS | **FAIL** | | **CLI optional (C)** | PASS — used it surgically | PASS — ignored it entirely | | **Plain tools (B)** | PASS | PASS | Measurement notes: token sums are per-request usage added over the whole session; entrim reports no prompt-cache reads, so +Q runs re-send full context every turn (their sums are inflated relative to openrouter's cached runs — e.g. B+Q's 260 K is 17 requests growing 2.3 K → 29.5 K; peak single request ~30 K, far under the 262 K limit). Peak context = largest single request. Cross-provider, compare wall clock / turns / tool calls, not raw token sums. ## Headlines 1. **Mandating the tool tripled the strong model's cost** (A vs B: 4:06 vs 0:39, 419 K vs 62 K tokens) for an equally-passing result. The cost is not the refactoring operations — it is the ceremony around them plus one genuine coverage gap (see "Why the CLI regime needs so many turns"). 2. **"Available, introduced, not mandated" is the safe default.** C never did worse than the alternative on either model: the strong model used the CLI surgically (extract dry-run → apply) and edited manually elsewhere; the weak model ignored the CLI entirely and simply edited. 3. **The weak model failed only in the mandated regime.** A+Q produced broken code and rationalized the failing tests as "expected". The same model passed in B+Q and C+Q with materially correct end states. ## The drift, documented The exercise's step 1 says: *introduce a lambda for the scorer-dependent code and move its definition to the top of the method, leaving the invocation where it is.* The scorer-dependent code is **both** scorer calls — `updateSelection()` and `lookupWeather()`: ```java // PRISTINE (exercise start) — LongRangeForecasts.java, lines 28–35 List expectedWeather = new ArrayList<>(); for (Instant interestingDate : interestingDates) { scorer.updateSelection(); // ← scorer-dependent, void side effect var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00"); long daysForward = ChronoUnit.DAYS.between(forecastDate, interestingDate); Boolean lookupWeather = scorer.lookupWeather(daysForward); // ← scorer-dependent, value expectedWeather.add(lookupWeather); } ``` After a correct step 1+2, the extracted method is **static and pure** — its only "dependency" is the lambda parameter, and the scorer calls live inside the lambda definition in the no-arg overload (reference solution): ```java // CORRECT — reference solution public Map longRangeForecast() { return longRangeForecast(daysForward -> { scorer.updateSelection(); // scorer-dependent code, inside the lambda return scorer.lookupWeather(daysForward); }); } private static List collectExpectedWeather(LongFunction dayOffsetToWeather) { var expectedWeather = new ArrayList(); for (Instant interestingDate : DATES) { long daysForward = ChronoUnit.DAYS.between(ANCHOR_DATE, interestingDate); expectedWeather.add(dayOffsetToWeather.apply(daysForward)); // loop is scorer-free } return expectedWeather; } ``` The **drift** (A+Q and C+Q both): lambda-izing only the value-producing call and leaving `updateSelection()` as loop machinery, so extraction carried the `scorer` field dependency into the new method: ```java // DRIFTED — A+Q / C+Q end state private List collectExpectedWeather(List dates, LongFunction dayOffsetToWeather) { List expectedWeather = new ArrayList<>(); for (Instant date : dates) { scorer.updateSelection(); // ← LEFT BEHIND: method still depends on the field var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00"); long daysForward = ChronoUnit.DAYS.between(forecastDate, date); expectedWeather.add(dayOffsetToWeather.apply(daysForward)); } return expectedWeather; } ``` Consequences of the drift: `collectExpectedWeather` is neither static nor pure, step 4 (moving it to another class) would be blocked, and the "unwanted external dependency" was never actually severed — the split is cosmetic. Why the gate still passed: the test injects its own pure lambda, and the stray `updateSelection()` only mutates the static `Scorer.flavour` field, which nothing in the forecast reads (`lookupWeather(daysForward)` ignores it; the field's real consumer is the unrelated `DailyForecasts.salesForecast()`). **PASS ≠ drift-free** — the deterministic gate verifies behavior, not design intent. Catch it mechanically: `grep -n scorer ` must return nothing, or simply try to make the extracted method `static` (it won't compile if the field reference remains). ### The debrief aha hiding in this line `updateSelection()` randomly re-rolls `Scorer.flavour` (see `Scorer.java:51–59`) — state that belongs to the *daily* forecast, which is the only real consumer (`DailyForecasts.java:28–30`, copy-paste residue into the long-range loop). Once the loop is split and the lambda extracted, it becomes visible that `updateSelection()`'s result is never used — **it is deletable outright**, and the approved file does not change (listed in `solution/README.md`). The refactoring does not just hide the dependency behind a lambda; it exposes that one of the two scorer calls was dead all along. ## A+Q's failure anatomy (the mandated-CLI run) Three stacked problems, all verified from the session log: 1. **Over-deletion (the actual gate failure).** Removing `String quarter`, it deleted BOTH the dead Q1/Q4 branch AND the flavour-reset loop in one edit. The reset loop is what populates the result map → forecast returns `{}` → `expected {Strawberry:12, Chocolate:12, Vanilla:17} but was <{}>`. The reference solution drops the Q1/Q4 branch too — but keeps the reset loop. Silent failure: compiles and runs, just semantically empty. 2. **It never saw its final test behavior.** Its last in-run `mvn -q test` (piped through `head -30`) hit a stale-bytecode artifact — incremental compilation kept the old test class calling the removed `longRangeForecast(String)` — throwing an `Error` at the call site whose exception name scrolled off the pipe. It concluded: *"The test fails because the old longRangeForecast(String) signature no longer exists — which is expected per the task instructions."* The AGENTS.md's "tests may fail — that is expected: the code must still compile" gave it a ready-made rationalization to stop on red. 3. **The flaky pristine test camouflaged everything.** `Scorer` is random, so mid-run assertion failures looked like expected noise; at one point its `received.txt` even showed `{Strawberry:12, Chocolate:12, Vanilla:17}` — a lucky random roll coincidentally equal to the solution's approved output, sitting in the tree while the real behavior was `{}`. ## Why the mandated CLI regime needs so many turns (Run A trace) Of A's 27 turns: ~8 discovery (reads, `--help` ×2, backup, coordinate computation via `cat -n`), ~8 CLI attempts (2× `extract`, 2× `decompose-conditional`, `remove-param`, `introduce-param` — the last four all hunting in vain for a subcommand that can turn `longRangeForecast(String)` into `longRangeForecast(LongFunction)` + overload), 3 manual edits, ~6 verify rounds (`mvn` ×5 + final diff). The point-based API forces a read → compute-coordinates → invoke → confirm cycle per operation, and every operation is a bash round-trip. Run B does it in 10 turns: 4 edits, 3 `mvn`, 1 read, 2 housekeeping. C shows the fix: with the tool merely *available* plus a 6-line intro, the agent used `extract` (dry-run → one rejected selection → corrected → apply) and edited manually for steps 1 and 3, with no shopping detour — 23 turns, 2:37, and the closest-to-reference diff of all six runs (dates hoisted into `collectExpectedWeather`). ## Tool limitations discovered (worth reflecting into the exercise) 1. **Lambdas are not first-class.** `introduce-param` cannot introduce a lambda expression as a parameter: applied to the step-3 lambda it *inlined the expression into the call site* — referencing the private `scorer` field from outside the class, non-compiling (observed in A and A+Q; C and C+Q avoided it after reading `--help`). The CLI's lambda support is the reverse direction: method *references* are converted to lambdas when rewriting call sites for parameter changes (golden-master approved), but a lambda expression is never the *subject* of an operation. The original cheat sheet's claim that "add lambda parameter" maps to `introduce-param`/`change-method-signature` was wrong and is corrected in `rehearse-with-agent.md`. 2. **`extract` rejects non-statement selections** with a diagnostic ("Selection does not contain any complete statements") — correct, safe behavior; C recovered from its one mis-selection in a single retry. 3. **`--dry-run` preview → apply is mandatory hygiene.** Both A and A+Q had their broken `introduce-param` results caught by the dry-run diff — the only reason they didn't ship non-compiling code. It also doubles every CLI round trip, part of the mandated-regime cost. 4. **Stale-bytecode mvn artifacts can mask real test results** when a public signature is removed without `clean` (the old test class still runs and throws `NoSuchMethodError`-style errors). Agents doing verification-by-mvn should prefer reading the assertion message, not just the exit status. ## Prompt corrections fed back into the exercise - The rehearsal prompt originally omitted the overload step; the reference test (`longRangeForecast(LongFunction)`) cannot compile without it. The prompt now includes step 3 with the **exact target signature** ("do not keep the `String quarter` parameter") — vaguer wording produced `longRangeForecast(String, LongFunction)` in run A's first attempt. - AGENTS.md must say "tests may fail, must still compile" up front — otherwise agents burn turns fighting the flaky pristine test (and A+Q ignored even that, see above). ## Implications for the learning hour - Expect the **tool-availability seam** to be where participants lose time: the CLI covers extract/rename/move, but the *interesting* mechanics of this kata (introduce lambda, slide statement, function-typed parameter) are manual. Make that explicit up front rather than letting participants discover it by trial. - **Time estimate from this rehearsal:** a strong pair with plain editing lands ~40 s of agent effort; with the CLI mandated, ~4 minutes. Humans will be slower on both, but the *ratio* (tool ceremony ≈ 3–6× on a 55-line file) is the honest expectation-setter. - The drift (scorer left in the extracted method) is the canonical mistake to seed the debrief: ask participants to run `grep scorer` on their extracted method. It happened in 2 of 6 runs, on both models, only under manual-edit conditions — humans will hit it too. ## Reproducing ```bash bash tools/rehearse/setup.sh # build + install + pristine + smoke test bash tools/rehearse/run-rehearsal.sh A B C # strong model, three regimes bash tools/rehearse/run-rehearsal.sh A+Q B+Q C+Q # weak model (entrim Qwen3.6-35B-A3B) ``` Raw sessions (jsonl, full transcripts) live in `/tmp/rehearsal/runs//` and are not committed; this directory carries the curated evidence (`measure.txt`, `verify.txt`, `diff.patch`, the exact AGENTS.md and prompt per run, the agent's final report, and the mvn failure tails).