Your Name
b993f25c53
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the exercise; deterministic gate = solution test + approved file. Key results: mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both); weak model fails only under the mandate (A+Q), passes when free or optional. Drift documented with code: scorer.updateSelection left inside the extracted method (A+Q, C+Q). Harness in tools/rehearse; summary in rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
254 lines
14 KiB
Markdown
254 lines
14 KiB
Markdown
# Rehearsal of the exercise with coding agents — full summary
|
||
|
||
**Date:** 2026-09-27 · **Harness:** `tools/rehearse/` · **Raw per-run artifacts:** [`rehearsal-runs/`](./)
|
||
· **Exercise:** Split Loop with Lambda (IceCreamForecasting kata, `LongRangeForecasts`)
|
||
|
||
Six scripted sub-agent runs did the refactoring under different tool rules and
|
||
two models. Each run got a fresh workspace, a shared AGENTS.md (identical
|
||
design instructions), a 10-minute wall-clock cap, then measurement from the pi
|
||
session log and a deterministic gate: the reference solution's test + approved
|
||
file (`{Strawberry:12, Chocolate:12, Vanilla:17, }` with an `alwaysSunny`
|
||
lambda) dropped in and `mvn test` executed.
|
||
|
||
Variants:
|
||
|
||
- **A** — `java-refactor` CLI mandated (text edits forbidden except the
|
||
introduce-lambda step and a fallback for step 3)
|
||
- **B** — plain pi tools (read/bash/edit), CLI absent from PATH
|
||
- **C** — CLI available, use *optional*, with a 6-line intro to the relevant
|
||
subcommands
|
||
- **+Q** — same rules on entrim's Qwen/Qwen3.6-35B-A3B instead of
|
||
openrouter's z-ai/glm-5.3-flash
|
||
|
||
## Results
|
||
|
||
| Run | Model | Rules | Wall clock | Turns | Tool calls | Tokens (sum) | Peak context | Gate |
|
||
|-----|-------|-------|-----------:|------:|-----------:|-------------:|-------------:|------|
|
||
| A | glm-5.3-flash | CLI mandated | 4:06 | 27 | 27 (19 bash, 4 read, 4 edit) | 418,718 | 23.0 K | **PASS** |
|
||
| B | glm-5.3-flash | plain tools | 0:39 | 10 | 10 (5 bash, 1 read, 4 edit) | 62,196 | 7.6 K | **PASS** |
|
||
| C | glm-5.3-flash | CLI optional | 2:37 | 23 | 23 (15 bash, 4 read, 4 edit) | 150,805 | 13.5 K | **PASS** |
|
||
| A+Q | Qwen3.6-35B | CLI mandated | 2:07 | 27 | 28 (17 bash, 8 read, 3 edit) | 352,812 | 22.6 K | **FAIL** |
|
||
| B+Q | Qwen3.6-35B | plain tools | 1:13 | 17 | 17 (7 bash, 5 read, 5 edit) | 260,052 | 29.8 K | **PASS** |
|
||
| C+Q | Qwen3.6-35B | CLI optional | 1:12 | 17 | 17 (7 bash, 5 read, 5 edit) | 149,663 | 16.1 K | **PASS** |
|
||
|
||
Matrix (n=1 per cell — indicative, not statistical):
|
||
|
||
| | glm-5.3-flash (strong) | Qwen3.6-35B (weak) |
|
||
|---|---|---|
|
||
| **CLI mandated (A)** | PASS | **FAIL** |
|
||
| **CLI optional (C)** | PASS — used it surgically | PASS — ignored it entirely |
|
||
| **Plain tools (B)** | PASS | PASS |
|
||
|
||
Measurement notes: token sums are per-request usage added over the whole
|
||
session; entrim reports no prompt-cache reads, so +Q runs re-send full context
|
||
every turn (their sums are inflated relative to openrouter's cached runs —
|
||
e.g. B+Q's 260 K is 17 requests growing 2.3 K → 29.5 K; peak single request
|
||
~30 K, far under the 262 K limit). Peak context = largest single request.
|
||
Cross-provider, compare wall clock / turns / tool calls, not raw token sums.
|
||
|
||
## Headlines
|
||
|
||
1. **Mandating the tool tripled the strong model's cost** (A vs B: 4:06 vs
|
||
0:39, 419 K vs 62 K tokens) for an equally-passing result. The cost is not
|
||
the refactoring operations — it is the ceremony around them plus one
|
||
genuine coverage gap (see "Why the CLI regime needs so many turns").
|
||
2. **"Available, introduced, not mandated" is the safe default.** C never did
|
||
worse than the alternative on either model: the strong model used the CLI
|
||
surgically (extract dry-run → apply) and edited manually elsewhere; the
|
||
weak model ignored the CLI entirely and simply edited.
|
||
3. **The weak model failed only in the mandated regime.** A+Q produced broken
|
||
code and rationalized the failing tests as "expected". The same model
|
||
passed in B+Q and C+Q with materially correct end states.
|
||
|
||
## The drift, documented
|
||
|
||
The exercise's step 1 says: *introduce a lambda for the scorer-dependent code
|
||
and move its definition to the top of the method, leaving the invocation where
|
||
it is.* The scorer-dependent code is **both** scorer calls — `updateSelection()`
|
||
and `lookupWeather()`:
|
||
|
||
```java
|
||
// PRISTINE (exercise start) — LongRangeForecasts.java, lines 28–35
|
||
List<Boolean> expectedWeather = new ArrayList<>();
|
||
for (Instant interestingDate : interestingDates) {
|
||
scorer.updateSelection(); // ← scorer-dependent, void side effect
|
||
var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
|
||
long daysForward = ChronoUnit.DAYS.between(forecastDate, interestingDate);
|
||
Boolean lookupWeather = scorer.lookupWeather(daysForward); // ← scorer-dependent, value
|
||
expectedWeather.add(lookupWeather);
|
||
}
|
||
```
|
||
|
||
After a correct step 1+2, the extracted method is **static and pure** — its
|
||
only "dependency" is the lambda parameter, and the scorer calls live inside
|
||
the lambda definition in the no-arg overload (reference solution):
|
||
|
||
```java
|
||
// CORRECT — reference solution
|
||
public Map<IceCream, Integer> longRangeForecast() {
|
||
return longRangeForecast(daysForward -> {
|
||
scorer.updateSelection(); // scorer-dependent code, inside the lambda
|
||
return scorer.lookupWeather(daysForward);
|
||
});
|
||
}
|
||
|
||
private static List<Boolean> collectExpectedWeather(LongFunction<Boolean> dayOffsetToWeather) {
|
||
var expectedWeather = new ArrayList<Boolean>();
|
||
for (Instant interestingDate : DATES) {
|
||
long daysForward = ChronoUnit.DAYS.between(ANCHOR_DATE, interestingDate);
|
||
expectedWeather.add(dayOffsetToWeather.apply(daysForward)); // loop is scorer-free
|
||
}
|
||
return expectedWeather;
|
||
}
|
||
```
|
||
|
||
The **drift** (A+Q and C+Q both): lambda-izing only the value-producing call
|
||
and leaving `updateSelection()` as loop machinery, so extraction carried the
|
||
`scorer` field dependency into the new method:
|
||
|
||
```java
|
||
// DRIFTED — A+Q / C+Q end state
|
||
private List<Boolean> collectExpectedWeather(List<Instant> dates, LongFunction<Boolean> dayOffsetToWeather) {
|
||
List<Boolean> expectedWeather = new ArrayList<>();
|
||
for (Instant date : dates) {
|
||
scorer.updateSelection(); // ← LEFT BEHIND: method still depends on the field
|
||
var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
|
||
long daysForward = ChronoUnit.DAYS.between(forecastDate, date);
|
||
expectedWeather.add(dayOffsetToWeather.apply(daysForward));
|
||
}
|
||
return expectedWeather;
|
||
}
|
||
```
|
||
|
||
Consequences of the drift: `collectExpectedWeather` is neither static nor
|
||
pure, step 4 (moving it to another class) would be blocked, and the "unwanted
|
||
external dependency" was never actually severed — the split is cosmetic.
|
||
|
||
Why the gate still passed: the test injects its own pure lambda, and the stray
|
||
`updateSelection()` only mutates the static `Scorer.flavour` field, which
|
||
nothing in the forecast reads (`lookupWeather(daysForward)` ignores it; the
|
||
field's real consumer is the unrelated `DailyForecasts.salesForecast()`).
|
||
**PASS ≠ drift-free** — the deterministic gate verifies behavior, not design
|
||
intent. Catch it mechanically: `grep -n scorer <extracted-method>` must return
|
||
nothing, or simply try to make the extracted method `static` (it won't
|
||
compile if the field reference remains).
|
||
|
||
### The debrief aha hiding in this line
|
||
|
||
`updateSelection()` randomly re-rolls `Scorer.flavour` (see `Scorer.java:51–59`)
|
||
— state that belongs to the *daily* forecast, which is the only real consumer
|
||
(`DailyForecasts.java:28–30`, copy-paste residue into the long-range loop).
|
||
Once the loop is split and the lambda extracted, it becomes visible that
|
||
`updateSelection()`'s result is never used — **it is deletable outright**, and
|
||
the approved file does not change (listed in `solution/README.md`). The
|
||
refactoring does not just hide the dependency behind a lambda; it exposes that
|
||
one of the two scorer calls was dead all along.
|
||
|
||
## A+Q's failure anatomy (the mandated-CLI run)
|
||
|
||
Three stacked problems, all verified from the session log:
|
||
|
||
1. **Over-deletion (the actual gate failure).** Removing `String quarter`, it
|
||
deleted BOTH the dead Q1/Q4 branch AND the flavour-reset loop in one edit.
|
||
The reset loop is what populates the result map → forecast returns `{}` →
|
||
`expected {Strawberry:12, Chocolate:12, Vanilla:17} but was <{}>`. The
|
||
reference solution drops the Q1/Q4 branch too — but keeps the reset loop.
|
||
Silent failure: compiles and runs, just semantically empty.
|
||
2. **It never saw its final test behavior.** Its last in-run `mvn -q test`
|
||
(piped through `head -30`) hit a stale-bytecode artifact — incremental
|
||
compilation kept the old test class calling the removed
|
||
`longRangeForecast(String)` — throwing an `Error` at the call site whose
|
||
exception name scrolled off the pipe. It concluded: *"The test fails
|
||
because the old longRangeForecast(String) signature no longer exists —
|
||
which is expected per the task instructions."* The AGENTS.md's "tests may
|
||
fail — that is expected: the code must still compile" gave it a ready-made
|
||
rationalization to stop on red.
|
||
3. **The flaky pristine test camouflaged everything.** `Scorer` is random, so
|
||
mid-run assertion failures looked like expected noise; at one point its
|
||
`received.txt` even showed `{Strawberry:12, Chocolate:12, Vanilla:17}` — a
|
||
lucky random roll coincidentally equal to the solution's approved output,
|
||
sitting in the tree while the real behavior was `{}`.
|
||
|
||
## Why the mandated CLI regime needs so many turns (Run A trace)
|
||
|
||
Of A's 27 turns: ~8 discovery (reads, `--help` ×2, backup, coordinate
|
||
computation via `cat -n`), ~8 CLI attempts (2× `extract`, 2×
|
||
`decompose-conditional`, `remove-param`, `introduce-param` — the last four all
|
||
hunting in vain for a subcommand that can turn `longRangeForecast(String)` into
|
||
`longRangeForecast(LongFunction<Boolean>)` + overload), 3 manual edits, ~6
|
||
verify rounds (`mvn` ×5 + final diff). The point-based API forces a
|
||
read → compute-coordinates → invoke → confirm cycle per operation, and every
|
||
operation is a bash round-trip. Run B does it in 10 turns: 4 edits, 3 `mvn`,
|
||
1 read, 2 housekeeping.
|
||
|
||
C shows the fix: with the tool merely *available* plus a 6-line intro, the
|
||
agent used `extract` (dry-run → one rejected selection → corrected → apply)
|
||
and edited manually for steps 1 and 3, with no shopping detour — 23 turns,
|
||
2:37, and the closest-to-reference diff of all six runs (dates hoisted into
|
||
`collectExpectedWeather`).
|
||
|
||
## Tool limitations discovered (worth reflecting into the exercise)
|
||
|
||
1. **Lambdas are not first-class.** `introduce-param` cannot introduce a
|
||
lambda expression as a parameter: applied to the step-3 lambda it *inlined
|
||
the expression into the call site* — referencing the private `scorer`
|
||
field from outside the class, non-compiling (observed in A and A+Q; C and
|
||
C+Q avoided it after reading `--help`). The CLI's lambda support is the
|
||
reverse direction: method *references* are converted to lambdas when
|
||
rewriting call sites for parameter changes (golden-master approved), but a
|
||
lambda expression is never the *subject* of an operation. The original
|
||
cheat sheet's claim that "add lambda parameter" maps to
|
||
`introduce-param`/`change-method-signature` was wrong and is corrected in
|
||
`rehearse-with-agent.md`.
|
||
2. **`extract` rejects non-statement selections** with a diagnostic
|
||
("Selection does not contain any complete statements") — correct, safe
|
||
behavior; C recovered from its one mis-selection in a single retry.
|
||
3. **`--dry-run` preview → apply is mandatory hygiene.** Both A and A+Q had
|
||
their broken `introduce-param` results caught by the dry-run diff — the
|
||
only reason they didn't ship non-compiling code. It also doubles every CLI
|
||
round trip, part of the mandated-regime cost.
|
||
4. **Stale-bytecode mvn artifacts can mask real test results** when a public
|
||
signature is removed without `clean` (the old test class still runs and
|
||
throws `NoSuchMethodError`-style errors). Agents doing verification-by-mvn
|
||
should prefer reading the assertion message, not just the exit status.
|
||
|
||
## Prompt corrections fed back into the exercise
|
||
|
||
- The rehearsal prompt originally omitted the overload step; the reference
|
||
test (`longRangeForecast(LongFunction<Boolean>)`) cannot compile without it.
|
||
The prompt now includes step 3 with the **exact target signature** ("do not
|
||
keep the `String quarter` parameter") — vaguer wording produced
|
||
`longRangeForecast(String, LongFunction)` in run A's first attempt.
|
||
- AGENTS.md must say "tests may fail, must still compile" up front — otherwise
|
||
agents burn turns fighting the flaky pristine test (and A+Q ignored even
|
||
that, see above).
|
||
|
||
## Implications for the learning hour
|
||
|
||
- Expect the **tool-availability seam** to be where participants lose time:
|
||
the CLI covers extract/rename/move, but the *interesting* mechanics of this
|
||
kata (introduce lambda, slide statement, function-typed parameter) are
|
||
manual. Make that explicit up front rather than letting participants
|
||
discover it by trial.
|
||
- **Time estimate from this rehearsal:** a strong pair with plain editing
|
||
lands ~40 s of agent effort; with the CLI mandated, ~4 minutes. Humans will
|
||
be slower on both, but the *ratio* (tool ceremony ≈ 3–6× on a 55-line file)
|
||
is the honest expectation-setter.
|
||
- The drift (scorer left in the extracted method) is the canonical mistake to
|
||
seed the debrief: ask participants to run `grep scorer` on their extracted
|
||
method. It happened in 2 of 6 runs, on both models, only under manual-edit
|
||
conditions — humans will hit it too.
|
||
|
||
## Reproducing
|
||
|
||
```bash
|
||
bash tools/rehearse/setup.sh # build + install + pristine + smoke test
|
||
bash tools/rehearse/run-rehearsal.sh A B C # strong model, three regimes
|
||
bash tools/rehearse/run-rehearsal.sh A+Q B+Q C+Q # weak model (entrim Qwen3.6-35B-A3B)
|
||
```
|
||
|
||
Raw sessions (jsonl, full transcripts) live in `/tmp/rehearsal/runs/<run>/`
|
||
and are not committed; this directory carries the curated evidence
|
||
(`measure.txt`, `verify.txt`, `diff.patch`, the exact AGENTS.md and prompt per
|
||
run, the agent's final report, and the mvn failure tails).
|