learning-hour-maker/ideas/slice-with-lambda/exercise/rehearsal-findings.md

# Rehearsal of the exercise with coding agents — full summary

**Date:** 2026-09-27 · **Harness:** `tools/rehearse/` · **Raw per-run artifacts:** [`rehearsal-runs/`](./)
· **Exercise:** Slice with Lambda (IceCreamForecasting kata, `LongRangeForecasts`)

Six scripted sub-agent runs did the refactoring under different tool rules and
two models. Each run got a fresh workspace, a shared AGENTS.md (identical
design instructions), a 10-minute wall-clock cap, then measurement from the pi
session log and a deterministic gate: the reference solution's test + approved
file (`{Strawberry:12, Chocolate:12, Vanilla:17, }` with an `alwaysSunny`
lambda) dropped in and `mvn test` executed.

Variants:

- **A** — `java-refactor` CLI mandated (text edits forbidden except the
  introduce-lambda step and a fallback for step 3)
- **B** — plain pi tools (read/bash/edit), CLI absent from PATH
- **C** — CLI available, use *optional*, with a 6-line intro to the relevant
  subcommands
- **+Q** — same rules on entrim's Qwen/Qwen3.6-35B-A3B instead of
  openrouter's z-ai/glm-5.3-flash

## Results

| Run | Model | Rules | Wall clock | Turns | Tool calls | Tokens (sum) | Peak context | Gate |
|-----|-------|-------|-----------:|------:|-----------:|-------------:|-------------:|------|
| A   | glm-5.3-flash | CLI mandated   | 4:06 | 27 | 27 (19 bash, 4 read, 4 edit) | 418,718 | 23.0 K | **PASS** |
| B   | glm-5.3-flash | plain tools    | 0:39 | 10 | 10 (5 bash, 1 read, 4 edit)  | 62,196  | 7.6 K  | **PASS** |
| C   | glm-5.3-flash | CLI optional   | 2:37 | 23 | 23 (15 bash, 4 read, 4 edit) | 150,805 | 13.5 K | **PASS** |
| A+Q | Qwen3.6-35B   | CLI mandated   | 2:07 | 27 | 28 (17 bash, 8 read, 3 edit) | 352,812 | 22.6 K | **FAIL** |
| B+Q | Qwen3.6-35B   | plain tools    | 1:13 | 17 | 17 (7 bash, 5 read, 5 edit)  | 260,052 | 29.8 K | **PASS** |
| C+Q | Qwen3.6-35B   | CLI optional   | 1:12 | 17 | 17 (7 bash, 5 read, 5 edit)  | 149,663 | 16.1 K | **PASS** |

Matrix (n=1 per cell — indicative, not statistical):

| | glm-5.3-flash (strong) | Qwen3.6-35B (weak) |
|---|---|---|
| **CLI mandated (A)** | PASS | **FAIL** |
| **CLI optional (C)** | PASS — used it surgically | PASS — ignored it entirely |
| **Plain tools (B)** | PASS | PASS |

Measurement notes: token sums are per-request usage added over the whole
session; entrim reports no prompt-cache reads, so +Q runs re-send full context
every turn (their sums are inflated relative to openrouter's cached runs —
e.g. B+Q's 260 K is 17 requests growing 2.3 K → 29.5 K; peak single request
~30 K, far under the 262 K limit). Peak context = largest single request.
Cross-provider, compare wall clock / turns / tool calls, not raw token sums.

## Headlines

1. **Mandating the tool tripled the strong model's cost** (A vs B: 4:06 vs
   0:39, 419 K vs 62 K tokens) for an equally-passing result. The cost is not
   the refactoring operations — it is the ceremony around them plus one
   genuine coverage gap (see "Why the CLI regime needs so many turns").
2. **"Available, introduced, not mandated" is the safe default.** C never did
   worse than the alternative on either model: the strong model used the CLI
   surgically (extract dry-run → apply) and edited manually elsewhere; the
   weak model ignored the CLI entirely and simply edited.
3. **The weak model failed only in the mandated regime.** A+Q produced broken
   code and rationalized the failing tests as "expected". The same model
   passed in B+Q and C+Q with materially correct end states.

## The drift, documented

The exercise's step 1 says: *introduce a lambda for the scorer-dependent code
and move its definition to the top of the method, leaving the invocation where
it is.* The scorer-dependent code is **both** scorer calls — `updateSelection()`
and `lookupWeather()`:

```java
// PRISTINE (exercise start) — LongRangeForecasts.java, lines 28–35
List<Boolean> expectedWeather = new ArrayList<>();
for (Instant interestingDate : interestingDates) {
    scorer.updateSelection();                                  // ← scorer-dependent, void side effect
    var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
    long daysForward = ChronoUnit.DAYS.between(forecastDate, interestingDate);
    Boolean lookupWeather = scorer.lookupWeather(daysForward);  // ← scorer-dependent, value
    expectedWeather.add(lookupWeather);
}
```

After a correct step 1+2, the extracted method is **static and pure** — its
only "dependency" is the lambda parameter, and the scorer calls live inside
the lambda definition in the no-arg overload (reference solution):

```java
// CORRECT — reference solution
public Map<IceCream, Integer> longRangeForecast() {
    return longRangeForecast(daysForward -> {
        scorer.updateSelection();          // scorer-dependent code, inside the lambda
        return scorer.lookupWeather(daysForward);
    });
}

private static List<Boolean> collectExpectedWeather(LongFunction<Boolean> dayOffsetToWeather) {
    var expectedWeather = new ArrayList<Boolean>();
    for (Instant interestingDate : DATES) {
        long daysForward = ChronoUnit.DAYS.between(ANCHOR_DATE, interestingDate);
        expectedWeather.add(dayOffsetToWeather.apply(daysForward));   // loop is scorer-free
    }
    return expectedWeather;
}
```

The **drift** (A+Q and C+Q both): lambda-izing only the value-producing call
and leaving `updateSelection()` as loop machinery, so extraction carried the
`scorer` field dependency into the new method:

```java
// DRIFTED — A+Q / C+Q end state
private List<Boolean> collectExpectedWeather(List<Instant> dates, LongFunction<Boolean> dayOffsetToWeather) {
    List<Boolean> expectedWeather = new ArrayList<>();
    for (Instant date : dates) {
        scorer.updateSelection();          // ← LEFT BEHIND: method still depends on the field
        var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
        long daysForward = ChronoUnit.DAYS.between(forecastDate, date);
        expectedWeather.add(dayOffsetToWeather.apply(daysForward));
    }
    return expectedWeather;
}
```

Consequences of the drift: `collectExpectedWeather` is neither static nor
pure, step 4 (moving it to another class) would be blocked, and the "unwanted
external dependency" was never actually severed — the split is cosmetic.

Why the gate still passed: the test injects its own pure lambda, and the stray
`updateSelection()` only mutates the static `Scorer.flavour` field, which
nothing in the forecast reads (`lookupWeather(daysForward)` ignores it; the
field's real consumer is the unrelated `DailyForecasts.salesForecast()`).
**PASS ≠ drift-free** — the deterministic gate verifies behavior, not design
intent. Catch it mechanically: `grep -n scorer <extracted-method>` must return
nothing, or simply try to make the extracted method `static` (it won't
compile if the field reference remains).

### The debrief aha hiding in this line

`updateSelection()` randomly re-rolls `Scorer.flavour` (see `Scorer.java:51–59`)
— state that belongs to the *daily* forecast, which is the only real consumer
(`DailyForecasts.java:28–30`, copy-paste residue into the long-range loop).
Once the loop is split and the lambda extracted, it becomes visible that
`updateSelection()`'s result is never used — **it is deletable outright**, and
the approved file does not change (listed in `solution/README.md`). The
refactoring does not just hide the dependency behind a lambda; it exposes that
one of the two scorer calls was dead all along.

## A+Q's failure anatomy (the mandated-CLI run)

Three stacked problems, all verified from the session log:

1. **Over-deletion (the actual gate failure).** Removing `String quarter`, it
   deleted BOTH the dead Q1/Q4 branch AND the flavour-reset loop in one edit.
   The reset loop is what populates the result map → forecast returns `{}` →
   `expected {Strawberry:12, Chocolate:12, Vanilla:17} but was <{}>`. The
   reference solution drops the Q1/Q4 branch too — but keeps the reset loop.
   Silent failure: compiles and runs, just semantically empty.
2. **It never saw its final test behavior.** Its last in-run `mvn -q test`
   (piped through `head -30`) hit a stale-bytecode artifact — incremental
   compilation kept the old test class calling the removed
   `longRangeForecast(String)` — throwing an `Error` at the call site whose
   exception name scrolled off the pipe. It concluded: *"The test fails
   because the old longRangeForecast(String) signature no longer exists —
   which is expected per the task instructions."* The AGENTS.md's "tests may
   fail — that is expected: the code must still compile" gave it a ready-made
   rationalization to stop on red.
3. **The flaky pristine test camouflaged everything.** `Scorer` is random, so
   mid-run assertion failures looked like expected noise; at one point its
   `received.txt` even showed `{Strawberry:12, Chocolate:12, Vanilla:17}` — a
   lucky random roll coincidentally equal to the solution's approved output,
   sitting in the tree while the real behavior was `{}`.

## Why the mandated CLI regime needs so many turns (Run A trace)

Of A's 27 turns: ~8 discovery (reads, `--help` ×2, backup, coordinate
computation via `cat -n`), ~8 CLI attempts (2× `extract`, 2×
`decompose-conditional`, `remove-param`, `introduce-param` — the last four all
hunting in vain for a subcommand that can turn `longRangeForecast(String)` into
`longRangeForecast(LongFunction<Boolean>)` + overload), 3 manual edits, ~6
verify rounds (`mvn` ×5 + final diff). The point-based API forces a
read → compute-coordinates → invoke → confirm cycle per operation, and every
operation is a bash round-trip. Run B does it in 10 turns: 4 edits, 3 `mvn`,
1 read, 2 housekeeping.

C shows the fix: with the tool merely *available* plus a 6-line intro, the
agent used `extract` (dry-run → one rejected selection → corrected → apply)
and edited manually for steps 1 and 3, with no shopping detour — 23 turns,
2:37, and the closest-to-reference diff of all six runs (dates hoisted into
`collectExpectedWeather`).

## Tool limitations discovered (worth reflecting into the exercise)

1. **Lambdas are not first-class.** `introduce-param` cannot introduce a
   lambda expression as a parameter: applied to the step-3 lambda it *inlined
   the expression into the call site* — referencing the private `scorer`
   field from outside the class, non-compiling (observed in A and A+Q; C and
   C+Q avoided it after reading `--help`). The CLI's lambda support is the
   reverse direction: method *references* are converted to lambdas when
   rewriting call sites for parameter changes (golden-master approved), but a
   lambda expression is never the *subject* of an operation. The original
   cheat sheet's claim that "add lambda parameter" maps to
   `introduce-param`/`change-method-signature` was wrong and is corrected in
   `rehearse-with-agent.md`.
2. **`extract` rejects non-statement selections** with a diagnostic
   ("Selection does not contain any complete statements") — correct, safe
   behavior; C recovered from its one mis-selection in a single retry.
3. **`--dry-run` preview → apply is mandatory hygiene.** Both A and A+Q had
   their broken `introduce-param` results caught by the dry-run diff — the
   only reason they didn't ship non-compiling code. It also doubles every CLI
   round trip, part of the mandated-regime cost.
4. **Stale-bytecode mvn artifacts can mask real test results** when a public
   signature is removed without `clean` (the old test class still runs and
   throws `NoSuchMethodError`-style errors). Agents doing verification-by-mvn
   should prefer reading the assertion message, not just the exit status.

## Prompt corrections fed back into the exercise

- The rehearsal prompt originally omitted the overload step; the reference
  test (`longRangeForecast(LongFunction<Boolean>)`) cannot compile without it.
  The prompt now includes step 3 with the **exact target signature** ("do not
  keep the `String quarter` parameter") — vaguer wording produced
  `longRangeForecast(String, LongFunction)` in run A's first attempt.
- AGENTS.md must say "tests may fail, must still compile" up front — otherwise
  agents burn turns fighting the flaky pristine test (and A+Q ignored even
  that, see above).

## Implications for the learning hour

- Expect the **tool-availability seam** to be where participants lose time:
  the CLI covers extract/rename/move, but the *interesting* mechanics of this
  kata (introduce lambda, slide statement, function-typed parameter) are
  manual. Make that explicit up front rather than letting participants
  discover it by trial.
- **Time estimate from this rehearsal:** a strong pair with plain editing
  lands ~40 s of agent effort; with the CLI mandated, ~4 minutes. Humans will
  be slower on both, but the *ratio* (tool ceremony ≈ 3–6× on a 55-line file)
  is the honest expectation-setter.
- The drift (scorer left in the extracted method) is the canonical mistake to
  seed the debrief: ask participants to run `grep scorer` on their extracted
  method. It happened in 2 of 6 runs, on both models, only under manual-edit
  conditions — humans will hit it too.

## Reproducing

```bash
bash tools/rehearse/setup.sh                # build + install + pristine + smoke test
bash tools/rehearse/run-rehearsal.sh A B C  # strong model, three regimes
bash tools/rehearse/run-rehearsal.sh A+Q B+Q C+Q  # weak model (entrim Qwen3.6-35B-A3B)
```

Raw sessions (jsonl, full transcripts) live in `/tmp/rehearsal/runs/<run>/`
and are not committed; this directory carries the curated evidence
(`measure.txt`, `verify.txt`, `diff.patch`, the exact AGENTS.md and prompt per
run, the agent's final report, and the mvn failure tails).