learning-hour-maker/ideas/slice-with-lambda/exercise/rehearsal-findings.md

Rehearsal of the exercise with coding agents — full summary

Date: 2026-09-27 · Harness: tools/rehearse/ · Raw per-run artifacts: rehearsal-runs/ · Exercise: Slice with Lambda (IceCreamForecasting kata, LongRangeForecasts)

Six scripted sub-agent runs did the refactoring under different tool rules and two models. Each run got a fresh workspace, a shared AGENTS.md (identical design instructions), a 10-minute wall-clock cap, then measurement from the pi session log and a deterministic gate: the reference solution's test + approved file ({Strawberry:12, Chocolate:12, Vanilla:17, } with an alwaysSunny lambda) dropped in and mvn test executed.

Variants:

• A — java-refactor CLI mandated (text edits forbidden except the introduce-lambda step and a fallback for step 3)
• B — plain pi tools (read/bash/edit), CLI absent from PATH
• C — CLI available, use optional, with a 6-line intro to the relevant subcommands
• +Q — same rules on entrim's Qwen/Qwen3.6-35B-A3B instead of openrouter's z-ai/glm-5.3-flash

Results

Run	Model	Rules	Wall clock	Turns	Tool calls	Tokens (sum)	Peak context	Gate
A	glm-5.3-flash	CLI mandated	4:06	27	27 (19 bash, 4 read, 4 edit)	418,718	23.0 K	PASS
B	glm-5.3-flash	plain tools	0:39	10	10 (5 bash, 1 read, 4 edit)	62,196	7.6 K	PASS
C	glm-5.3-flash	CLI optional	2:37	23	23 (15 bash, 4 read, 4 edit)	150,805	13.5 K	PASS
A+Q	Qwen3.6-35B	CLI mandated	2:07	27	28 (17 bash, 8 read, 3 edit)	352,812	22.6 K	FAIL
B+Q	Qwen3.6-35B	plain tools	1:13	17	17 (7 bash, 5 read, 5 edit)	260,052	29.8 K	PASS
C+Q	Qwen3.6-35B	CLI optional	1:12	17	17 (7 bash, 5 read, 5 edit)	149,663	16.1 K	PASS

Matrix (n=1 per cell — indicative, not statistical):

	glm-5.3-flash (strong)	Qwen3.6-35B (weak)
CLI mandated (A)	PASS	FAIL
CLI optional (C)	PASS — used it surgically	PASS — ignored it entirely
Plain tools (B)	PASS	PASS

Measurement notes: token sums are per-request usage added over the whole session; entrim reports no prompt-cache reads, so +Q runs re-send full context every turn (their sums are inflated relative to openrouter's cached runs — e.g. B+Q's 260 K is 17 requests growing 2.3 K → 29.5 K; peak single request ~30 K, far under the 262 K limit). Peak context = largest single request. Cross-provider, compare wall clock / turns / tool calls, not raw token sums.

Headlines

1. Mandating the tool tripled the strong model's cost (A vs B: 4:06 vs 0:39, 419 K vs 62 K tokens) for an equally-passing result. The cost is not the refactoring operations — it is the ceremony around them plus one genuine coverage gap (see "Why the CLI regime needs so many turns").
2. "Available, introduced, not mandated" is the safe default. C never did worse than the alternative on either model: the strong model used the CLI surgically (extract dry-run → apply) and edited manually elsewhere; the weak model ignored the CLI entirely and simply edited.
3. The weak model failed only in the mandated regime. A+Q produced broken code and rationalized the failing tests as "expected". The same model passed in B+Q and C+Q with materially correct end states.

The drift, documented

The exercise's step 1 says: introduce a lambda for the scorer-dependent code and move its definition to the top of the method, leaving the invocation where it is. The scorer-dependent code is both scorer calls — updateSelection() and lookupWeather():

// PRISTINE (exercise start) — LongRangeForecasts.java, lines 28–35
List<Boolean> expectedWeather = new ArrayList<>();
for (Instant interestingDate : interestingDates) {
    scorer.updateSelection();                                  // ← scorer-dependent, void side effect
    var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
    long daysForward = ChronoUnit.DAYS.between(forecastDate, interestingDate);
    Boolean lookupWeather = scorer.lookupWeather(daysForward);  // ← scorer-dependent, value
    expectedWeather.add(lookupWeather);
}

After a correct step 1+2, the extracted method is static and pure — its only "dependency" is the lambda parameter, and the scorer calls live inside the lambda definition in the no-arg overload (reference solution):

// CORRECT — reference solution
public Map<IceCream, Integer> longRangeForecast() {
    return longRangeForecast(daysForward -> {
        scorer.updateSelection();          // scorer-dependent code, inside the lambda
        return scorer.lookupWeather(daysForward);
    });
}

private static List<Boolean> collectExpectedWeather(LongFunction<Boolean> dayOffsetToWeather) {
    var expectedWeather = new ArrayList<Boolean>();
    for (Instant interestingDate : DATES) {
        long daysForward = ChronoUnit.DAYS.between(ANCHOR_DATE, interestingDate);
        expectedWeather.add(dayOffsetToWeather.apply(daysForward));   // loop is scorer-free
    }
    return expectedWeather;
}

The drift (A+Q and C+Q both): lambda-izing only the value-producing call and leaving updateSelection() as loop machinery, so extraction carried the scorer field dependency into the new method:

// DRIFTED — A+Q / C+Q end state
private List<Boolean> collectExpectedWeather(List<Instant> dates, LongFunction<Boolean> dayOffsetToWeather) {
    List<Boolean> expectedWeather = new ArrayList<>();
    for (Instant date : dates) {
        scorer.updateSelection();          // ← LEFT BEHIND: method still depends on the field
        var forecastDate = Instant.parse("2023-04-26T14:00:00.000-07:00");
        long daysForward = ChronoUnit.DAYS.between(forecastDate, date);
        expectedWeather.add(dayOffsetToWeather.apply(daysForward));
    }
    return expectedWeather;
}

Consequences of the drift: collectExpectedWeather is neither static nor pure, step 4 (moving it to another class) would be blocked, and the "unwanted external dependency" was never actually severed — the split is cosmetic.

Why the gate still passed: the test injects its own pure lambda, and the stray updateSelection() only mutates the static Scorer.flavour field, which nothing in the forecast reads (lookupWeather(daysForward) ignores it; the field's real consumer is the unrelated DailyForecasts.salesForecast()). PASS ≠ drift-free — the deterministic gate verifies behavior, not design intent. Catch it mechanically: grep -n scorer <extracted-method> must return nothing, or simply try to make the extracted method static (it won't compile if the field reference remains).

The debrief aha hiding in this line

updateSelection() randomly re-rolls Scorer.flavour (see Scorer.java:51–59) — state that belongs to the daily forecast, which is the only real consumer (DailyForecasts.java:28–30, copy-paste residue into the long-range loop). Once the loop is split and the lambda extracted, it becomes visible that updateSelection()'s result is never used — it is deletable outright, and the approved file does not change (listed in solution/README.md). The refactoring does not just hide the dependency behind a lambda; it exposes that one of the two scorer calls was dead all along.

A+Q's failure anatomy (the mandated-CLI run)

Three stacked problems, all verified from the session log:

1. Over-deletion (the actual gate failure). Removing String quarter, it deleted BOTH the dead Q1/Q4 branch AND the flavour-reset loop in one edit. The reset loop is what populates the result map → forecast returns {} → expected {Strawberry:12, Chocolate:12, Vanilla:17} but was <{}>. The reference solution drops the Q1/Q4 branch too — but keeps the reset loop. Silent failure: compiles and runs, just semantically empty.
2. It never saw its final test behavior. Its last in-run mvn -q test (piped through head -30) hit a stale-bytecode artifact — incremental compilation kept the old test class calling the removed longRangeForecast(String) — throwing an Error at the call site whose exception name scrolled off the pipe. It concluded: "The test fails because the old longRangeForecast(String) signature no longer exists — which is expected per the task instructions." The AGENTS.md's "tests may fail — that is expected: the code must still compile" gave it a ready-made rationalization to stop on red.
3. The flaky pristine test camouflaged everything. Scorer is random, so mid-run assertion failures looked like expected noise; at one point its received.txt even showed {Strawberry:12, Chocolate:12, Vanilla:17} — a lucky random roll coincidentally equal to the solution's approved output, sitting in the tree while the real behavior was {}.

Why the mandated CLI regime needs so many turns (Run A trace)

Of A's 27 turns: ~8 discovery (reads, --help ×2, backup, coordinate computation via cat -n), ~8 CLI attempts (2× extract, 2× decompose-conditional, remove-param, introduce-param — the last four all hunting in vain for a subcommand that can turn longRangeForecast(String) into longRangeForecast(LongFunction<Boolean>) + overload), 3 manual edits, ~6 verify rounds (mvn ×5 + final diff). The point-based API forces a read → compute-coordinates → invoke → confirm cycle per operation, and every operation is a bash round-trip. Run B does it in 10 turns: 4 edits, 3 mvn, 1 read, 2 housekeeping.

C shows the fix: with the tool merely available plus a 6-line intro, the agent used extract (dry-run → one rejected selection → corrected → apply) and edited manually for steps 1 and 3, with no shopping detour — 23 turns, 2:37, and the closest-to-reference diff of all six runs (dates hoisted into collectExpectedWeather).

Tool limitations discovered (worth reflecting into the exercise)

1. Lambdas are not first-class. introduce-param cannot introduce a lambda expression as a parameter: applied to the step-3 lambda it inlined the expression into the call site — referencing the private scorer field from outside the class, non-compiling (observed in A and A+Q; C and C+Q avoided it after reading --help). The CLI's lambda support is the reverse direction: method references are converted to lambdas when rewriting call sites for parameter changes (golden-master approved), but a lambda expression is never the subject of an operation. The original cheat sheet's claim that "add lambda parameter" maps to introduce-param/change-method-signature was wrong and is corrected in rehearse-with-agent.md.
2. extract rejects non-statement selections with a diagnostic ("Selection does not contain any complete statements") — correct, safe behavior; C recovered from its one mis-selection in a single retry.
3. --dry-run preview → apply is mandatory hygiene. Both A and A+Q had their broken introduce-param results caught by the dry-run diff — the only reason they didn't ship non-compiling code. It also doubles every CLI round trip, part of the mandated-regime cost.
4. Stale-bytecode mvn artifacts can mask real test results when a public signature is removed without clean (the old test class still runs and throws NoSuchMethodError-style errors). Agents doing verification-by-mvn should prefer reading the assertion message, not just the exit status.

Prompt corrections fed back into the exercise

• The rehearsal prompt originally omitted the overload step; the reference test (longRangeForecast(LongFunction<Boolean>)) cannot compile without it. The prompt now includes step 3 with the exact target signature ("do not keep the String quarter parameter") — vaguer wording produced longRangeForecast(String, LongFunction) in run A's first attempt.
• AGENTS.md must say "tests may fail, must still compile" up front — otherwise agents burn turns fighting the flaky pristine test (and A+Q ignored even that, see above).

Implications for the learning hour

• Expect the tool-availability seam to be where participants lose time: the CLI covers extract/rename/move, but the interesting mechanics of this kata (introduce lambda, slide statement, function-typed parameter) are manual. Make that explicit up front rather than letting participants discover it by trial.
• Time estimate from this rehearsal: a strong pair with plain editing lands ~40 s of agent effort; with the CLI mandated, ~4 minutes. Humans will be slower on both, but the ratio (tool ceremony ≈ 3–6× on a 55-line file) is the honest expectation-setter.
• The drift (scorer left in the extracted method) is the canonical mistake to seed the debrief: ask participants to run grep scorer on their extracted method. It happened in 2 of 6 runs, on both models, only under manual-edit conditions — humans will hit it too.

Reproducing

bash tools/rehearse/setup.sh                # build + install + pristine + smoke test
bash tools/rehearse/run-rehearsal.sh A B C  # strong model, three regimes
bash tools/rehearse/run-rehearsal.sh A+Q B+Q C+Q  # weak model (entrim Qwen3.6-35B-A3B)

Raw sessions (jsonl, full transcripts) live in /tmp/rehearsal/runs/<run>/ and are not committed; this directory carries the curated evidence (measure.txt, verify.txt, diff.patch, the exact AGENTS.md and prompt per run, the agent's final report, and the mvn failure tails).