Your Name
b993f25c53
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the exercise; deterministic gate = solution test + approved file. Key results: mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both); weak model fails only under the mandate (A+Q), passes when free or optional. Drift documented with code: scorer.updateSelection left inside the extracted method (A+Q, C+Q). Harness in tools/rehearse; summary in rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
81 lines
4.0 KiB
Markdown
81 lines
4.0 KiB
Markdown
# Rehearsing the exercise with a coding agent + java-refactoring-mcp
|
|
|
|
How Willem can rehearse the exercise (and test whether an agent can do it)
|
|
using [java-refactoring-mcp](https://github.com/pfichtner/java-refactoring-mcp)'s
|
|
CLI as the semantic refactoring backend. One-time setup, then a repeatable
|
|
session script.
|
|
|
|
## One-time setup
|
|
|
|
```bash
|
|
# 1. Build the tool (needs Java 21 + Maven 3.8+)
|
|
git clone https://github.com/pfichtner/java-refactoring-mcp
|
|
cd java-refactoring-mcp && mvn package -DskipTests
|
|
# → refactor-mcp/target/refactor-mcp-*-fat.jar
|
|
|
|
# 2. Install the CLI wrapper (see that repo's README, "CLI (optional)"):
|
|
# gives you the `java-refactor` command.
|
|
|
|
# 3. Sanctioned dirs for the MCP server (it refuses paths outside this list)
|
|
# — check the repo docs for the exact env var / flag.
|
|
|
|
# 4. Copy the exercise (NOT the solution folder) to a scratch workspace
|
|
cp -r exercise/java ~/scratch/prefactored-longrange
|
|
```
|
|
|
|
## Session script
|
|
|
|
Run the agent in `~/scratch/prefactored-longrange` with a prompt like:
|
|
|
|
> You may only change code by invoking `java-refactor` CLI commands
|
|
> (`java-refactor --help` lists them). Never edit `.java` files with text
|
|
> edits, except lines marked by the CLI as needing manual work.
|
|
> After each refactoring, run `mvn -q test` (it may fail — that is expected;
|
|
> the dependency on `Scorer` makes the method untestable — but it must
|
|
> still *compile*).
|
|
>
|
|
> Goal: in `LongRangeForecasts.longRangeForecast()`, the loop over `DATES`
|
|
> collects weather by calling `scorer` (an unwanted external dependency).
|
|
>
|
|
> 1. Introduce a lambda for the scorer-dependent code and move its
|
|
> definition to the top of the method, leaving the invocation where it is.
|
|
> (No tool supports this — do it as a minimal manual edit.)
|
|
> 2. Extract the weather-collection loop into a new method
|
|
> `collectExpectedWeather` that takes the lambda as a parameter
|
|
> (suggested type `LongFunction<Boolean>`).
|
|
> 3. Stop. Report the final diff.
|
|
|
|
Then compare the agent's end state against `exercise/solution/` (or diff
|
|
against it directly) and run `mvn test` with the solution's test + approved
|
|
file dropped in: it must pass deterministically.
|
|
|
|
## What this rehearsal tells you
|
|
|
|
- The mechanics-with-names prompt (introduce lambda / slide statement /
|
|
extract method + introduce parameter) is unambiguous enough that an agent
|
|
with weak tool-calling (qwen 35B) can follow it — if it can, a human
|
|
pair can too.
|
|
- Which steps the agent drifts on (usually step 1, where it tends to skip
|
|
the lambda and just extract a method that closes over `scorer` — the
|
|
exact mistake participants make). Collect these failure modes: they are
|
|
gold for the concept slide and the debrief.
|
|
- Time estimate for the exercise: run it twice, once tool-only, once with
|
|
the manual lambda edit. The gap between the two is the "no tool support"
|
|
friction participants will feel.
|
|
|
|
## Tool coverage cheat sheet
|
|
|
|
| Mechanic | Tool support |
|
|
|---|---|
|
|
| introduce lambda | none — manual edit |
|
|
| slide statement | none — manual edit |
|
|
| extract method | `java-refactor extract` |
|
|
| add lambda parameter | none — manual edit (`introduce-param` / `change-method-signature` cannot introduce a lambda expression as a parameter: `introduce-param` on the lambda inlines it into the call site, referencing the private `scorer` field from outside the class → does not compile. Lambdas are only handled as rewrite *targets* — method references get converted to lambdas at call sites — never as the subject of an operation) |
|
|
| move method to another class (step 4, not in scope) | `java-refactor move-method` (call sites not rewritten) |
|
|
|
|
Rehearsal evidence (2026-09-27, see `rehearsal-findings.md`): with the CLI
|
|
merely *available* (not mandated) plus a 6-line subcommand intro, glm-5.3-flash
|
|
passed in 2:37 / 23 turns — using `extract` (dry-run → apply) for step 2 and
|
|
editing manually for steps 1 and 3, with no tool-shopping detour. Mandating the
|
|
CLI (variant A) tripled the turns of plain editing (B) on this 55-line file.
|