Add agent rehearsal harness, findings, and per-run evidence for split-loop-with-lambda

Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the
exercise; deterministic gate = solution test + approved file. Key results:
mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both);
weak model fails only under the mandate (A+Q), passes when free or optional.
Drift documented with code: scorer.updateSelection left inside the extracted
method (A+Q, C+Q). Harness in tools/rehearse; summary in
rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in
rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
This commit is contained in:
Your Name
2026-09-27 17:36:19 +00:00
parent a27e32688a
commit b993f25c53
51 changed files with 2281 additions and 2 deletions
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Stage 1 — one-time setup: build java-refactoring-mcp, install the CLI wrapper,
# create the pristine kata workspace, and smoke-test the CLI.
# Idempotent: safe to re-run. Artifacts live in /tmp/rehearsal (scratch space).
set -euo pipefail
MCP_REPO=/tmp/java-refactoring-mcp
KATA=/tmp/IceCreamForecasting-Refactoring-Kata
SCRATCH=/tmp/rehearsal
MVN=/tmp/apache-maven-3.9.9/bin/mvn
# Temurin JDK downloaded to /tmp by this script (system has JRE only, no javac)
if [ ! -x /tmp/jdk21/bin/javac ]; then
echo "== 0. Downloading Temurin JDK 21 to /tmp/jdk21 (system has no javac) =="
curl -sL -o /tmp/jdk21.tar.gz \
"https://api.adoptium.net/v3/binary/latest/21/ga/linux/x64/jdk/hotspot/normal/eclipse"
mkdir -p /tmp/jdk21
tar -xzf /tmp/jdk21.tar.gz -C /tmp/jdk21 --strip-components=1
fi
export JAVA_HOME=/tmp/jdk21
export PATH="$JAVA_HOME/bin:$PATH"
mkdir -p "$SCRATCH/bin"
echo "== 1. Build the tool (skip if fat jars exist) =="
CLI_JAR=$(ls "$MCP_REPO"/refactor-cli/target/refactor-cli-*-fat.jar 2>/dev/null | head -1 || true)
MCP_JAR=$(ls "$MCP_REPO"/refactor-mcp/target/refactor-mcp-*-fat.jar 2>/dev/null | head -1 || true)
if [ -z "${CLI_JAR:-}" ]; then
(cd "$MCP_REPO" && "$MVN" -q package -DskipTests)
fi
CLI_JAR=$(ls "$MCP_REPO"/refactor-cli/target/refactor-cli-*-fat.jar | head -1)
MCP_JAR=$(ls "$MCP_REPO"/refactor-mcp/target/refactor-mcp-*-fat.jar | head -1)
echo " CLI jar: $CLI_JAR"
echo " MCP jar: $MCP_JAR (for reference; not needed by this rehearsal)"
echo " JAVA_HOME: $JAVA_HOME"
echo "== 2. Install CLI wrapper =="
cat > "$SCRATCH/bin/java-refactor" <<EOF
#!/usr/bin/env bash
exec "$JAVA_HOME/bin/java" -jar "$CLI_JAR" "\$@"
EOF
chmod +x "$SCRATCH/bin/java-refactor"
echo "== 3. Pristine kata workspace (never touched by runs) =="
rm -rf "$SCRATCH/pristine"
cp -r "$KATA/java" "$SCRATCH/pristine"
rm -rf "$SCRATCH/pristine/target"
echo "== 4. Baseline: pristine compiles (tests EXPECTED to fail — Scorer is flaky by design) =="
( cd "$SCRATCH/pristine" \
&& PATH="$JAVA_HOME/bin:/tmp/apache-maven-3.9.9/bin:$PATH" \
mvn -q test-compile ) \
&& echo " test-compile OK" \
|| { echo " test-compile FAILED"; exit 1; }
# Demonstrate the expected baseline failure (flaky approval via Scorer's Random + today's date):
( cd "$SCRATCH/pristine" \
&& PATH="$JAVA_HOME/bin:/tmp/apache-maven-3.9.9/bin:$PATH" \
mvn -q test ) > "$SCRATCH/baseline-test.log" 2>&1 \
&& echo " NOTE: pristine tests unexpectedly PASSED" \
|| echo " pristine tests fail as expected (untestable Scorer dependency) — see baseline-test.log"
echo "== 5. Smoke test: java-refactor --help =="
PATH="$SCRATCH/bin:$PATH" java-refactor --help > "$SCRATCH/smoke-help.txt" 2>&1 \
&& head -3 "$SCRATCH/smoke-help.txt"
echo "== 6. Smoke test: dry-run extract of the weather loop =="
# LongRangeForecasts.java lines 29-36: the for-loop over interestingDates.
rm -rf "$SCRATCH/smoke"
cp -r "$SCRATCH/pristine" "$SCRATCH/smoke"
( cd "$SCRATCH/smoke" \
&& PATH="$SCRATCH/bin:$PATH" java-refactor extract \
--file src/main/java/codingdojo/LongRangeForecasts.java \
--start-line 29 --start-column 9 \
--end-line 36 --end-column 10 \
--name collectExpectedWeather --dry-run ) \
> "$SCRATCH/smoke-extract.txt" 2>&1 \
&& echo " dry-run OK (see /tmp/rehearsal/smoke-extract.txt)" \
|| { echo " dry-run FAILED — inspect /tmp/rehearsal/smoke-extract.txt"; exit 1; }
echo
echo "Stage 1 done. Artifacts in $SCRATCH:"
ls -la "$SCRATCH"