Add agent rehearsal harness, findings, and per-run evidence for split-loop-with-lambda
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the exercise; deterministic gate = solution test + approved file. Key results: mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both); weak model fails only under the mandate (A+Q), passes when free or optional. Drift documented with code: scorer.updateSelection left inside the extracted method (A+Q, C+Q). Harness in tools/rehearse; summary in rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
This commit is contained in:
Your Name
Executable
+151
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stage 2 — A/B rehearsal: spawn a pi sub-agent per variant to do the refactoring,
|
||||
# with and without the java-refactor CLI, capped at 10 minutes wall clock each.
|
||||
#
|
||||
# usage: bash tools/rehearse/run-rehearsal.sh [A] [B] (default: both)
|
||||
#
|
||||
# For each run: fresh workspace from pristine, AGENTS.md with the shared task +
|
||||
# variant-specific tool rules, pi -p with its own session dir, then measurement
|
||||
# (tokens/wall clock) and deterministic verification (solution test + approved file).
|
||||
set -uo pipefail
|
||||
|
||||
SCRATCH=/tmp/rehearsal
|
||||
PRISTINE="$SCRATCH/pristine"
|
||||
RUNS_DIR="$SCRATCH/runs"
|
||||
REPO=/home/cardfanatic/qwan/exercises/learning-hour-maker
|
||||
SOLUTION="$REPO/ideas/split-loop-with-lambda/exercise/solution"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CAP_SECONDS=600
|
||||
|
||||
[ -d "$PRISTINE" ] || { echo "Run tools/rehearse/setup.sh first (no $PRISTINE)"; exit 1; }
|
||||
WHICH="${*:-A B}"
|
||||
|
||||
# Variant -> pi model override (empty = pi settings default).
|
||||
# +Q suffix: same variant's rules on entrim Qwen3.6-35B-A3B (weak-tool-caller datapoint).
|
||||
case "$WHICH" in
|
||||
A+Q) VARIANT=A; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
||||
B+Q) VARIANT=B; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
||||
C+Q) VARIANT=C; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
||||
*) VARIANT="$WHICH"; MODEL_ARG=() ;;
|
||||
esac
|
||||
RUN="$WHICH"
|
||||
|
||||
PROMPT='Read AGENTS.md in this directory and execute the refactoring task it describes. Run mvn -q test after each change. When done, stop and report the final diff of LongRangeForecasts.java.'
|
||||
|
||||
for RUN in $WHICH; do
|
||||
RUN_DIR="$RUNS_DIR/$RUN"
|
||||
WS="$RUN_DIR/workspace"
|
||||
rm -rf "$RUN_DIR"
|
||||
mkdir -p "$WS" "$RUN_DIR/session"
|
||||
cp -r "$PRISTINE/." "$WS/"
|
||||
|
||||
# --- AGENTS.md: shared design instructions (identical for A and B) ---
|
||||
cat > "$WS/AGENTS.md" <<'EOF'
|
||||
# Rehearsal workspace — refactoring task
|
||||
|
||||
You are refactoring `src/main/java/codingdojo/LongRangeForecasts.java` in this
|
||||
Maven project. Run `mvn -q test` after each change (mvn is on your PATH). It may
|
||||
fail — that is expected: the dependency on `Scorer` makes the method untestable —
|
||||
but the code must still *compile*.
|
||||
|
||||
## Goal
|
||||
|
||||
In `LongRangeForecasts.longRangeForecast()`, the loop over `interestingDates`
|
||||
collects weather by calling `scorer` (an unwanted external dependency).
|
||||
|
||||
1. Introduce a lambda for the scorer-dependent code and move its definition to
|
||||
the top of the method, leaving the invocation where it is. (No refactoring
|
||||
tool supports this — do it as a minimal manual edit.)
|
||||
2. Extract the weather-collection loop into a new method `collectExpectedWeather`
|
||||
that takes the lambda as a parameter (suggested type `LongFunction<Boolean>`).
|
||||
3. Add the lambda as a parameter to `longRangeForecast` itself: the signature
|
||||
must become `longRangeForecast(LongFunction<Boolean> dayOffsetToWeather)`,
|
||||
with a public no-arg overload `longRangeForecast()` that delegates and wraps
|
||||
the `scorer` field in a lambda. Do not keep the `String quarter` parameter.
|
||||
(`java-refactor change-method-signature` or `java-refactor introduce-param`
|
||||
may help; fall back to a minimal manual edit if no subcommand can do it.)
|
||||
4. Stop. Report the final diff.
|
||||
EOF
|
||||
|
||||
# --- variant-specific tool rules ---
|
||||
if [ "$VARIANT" = "A" ]; then
|
||||
cat >> "$WS/AGENTS.md" <<'EOF'
|
||||
|
||||
## Tool rules (variant A)
|
||||
You may only change code by invoking `java-refactor` CLI commands
|
||||
(`java-refactor --help` lists them; it is on your PATH). Never edit `.java`
|
||||
files with text edits — the ONLY exceptions are the introduce-lambda edit
|
||||
(step 1) and, if no `java-refactor` subcommand can accomplish it, a minimal
|
||||
manual edit for step 3.
|
||||
EOF
|
||||
elif [ "$VARIANT" = "B" ]; then
|
||||
cat >> "$WS/AGENTS.md" <<'EOF'
|
||||
|
||||
## Tool rules (variant B)
|
||||
You may change code with your normal file-editing tools.
|
||||
EOF
|
||||
else
|
||||
cat >> "$WS/AGENTS.md" <<'EOF'
|
||||
|
||||
## Tool rules (variant C)
|
||||
A semantic Java refactoring CLI, `java-refactor`, is on your PATH
|
||||
(`java-refactor --help` lists the subcommands). Using it is OPTIONAL — you
|
||||
may also change code with your normal file-editing tools. Choose per step
|
||||
whatever you think is most reliable.
|
||||
|
||||
Subcommands that may be relevant for this task:
|
||||
- `extract` — extract selected statements into a new private method
|
||||
(selection given as start/end line+column; `--dry-run` previews)
|
||||
- `introduce-param` / `change-method-signature` — alter a method's parameters
|
||||
- `rename` — rename a symbol project-wide
|
||||
EOF
|
||||
fi
|
||||
cp "$WS/AGENTS.md" "$RUN_DIR/agents-md.txt"
|
||||
printf '%s\n' "$PROMPT" > "$RUN_DIR/prompt.txt"
|
||||
|
||||
# --- spawn the sub-agent, fresh session dir per run ---
|
||||
# PATH: maven for both; the java-refactor wrapper ONLY for run A.
|
||||
echo "== Run $RUN: starting sub-agent (cap ${CAP_SECONDS}s) =="
|
||||
(
|
||||
cd "$WS"
|
||||
export JAVA_HOME=/tmp/jdk21
|
||||
if [ "$VARIANT" = "A" ] || [ "$VARIANT" = "C" ]; then
|
||||
export PATH="$SCRATCH/bin:/tmp/apache-maven-3.9.9/bin:$PATH"
|
||||
else
|
||||
export PATH="/tmp/apache-maven-3.9.9/bin:$PATH"
|
||||
fi
|
||||
/usr/bin/time -v -o "$RUN_DIR/time.txt" \
|
||||
timeout -k 15 "$CAP_SECONDS" \
|
||||
pi -p -a "${MODEL_ARG[@]}" --session-dir "$RUN_DIR/session" \
|
||||
--name "rehearsal-$RUN" "$PROMPT"
|
||||
) > "$RUN_DIR/pi-stdout.txt" 2> "$RUN_DIR/pi-stderr.txt"
|
||||
echo " exit=$? (124 = timed out)"
|
||||
|
||||
echo "== Run $RUN: measuring =="
|
||||
bash "$SCRIPT_DIR/measure_run.sh" "$RUN_DIR" | tee "$RUN_DIR/measure.txt"
|
||||
|
||||
echo "== Run $RUN: verifying =="
|
||||
bash "$SCRIPT_DIR/verify_outcome.sh" "$RUN_DIR" | tee "$RUN_DIR/verify.txt"
|
||||
done
|
||||
|
||||
# --- combined report ---
|
||||
REPORT="$SCRATCH/report.md"
|
||||
{
|
||||
echo "# Rehearsal report — split-loop-with-lambda ($(date -Iseconds))"
|
||||
echo
|
||||
for RUN in $WHICH; do
|
||||
RUN_DIR="$RUNS_DIR/$RUN"
|
||||
echo "## Run $RUN"
|
||||
echo
|
||||
echo '```'
|
||||
cat "$RUN_DIR/measure.txt" 2>/dev/null || echo "(measurement missing)"
|
||||
echo
|
||||
cat "$RUN_DIR/verify.txt" 2>/dev/null || echo "(verification missing)"
|
||||
echo '```'
|
||||
echo
|
||||
echo "Full artifacts: \`$RUN_DIR\` (diff.patch, pi-stdout.txt, session/)"
|
||||
echo
|
||||
done
|
||||
} > "$REPORT"
|
||||
echo "Report: $REPORT"
|
||||
cat "$REPORT"
|
||||
Reference in New Issue
Block a user