Your Name
b993f25c53
Six scripted pi sub-agent runs (3 tool regimes × 2 models) rehearsed the exercise; deterministic gate = solution test + approved file. Key results: mandated CLI triples cost on the strong model (4:06 vs 0:39, PASS both); weak model fails only under the mandate (A+Q), passes when free or optional. Drift documented with code: scorer.updateSelection left inside the extracted method (A+Q, C+Q). Harness in tools/rehearse; summary in rehearsal-findings.md; per-run evidence in rehearsal-runs/; cheat sheet in rehearse-with-agent.md corrected (lambda-typed parameters unsupported).
152 lines
6.0 KiB
Bash
Executable File
152 lines
6.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Stage 2 — A/B rehearsal: spawn a pi sub-agent per variant to do the refactoring,
|
|
# with and without the java-refactor CLI, capped at 10 minutes wall clock each.
|
|
#
|
|
# usage: bash tools/rehearse/run-rehearsal.sh [A] [B] (default: both)
|
|
#
|
|
# For each run: fresh workspace from pristine, AGENTS.md with the shared task +
|
|
# variant-specific tool rules, pi -p with its own session dir, then measurement
|
|
# (tokens/wall clock) and deterministic verification (solution test + approved file).
|
|
set -uo pipefail
|
|
|
|
SCRATCH=/tmp/rehearsal
|
|
PRISTINE="$SCRATCH/pristine"
|
|
RUNS_DIR="$SCRATCH/runs"
|
|
REPO=/home/cardfanatic/qwan/exercises/learning-hour-maker
|
|
SOLUTION="$REPO/ideas/split-loop-with-lambda/exercise/solution"
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
CAP_SECONDS=600
|
|
|
|
[ -d "$PRISTINE" ] || { echo "Run tools/rehearse/setup.sh first (no $PRISTINE)"; exit 1; }
|
|
WHICH="${*:-A B}"
|
|
|
|
# Variant -> pi model override (empty = pi settings default).
|
|
# +Q suffix: same variant's rules on entrim Qwen3.6-35B-A3B (weak-tool-caller datapoint).
|
|
case "$WHICH" in
|
|
A+Q) VARIANT=A; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
|
B+Q) VARIANT=B; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
|
C+Q) VARIANT=C; MODEL_ARG=(--provider entrim --model Qwen/Qwen3.6-35B-A3B) ;;
|
|
*) VARIANT="$WHICH"; MODEL_ARG=() ;;
|
|
esac
|
|
RUN="$WHICH"
|
|
|
|
PROMPT='Read AGENTS.md in this directory and execute the refactoring task it describes. Run mvn -q test after each change. When done, stop and report the final diff of LongRangeForecasts.java.'
|
|
|
|
for RUN in $WHICH; do
|
|
RUN_DIR="$RUNS_DIR/$RUN"
|
|
WS="$RUN_DIR/workspace"
|
|
rm -rf "$RUN_DIR"
|
|
mkdir -p "$WS" "$RUN_DIR/session"
|
|
cp -r "$PRISTINE/." "$WS/"
|
|
|
|
# --- AGENTS.md: shared design instructions (identical for A and B) ---
|
|
cat > "$WS/AGENTS.md" <<'EOF'
|
|
# Rehearsal workspace — refactoring task
|
|
|
|
You are refactoring `src/main/java/codingdojo/LongRangeForecasts.java` in this
|
|
Maven project. Run `mvn -q test` after each change (mvn is on your PATH). It may
|
|
fail — that is expected: the dependency on `Scorer` makes the method untestable —
|
|
but the code must still *compile*.
|
|
|
|
## Goal
|
|
|
|
In `LongRangeForecasts.longRangeForecast()`, the loop over `interestingDates`
|
|
collects weather by calling `scorer` (an unwanted external dependency).
|
|
|
|
1. Introduce a lambda for the scorer-dependent code and move its definition to
|
|
the top of the method, leaving the invocation where it is. (No refactoring
|
|
tool supports this — do it as a minimal manual edit.)
|
|
2. Extract the weather-collection loop into a new method `collectExpectedWeather`
|
|
that takes the lambda as a parameter (suggested type `LongFunction<Boolean>`).
|
|
3. Add the lambda as a parameter to `longRangeForecast` itself: the signature
|
|
must become `longRangeForecast(LongFunction<Boolean> dayOffsetToWeather)`,
|
|
with a public no-arg overload `longRangeForecast()` that delegates and wraps
|
|
the `scorer` field in a lambda. Do not keep the `String quarter` parameter.
|
|
(`java-refactor change-method-signature` or `java-refactor introduce-param`
|
|
may help; fall back to a minimal manual edit if no subcommand can do it.)
|
|
4. Stop. Report the final diff.
|
|
EOF
|
|
|
|
# --- variant-specific tool rules ---
|
|
if [ "$VARIANT" = "A" ]; then
|
|
cat >> "$WS/AGENTS.md" <<'EOF'
|
|
|
|
## Tool rules (variant A)
|
|
You may only change code by invoking `java-refactor` CLI commands
|
|
(`java-refactor --help` lists them; it is on your PATH). Never edit `.java`
|
|
files with text edits — the ONLY exceptions are the introduce-lambda edit
|
|
(step 1) and, if no `java-refactor` subcommand can accomplish it, a minimal
|
|
manual edit for step 3.
|
|
EOF
|
|
elif [ "$VARIANT" = "B" ]; then
|
|
cat >> "$WS/AGENTS.md" <<'EOF'
|
|
|
|
## Tool rules (variant B)
|
|
You may change code with your normal file-editing tools.
|
|
EOF
|
|
else
|
|
cat >> "$WS/AGENTS.md" <<'EOF'
|
|
|
|
## Tool rules (variant C)
|
|
A semantic Java refactoring CLI, `java-refactor`, is on your PATH
|
|
(`java-refactor --help` lists the subcommands). Using it is OPTIONAL — you
|
|
may also change code with your normal file-editing tools. Choose per step
|
|
whatever you think is most reliable.
|
|
|
|
Subcommands that may be relevant for this task:
|
|
- `extract` — extract selected statements into a new private method
|
|
(selection given as start/end line+column; `--dry-run` previews)
|
|
- `introduce-param` / `change-method-signature` — alter a method's parameters
|
|
- `rename` — rename a symbol project-wide
|
|
EOF
|
|
fi
|
|
cp "$WS/AGENTS.md" "$RUN_DIR/agents-md.txt"
|
|
printf '%s\n' "$PROMPT" > "$RUN_DIR/prompt.txt"
|
|
|
|
# --- spawn the sub-agent, fresh session dir per run ---
|
|
# PATH: maven for both; the java-refactor wrapper ONLY for run A.
|
|
echo "== Run $RUN: starting sub-agent (cap ${CAP_SECONDS}s) =="
|
|
(
|
|
cd "$WS"
|
|
export JAVA_HOME=/tmp/jdk21
|
|
if [ "$VARIANT" = "A" ] || [ "$VARIANT" = "C" ]; then
|
|
export PATH="$SCRATCH/bin:/tmp/apache-maven-3.9.9/bin:$PATH"
|
|
else
|
|
export PATH="/tmp/apache-maven-3.9.9/bin:$PATH"
|
|
fi
|
|
/usr/bin/time -v -o "$RUN_DIR/time.txt" \
|
|
timeout -k 15 "$CAP_SECONDS" \
|
|
pi -p -a "${MODEL_ARG[@]}" --session-dir "$RUN_DIR/session" \
|
|
--name "rehearsal-$RUN" "$PROMPT"
|
|
) > "$RUN_DIR/pi-stdout.txt" 2> "$RUN_DIR/pi-stderr.txt"
|
|
echo " exit=$? (124 = timed out)"
|
|
|
|
echo "== Run $RUN: measuring =="
|
|
bash "$SCRIPT_DIR/measure_run.sh" "$RUN_DIR" | tee "$RUN_DIR/measure.txt"
|
|
|
|
echo "== Run $RUN: verifying =="
|
|
bash "$SCRIPT_DIR/verify_outcome.sh" "$RUN_DIR" | tee "$RUN_DIR/verify.txt"
|
|
done
|
|
|
|
# --- combined report ---
|
|
REPORT="$SCRATCH/report.md"
|
|
{
|
|
echo "# Rehearsal report — split-loop-with-lambda ($(date -Iseconds))"
|
|
echo
|
|
for RUN in $WHICH; do
|
|
RUN_DIR="$RUNS_DIR/$RUN"
|
|
echo "## Run $RUN"
|
|
echo
|
|
echo '```'
|
|
cat "$RUN_DIR/measure.txt" 2>/dev/null || echo "(measurement missing)"
|
|
echo
|
|
cat "$RUN_DIR/verify.txt" 2>/dev/null || echo "(verification missing)"
|
|
echo '```'
|
|
echo
|
|
echo "Full artifacts: \`$RUN_DIR\` (diff.patch, pi-stdout.txt, session/)"
|
|
echo
|
|
done
|
|
} > "$REPORT"
|
|
echo "Report: $REPORT"
|
|
cat "$REPORT"
|