Compare commits
18
Commits
cab445e603
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f6d7416b00 | ||
|
|
93a5675f06 | ||
|
|
113878e83f | ||
|
|
7faabcb038 | ||
|
|
d7eabfffbb | ||
|
|
823af3c486 | ||
|
|
040513e1d6 | ||
|
|
383cb46fe7 | ||
|
|
45a13fd08c | ||
|
|
ce4d6c5971 | ||
|
|
98e18643c5 | ||
|
|
a38c76c65e | ||
|
|
0cf13ed54e | ||
|
|
c62eb432bf | ||
|
|
f8de509ec6 | ||
|
|
36b30c1f02 | ||
|
|
fd0c343729 | ||
|
|
a4181af13e |
@@ -1,3 +1,4 @@
|
||||
node_modules/
|
||||
.pnpm-store/
|
||||
pnpm-lock.yaml
|
||||
.pi/llm-metrics.log
|
||||
|
||||
@@ -6,10 +6,34 @@ Experimental monorepo for [Pi coding agent](https://github.com/mariozechner/pi-c
|
||||
|
||||
### `pi-turn-limit`
|
||||
|
||||
Limits the number of turns (agent round-trips) in a Pi session. When the limit is exceeded, the agent aborts with an error notification.
|
||||
Limits the number of turns (agent round-trips) in a Pi session. When the limit is reached, the user is prompted to continue or abort. Use when you want to be in-the-loop, or when a model misbehaves and does too many tool calls. It is a good way to control the *Time To Next Interaction*.
|
||||
|
||||
- **Default limit:** 25 turns
|
||||
- **Override:** set `PI_MAX_TURNS` environment variable
|
||||
- **Override:** set `PI_MAX_TURNS` environment variable to a positive integer
|
||||
- **Unlimited:** set `PI_MAX_TURNS=unlimited` or run the `turn-limit unlimited` command to disable the boundary check entirely (the counter still increments for observability)
|
||||
- **Re-enable:** switch from unlimited back to a number via `turn-limit <N>`; the counter resets to 0
|
||||
|
||||
See [packages/pi-turn-limit/README.md](packages/pi-turn-limit/README.md) for details and the [Allium spec](packages/pi-turn-limit/turn-limit.allium).
|
||||
|
||||
### `pi-notifications`
|
||||
|
||||
Audio alerts via `afplay` when the agent finishes a turn. Run the agent and step away — you'll hear when input is needed. No multi-tasking required, but it gives you the breathing room to stretch, grab a coffee, or write something down without staring at the screen.
|
||||
|
||||
- **Config:** `PI_NOTIFICATION_ENABLED`, `PI_NOTIFICATION_AGENT_END`, `PI_NOTIFICATION_AUDIO` (defaults to macOS Glass sound)
|
||||
- **Platform:** macOS (uses `afplay`)
|
||||
|
||||
See [packages/pi-notifications/README.md](packages/pi-notifications/README.md) for details.
|
||||
|
||||
### `pi-llm-performance`
|
||||
|
||||
Captures and displays LLM inference performance metrics (TTFT, prefill/generation throughput, combined speed) after each prompt. Lets you benchmark shiny new local inference server optimizations at a glance — no need to dig through different server logs.
|
||||
|
||||
- **Output:** TUI notification + status bar (`📊 tok/s`) + JSONL log at `.pi/llm-metrics.log`
|
||||
- **Sanity checks:** Warns when generation speed exceeds 500 tok/s (physically impossible)
|
||||
|
||||
See [packages/pi-llm-performance/README.md](packages/pi-llm-performance/README.md) for details.
|
||||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
{"type":"config","name":"Turn-limit extension naming","metricName":"extension_name","metricUnit":"","bestDirection":"lower"}
|
||||
@@ -1,5 +1,6 @@
|
||||
[tools]
|
||||
bun = "latest"
|
||||
deno = "latest"
|
||||
elixir = "latest"
|
||||
erlang = "latest"
|
||||
node = "24"
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# pi-llm-performance
|
||||
|
||||
Pi coding agent extension that captures and displays LLM inference performance metrics.
|
||||
|
||||
## Why
|
||||
|
||||
Understanding model performance helps you:
|
||||
|
||||
- **Compare models** — measure throughput differences between providers and model sizes
|
||||
- **Debug slowdowns** — spot when prefill or generation degrades unexpectedly
|
||||
- **Validate hardware** — confirm your setup delivers expected token throughput
|
||||
- **Tune parameters** — evaluate the impact of speculative decoding, context window size, etc.
|
||||
|
||||
## What it measures
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| **TTFT** | Time to first token (ms) — how long before you see output |
|
||||
| **Prefill speed** | Input tokens processed per second during the prefill phase |
|
||||
| **Generation speed** | Output tokens generated per second during the generation phase |
|
||||
| **Combined speed** | Total tokens (input + output) per second across the full prompt |
|
||||
|
||||
## How it works
|
||||
|
||||
The extension hooks into pi's agent lifecycle events:
|
||||
|
||||
| Event | Behavior |
|
||||
|-------|----------|
|
||||
| `agent_start` | Records provider/model, resets counters |
|
||||
| `turn_start` | Marks turn boundary |
|
||||
| `message_update` | Captures TTFT on first token delta |
|
||||
| `turn_end` | Records token counts and turn duration |
|
||||
| `agent_end` | Aggregates metrics, displays in TUI, logs to `.pi/llm-metrics.log` |
|
||||
|
||||
## Output
|
||||
|
||||
### TUI notification
|
||||
|
||||
After each prompt completes, a notification shows:
|
||||
|
||||
```
|
||||
📊 Performance: llama.cpp/Qwen3.6-35B-A3B-MXFP4_MOE.gguf
|
||||
Prefill: 1,240 tokens @ 68.3 tok/s
|
||||
Generation: 312 tokens @ 89.9 tok/s
|
||||
Combined: 1,552 tokens @ 78.4 tok/s (19.8s total)
|
||||
TTFT: 1250ms
|
||||
```
|
||||
|
||||
### Status bar
|
||||
|
||||
The footer status shows combined throughput: `📊 78.4 tok/s`
|
||||
|
||||
### Log file
|
||||
|
||||
Each prompt writes a JSONL entry to `.pi/llm-metrics.log`:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-04-28T10:05:00.000Z",
|
||||
"provider": "llama.cpp",
|
||||
"model": "Qwen3.6-35B-A3B-MXFP4_MOE.gguf",
|
||||
"turnCount": 1,
|
||||
"inputTokens": 1240,
|
||||
"outputTokens": 312,
|
||||
"totalTokens": 1552,
|
||||
"prefillTokensPerSec": 68.3,
|
||||
"generationTokensPerSec": 89.9,
|
||||
"combinedTokensPerSec": 78.4,
|
||||
"totalDurationMs": 19800,
|
||||
"timeToFirstTokenMs": 1250,
|
||||
"rawTimestamps": {
|
||||
"ttftMs": 1250,
|
||||
"generationDurationMs": 18550,
|
||||
"turns": [{"turnId": "turn-0", "durationMs": 19800, "ttftMs": 1250}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Sanity checks
|
||||
|
||||
The extension warns to the console if generation speed exceeds 500 tok/s (physically impossible for any known model/hardware setup). This helps catch timing bugs early.
|
||||
|
||||
## Development
|
||||
|
||||
This package lives in the `pi-extensions` monorepo.
|
||||
|
||||
```bash
|
||||
pnpm install # workspace setup
|
||||
deno test # run tests
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"imports": {
|
||||
"@std/assert": "jsr:@std/assert@^1.0.0"
|
||||
},
|
||||
"tasks": {
|
||||
"test": "deno test src/"
|
||||
}
|
||||
}
|
||||
Generated
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"version": "5",
|
||||
"specifiers": {
|
||||
"jsr:@std/assert@*": "1.0.19",
|
||||
"jsr:@std/assert@^1.0.19": "1.0.19",
|
||||
"jsr:@std/internal@^1.0.12": "1.0.12",
|
||||
"jsr:@std/testing@*": "1.0.18"
|
||||
},
|
||||
"jsr": {
|
||||
"@std/assert@1.0.19": {
|
||||
"integrity": "eaada96ee120cb980bc47e040f82814d786fe8162ecc53c91d8df60b8755991e",
|
||||
"dependencies": [
|
||||
"jsr:@std/internal"
|
||||
]
|
||||
},
|
||||
"@std/internal@1.0.12": {
|
||||
"integrity": "972a634fd5bc34b242024402972cd5143eac68d8dffaca5eaa4dba30ce17b027"
|
||||
},
|
||||
"@std/testing@1.0.18": {
|
||||
"integrity": "d3152f57b11666bf6358d0e127c7e3488e91178b0c2d8fbf0793e1c53cd13cb1",
|
||||
"dependencies": [
|
||||
"jsr:@std/assert@^1.0.19"
|
||||
]
|
||||
}
|
||||
},
|
||||
"workspace": {
|
||||
"dependencies": [
|
||||
"jsr:@std/assert@1"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "pi-llm-performance",
|
||||
"version": "0.1.0",
|
||||
"description": "LLM performance metrics extension",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./src/llm-performance-metrics.ts"
|
||||
},
|
||||
"keywords": ["pi-package"],
|
||||
"pi": {
|
||||
"extensions": ["src/llm-performance-metrics.ts"]
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@mariozechner/pi-coding-agent": "*"
|
||||
},
|
||||
"license": "MIT"
|
||||
}
|
||||
@@ -0,0 +1,558 @@
|
||||
import {
|
||||
calculateTurnMetrics,
|
||||
aggregatePromptMetrics,
|
||||
formatMetricsForDisplay,
|
||||
toLogEntry,
|
||||
type TurnMetrics,
|
||||
type PromptMetrics,
|
||||
} from "./llm-metrics-core.ts";
|
||||
import { assertEquals, assertGreaterOrEqual, assertLessOrEqual } from "jsr:@std/assert";
|
||||
|
||||
Deno.test("calculateTurnMetrics - creates turn metrics object", () => {
|
||||
const result = calculateTurnMetrics({
|
||||
turnId: "turn-1",
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
durationMs: 2000,
|
||||
timeToFirstTokenMs: 500,
|
||||
});
|
||||
|
||||
assertEquals(result.turnId, "turn-1");
|
||||
assertEquals(result.inputTokens, 100);
|
||||
assertEquals(result.outputTokens, 50);
|
||||
assertEquals(result.durationMs, 2000);
|
||||
assertEquals(result.timeToFirstTokenMs, 500);
|
||||
});
|
||||
|
||||
Deno.test("calculateTurnMetrics - handles missing timeToFirstToken", () => {
|
||||
const result = calculateTurnMetrics({
|
||||
turnId: "turn-1",
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
durationMs: 2000,
|
||||
});
|
||||
|
||||
assertEquals(result.timeToFirstTokenMs, undefined);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - aggregates single turn", () => {
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-1",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 5000,
|
||||
timeToFirstTokenMs: 800,
|
||||
},
|
||||
];
|
||||
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
assertEquals(result.provider, "anthropic");
|
||||
assertEquals(result.model, "claude-sonnet-4");
|
||||
assertEquals(result.turnCount, 1);
|
||||
assertEquals(result.inputTokens, 1000);
|
||||
assertEquals(result.outputTokens, 200);
|
||||
assertEquals(result.totalTokens, 1200);
|
||||
assertEquals(result.totalDurationMs, 5000);
|
||||
assertEquals(result.timeToFirstTokenMs, 800);
|
||||
|
||||
// Tokens per second calculations
|
||||
// prefill: 1000 input tokens / 0.8s TTFT = 1250 tok/s
|
||||
assertEquals(result.prefillTokensPerSec, 1250);
|
||||
// generation: 200 output tokens / 4.2s (5s - 0.8s) = 47.62 tok/s
|
||||
assertGreaterOrEqual(result.generationTokensPerSec, 47.6);
|
||||
assertLessOrEqual(result.generationTokensPerSec, 47.7);
|
||||
// combined: 1200 total tokens / 5s = 240 tok/s
|
||||
assertEquals(result.combinedTokensPerSec, 240);
|
||||
|
||||
// rawTimestamps
|
||||
assertEquals(result.rawTimestamps?.ttftMs, 800);
|
||||
assertEquals(result.rawTimestamps?.allTtftMs, [800]);
|
||||
assertEquals(result.rawTimestamps?.generationDurationMs, 4200);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - aggregates multiple turns", () => {
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-1",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 3000,
|
||||
timeToFirstTokenMs: 800,
|
||||
},
|
||||
{
|
||||
turnId: "turn-2",
|
||||
inputTokens: 500,
|
||||
outputTokens: 150,
|
||||
durationMs: 2000,
|
||||
},
|
||||
{
|
||||
turnId: "turn-3",
|
||||
inputTokens: 300,
|
||||
outputTokens: 100,
|
||||
durationMs: 1500,
|
||||
},
|
||||
];
|
||||
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "openai",
|
||||
model: "gpt-4o",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
assertEquals(result.turnCount, 3);
|
||||
assertEquals(result.inputTokens, 1800); // 1000 + 500 + 300
|
||||
assertEquals(result.outputTokens, 450); // 200 + 150 + 100
|
||||
assertEquals(result.totalTokens, 2250);
|
||||
assertEquals(result.totalDurationMs, 6500); // 3000 + 2000 + 1500
|
||||
assertEquals(result.timeToFirstTokenMs, 800); // From first turn only
|
||||
|
||||
// Tokens per second: prefill uses TTFT (0.8s), generation uses (total - TTFT) = 5.7s
|
||||
// prefill: 1800 / 0.8 = 2250 tok/s
|
||||
assertEquals(result.prefillTokensPerSec, 2250);
|
||||
// generation: 450 / 5.7 = 78.95 tok/s
|
||||
assertGreaterOrEqual(result.generationTokensPerSec, 78.9);
|
||||
assertLessOrEqual(result.generationTokensPerSec, 79.0);
|
||||
// combined: 2250 / 6.5 = 346.15 tok/s
|
||||
assertGreaterOrEqual(result.combinedTokensPerSec, 346.1);
|
||||
assertLessOrEqual(result.combinedTokensPerSec, 346.2);
|
||||
|
||||
// rawTimestamps: only turn-1 has valid TTFT, turns 2+ have none
|
||||
assertEquals(result.rawTimestamps?.ttftMs, 800);
|
||||
assertEquals(result.rawTimestamps?.allTtftMs, [800]);
|
||||
assertEquals(result.rawTimestamps?.generationDurationMs, 5700);
|
||||
assertEquals(result.rawTimestamps?.turns.length, 3);
|
||||
assertEquals(result.rawTimestamps?.turns[0].ttftMs, 800);
|
||||
assertEquals(result.rawTimestamps?.turns[1].ttftMs, undefined);
|
||||
assertEquals(result.rawTimestamps?.turns[2].ttftMs, undefined);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - handles empty turn list", () => {
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnMetrics: [],
|
||||
});
|
||||
|
||||
assertEquals(result.turnCount, 0);
|
||||
assertEquals(result.inputTokens, 0);
|
||||
assertEquals(result.outputTokens, 0);
|
||||
assertEquals(result.totalTokens, 0);
|
||||
assertEquals(result.prefillTokensPerSec, 0);
|
||||
assertEquals(result.generationTokensPerSec, 0);
|
||||
assertEquals(result.combinedTokensPerSec, 0);
|
||||
assertEquals(result.totalDurationMs, 0);
|
||||
assertEquals(result.timeToFirstTokenMs, undefined);
|
||||
});
|
||||
|
||||
Deno.test("formatMetricsForDisplay - formats single turn metrics", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnCount: 1,
|
||||
inputTokens: 1250,
|
||||
outputTokens: 342,
|
||||
totalTokens: 1592,
|
||||
prefillTokensPerSec: 482.1,
|
||||
generationTokensPerSec: 18.3,
|
||||
combinedTokensPerSec: 38.0,
|
||||
totalDurationMs: 21600,
|
||||
timeToFirstTokenMs: 850,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const display = formatMetricsForDisplay(metrics);
|
||||
|
||||
assertEquals(display.includes("anthropic/claude-sonnet-4"), true);
|
||||
assertEquals(display.includes("1,250 tokens"), true);
|
||||
assertEquals(display.includes("482.1 tok/s"), true);
|
||||
assertEquals(display.includes("342 tokens"), true);
|
||||
assertEquals(display.includes("18.3 tok/s"), true);
|
||||
assertEquals(display.includes("1,592 tokens"), true);
|
||||
assertEquals(display.includes("38.0 tok/s"), true);
|
||||
assertEquals(display.includes("21.6s"), true);
|
||||
assertEquals(display.includes("TTFT: 850ms"), true);
|
||||
});
|
||||
|
||||
Deno.test("formatMetricsForDisplay - formats duration as minutes when over 60s", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "openai",
|
||||
model: "gpt-4o",
|
||||
turnCount: 1,
|
||||
inputTokens: 5000,
|
||||
outputTokens: 1000,
|
||||
totalTokens: 6000,
|
||||
prefillTokensPerSec: 50,
|
||||
generationTokensPerSec: 10,
|
||||
combinedTokensPerSec: 60,
|
||||
totalDurationMs: 120000, // 2 minutes
|
||||
timeToFirstTokenMs: 1500,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const display = formatMetricsForDisplay(metrics);
|
||||
|
||||
assertEquals(display.includes("2.0m"), true);
|
||||
});
|
||||
|
||||
Deno.test("formatMetricsForDisplay - omits turn count when single turn", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnCount: 1,
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
totalTokens: 150,
|
||||
prefillTokensPerSec: 20,
|
||||
generationTokensPerSec: 10,
|
||||
combinedTokensPerSec: 30,
|
||||
totalDurationMs: 5000,
|
||||
timeToFirstTokenMs: 500,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const display = formatMetricsForDisplay(metrics);
|
||||
|
||||
assertEquals(display.includes("Turns: 1"), false);
|
||||
});
|
||||
|
||||
Deno.test("formatMetricsForDisplay - omits prefill/generation when TTFT is unavailable", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "openai",
|
||||
model: "gpt-4o",
|
||||
turnCount: 1,
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
totalTokens: 1200,
|
||||
prefillTokensPerSec: 0,
|
||||
generationTokensPerSec: 0,
|
||||
combinedTokensPerSec: 240,
|
||||
totalDurationMs: 5000,
|
||||
timeToFirstTokenMs: undefined,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const display = formatMetricsForDisplay(metrics);
|
||||
|
||||
assertEquals(display.includes("Prefill:"), false);
|
||||
assertEquals(display.includes("Generation:"), false);
|
||||
assertEquals(display.includes("1,200 tokens"), true);
|
||||
assertEquals(display.includes("240.0 tok/s"), true);
|
||||
});
|
||||
|
||||
Deno.test("formatMetricsForDisplay - shows turn count when multiple turns", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnCount: 3,
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
totalTokens: 150,
|
||||
prefillTokensPerSec: 20,
|
||||
generationTokensPerSec: 10,
|
||||
combinedTokensPerSec: 30,
|
||||
totalDurationMs: 5000,
|
||||
timeToFirstTokenMs: 500,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const display = formatMetricsForDisplay(metrics);
|
||||
|
||||
assertEquals(display.includes("Turns: 3"), true);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - uses first valid TTFT when turn-0 has none", () => {
|
||||
// Edge case: turn-0 has no TTFT, turn-1 does. Should use turn-1's TTFT.
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-0",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 3000,
|
||||
// No timeToFirstTokenMs
|
||||
},
|
||||
{
|
||||
turnId: "turn-1",
|
||||
inputTokens: 500,
|
||||
outputTokens: 150,
|
||||
durationMs: 2000,
|
||||
timeToFirstTokenMs: 600,
|
||||
},
|
||||
];
|
||||
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "llama.cpp",
|
||||
model: "Qwen3.6-35B",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
// First valid TTFT is from turn-1 (600ms)
|
||||
assertEquals(result.rawTimestamps?.allTtftMs, [600]);
|
||||
assertEquals(result.rawTimestamps?.ttftMs, 600);
|
||||
// Generation duration = totalDuration - firstValidTTFT = 5000 - 600 = 4400
|
||||
assertEquals(result.rawTimestamps?.generationDurationMs, 4400);
|
||||
// prefill: 1500 / 0.6 = 2500
|
||||
assertEquals(result.prefillTokensPerSec, 2500);
|
||||
// generation: 350 / 4.4 = 79.55
|
||||
assertGreaterOrEqual(result.generationTokensPerSec, 79.5);
|
||||
assertLessOrEqual(result.generationTokensPerSec, 79.6);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - filters out negative TTFT values", () => {
|
||||
// Simulates the bug where turn-2 got TTFT=-20390 from the old global-firstToken code
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-0",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 3000,
|
||||
timeToFirstTokenMs: 800,
|
||||
},
|
||||
{
|
||||
turnId: "turn-1",
|
||||
inputTokens: 500,
|
||||
outputTokens: 150,
|
||||
durationMs: 2000,
|
||||
timeToFirstTokenMs: -5000, // Invalid: negative
|
||||
},
|
||||
];
|
||||
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "llama.cpp",
|
||||
model: "Qwen3.6-35B",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
// Only turn-0's TTFT (800) should be used; turn-1's negative value is filtered
|
||||
assertEquals(result.rawTimestamps?.allTtftMs, [800]);
|
||||
assertEquals(result.rawTimestamps?.ttftMs, 800);
|
||||
// Generation duration = totalDuration - firstTurnTTFT = 5000 - 800 = 4200
|
||||
assertEquals(result.rawTimestamps?.generationDurationMs, 4200);
|
||||
// prefill: 1500 / 0.8 = 1875
|
||||
assertEquals(result.prefillTokensPerSec, 1875);
|
||||
// generation: 350 / 4.2 = 83.33
|
||||
assertGreaterOrEqual(result.generationTokensPerSec, 83.3);
|
||||
assertLessOrEqual(result.generationTokensPerSec, 83.4);
|
||||
});
|
||||
|
||||
Deno.test("toLogEntry - creates JSON-serializable log entry", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnCount: 2,
|
||||
inputTokens: 1250,
|
||||
outputTokens: 342,
|
||||
totalTokens: 1592,
|
||||
prefillTokensPerSec: 482.12345,
|
||||
generationTokensPerSec: 18.34567,
|
||||
combinedTokensPerSec: 38.09876,
|
||||
totalDurationMs: 21600,
|
||||
timeToFirstTokenMs: 850,
|
||||
rawTimestamps: {
|
||||
ttftMs: 850,
|
||||
allTtftMs: [850],
|
||||
generationDurationMs: 20750,
|
||||
turns: [],
|
||||
},
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const logEntry = toLogEntry(metrics);
|
||||
|
||||
assertEquals(logEntry.provider, "anthropic");
|
||||
assertEquals(logEntry.model, "claude-sonnet-4");
|
||||
assertEquals(logEntry.turnCount, 2);
|
||||
assertEquals(logEntry.inputTokens, 1250);
|
||||
assertEquals(logEntry.outputTokens, 342);
|
||||
assertEquals(logEntry.totalTokens, 1592);
|
||||
// Rounded to 2 decimal places
|
||||
assertEquals(logEntry.prefillTokensPerSec, 482.12);
|
||||
assertEquals(logEntry.generationTokensPerSec, 18.35);
|
||||
assertEquals(logEntry.combinedTokensPerSec, 38.1);
|
||||
assertEquals(logEntry.totalDurationMs, 21600);
|
||||
assertEquals(logEntry.timeToFirstTokenMs, 850);
|
||||
|
||||
// Should have ISO timestamp
|
||||
assertEquals(logEntry.timestamp.includes("T"), true);
|
||||
assertEquals(logEntry.timestamp.includes("Z"), true);
|
||||
|
||||
// Should be JSON serializable
|
||||
const json = JSON.stringify(logEntry);
|
||||
assertEquals(json.length > 0, true);
|
||||
const parsed = JSON.parse(json);
|
||||
assertEquals(parsed.provider, "anthropic");
|
||||
|
||||
// rawTimestamps should be included
|
||||
assertEquals(logEntry.rawTimestamps?.ttftMs, 850);
|
||||
assertEquals(logEntry.rawTimestamps?.allTtftMs, [850]);
|
||||
assertEquals(logEntry.rawTimestamps?.generationDurationMs, 20750);
|
||||
assertEquals(logEntry.rawTimestamps?.turns.length, 0);
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - warns when generation speed is physically impossible", () => {
|
||||
const originalWarn = console.warn;
|
||||
let warnCall: string | undefined;
|
||||
console.warn = (msg: string) => { warnCall = msg; };
|
||||
|
||||
try {
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-0",
|
||||
inputTokens: 100,
|
||||
outputTokens: 1000,
|
||||
durationMs: 1000,
|
||||
timeToFirstTokenMs: 100,
|
||||
},
|
||||
];
|
||||
|
||||
aggregatePromptMetrics({
|
||||
provider: "llama.cpp",
|
||||
model: "Qwen3.6-35B",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
// generation: 1000 / 0.9 = 1111.11 tok/s > 500
|
||||
assertGreaterOrEqual(warnCall, "");
|
||||
assertEquals(warnCall!.includes("Suspicious generation speed"), true);
|
||||
assertEquals(warnCall!.includes("1111.1 tok/s"), true);
|
||||
assertEquals(warnCall!.includes("output=1000"), true);
|
||||
} finally {
|
||||
console.warn = originalWarn;
|
||||
}
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - does not warn for normal speeds", () => {
|
||||
const originalWarn = console.warn;
|
||||
let warnCall: string | undefined;
|
||||
console.warn = (msg: string) => { warnCall = msg; };
|
||||
|
||||
try {
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-0",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 5000,
|
||||
timeToFirstTokenMs: 800,
|
||||
},
|
||||
];
|
||||
|
||||
aggregatePromptMetrics({
|
||||
provider: "llama.cpp",
|
||||
model: "Qwen3.6-35B",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
assertEquals(warnCall, undefined);
|
||||
} finally {
|
||||
console.warn = originalWarn;
|
||||
}
|
||||
});
|
||||
|
||||
Deno.test("aggregatePromptMetrics - uses full duration when TTFT is undefined", () => {
|
||||
const turnMetrics: TurnMetrics[] = [
|
||||
{
|
||||
turnId: "turn-1",
|
||||
inputTokens: 1000,
|
||||
outputTokens: 200,
|
||||
durationMs: 5000,
|
||||
// No timeToFirstTokenMs
|
||||
},
|
||||
];
|
||||
|
||||
const result = aggregatePromptMetrics({
|
||||
provider: "openai",
|
||||
model: "gpt-4o",
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
assertEquals(result.turnCount, 1);
|
||||
assertEquals(result.inputTokens, 1000);
|
||||
assertEquals(result.outputTokens, 200);
|
||||
// Without TTFT, prefill and generation rates are 0 (cannot separate phases)
|
||||
// Only combined rate is meaningful
|
||||
assertEquals(result.prefillTokensPerSec, 0);
|
||||
assertEquals(result.generationTokensPerSec, 0);
|
||||
assertEquals(result.combinedTokensPerSec, 240);
|
||||
});
|
||||
|
||||
Deno.test("toLogEntry - handles missing timeToFirstToken", () => {
|
||||
const metrics: PromptMetrics = {
|
||||
provider: "anthropic",
|
||||
model: "claude-sonnet-4",
|
||||
turnCount: 1,
|
||||
inputTokens: 100,
|
||||
outputTokens: 50,
|
||||
totalTokens: 150,
|
||||
prefillTokensPerSec: 20,
|
||||
generationTokensPerSec: 10,
|
||||
combinedTokensPerSec: 30,
|
||||
totalDurationMs: 5000,
|
||||
timeToFirstTokenMs: undefined,
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const logEntry = toLogEntry(metrics);
|
||||
|
||||
assertEquals(logEntry.timeToFirstTokenMs, undefined);
|
||||
});
|
||||
|
||||
Deno.test("Integration - full flow from turns to log entry", () => {
|
||||
// Simulate a real scenario with multiple turns
|
||||
const turn1 = calculateTurnMetrics({
|
||||
turnId: "turn-1",
|
||||
inputTokens: 2000,
|
||||
outputTokens: 500,
|
||||
durationMs: 8000,
|
||||
timeToFirstTokenMs: 1200,
|
||||
});
|
||||
|
||||
const turn2 = calculateTurnMetrics({
|
||||
turnId: "turn-2",
|
||||
inputTokens: 800,
|
||||
outputTokens: 200,
|
||||
durationMs: 3000,
|
||||
});
|
||||
|
||||
const promptMetrics = aggregatePromptMetrics({
|
||||
provider: "groq",
|
||||
model: "llama-3.1-70b",
|
||||
turnMetrics: [turn1, turn2],
|
||||
});
|
||||
|
||||
const display = formatMetricsForDisplay(promptMetrics);
|
||||
const logEntry = toLogEntry(promptMetrics);
|
||||
|
||||
// Verify aggregation
|
||||
assertEquals(promptMetrics.turnCount, 2);
|
||||
assertEquals(promptMetrics.inputTokens, 2800);
|
||||
assertEquals(promptMetrics.outputTokens, 700);
|
||||
assertEquals(promptMetrics.totalTokens, 3500);
|
||||
assertEquals(promptMetrics.totalDurationMs, 11000);
|
||||
assertEquals(promptMetrics.timeToFirstTokenMs, 1200);
|
||||
|
||||
// Verify corrected rate calculations
|
||||
// prefill: 2800 / 1.2 = 2333.33 tok/s
|
||||
assertGreaterOrEqual(promptMetrics.prefillTokensPerSec, 2333.3);
|
||||
assertLessOrEqual(promptMetrics.prefillTokensPerSec, 2333.4);
|
||||
// generation: 700 / 9.8 = 71.43 tok/s
|
||||
assertGreaterOrEqual(promptMetrics.generationTokensPerSec, 71.4);
|
||||
assertLessOrEqual(promptMetrics.generationTokensPerSec, 71.5);
|
||||
// combined: 3500 / 11 = 318.18 tok/s
|
||||
assertGreaterOrEqual(promptMetrics.combinedTokensPerSec, 318.1);
|
||||
assertLessOrEqual(promptMetrics.combinedTokensPerSec, 318.2);
|
||||
|
||||
// Verify display contains key info
|
||||
assertEquals(display.includes("groq/llama-3.1-70b"), true);
|
||||
assertEquals(display.includes("TTFT: 1200ms"), true);
|
||||
|
||||
// Verify log entry
|
||||
assertEquals(logEntry.provider, "groq");
|
||||
assertEquals(logEntry.model, "llama-3.1-70b");
|
||||
assertEquals(logEntry.turnCount, 2);
|
||||
});
|
||||
@@ -0,0 +1,234 @@
|
||||
// Functional core for LLM performance metrics calculation
|
||||
|
||||
// Extracted warning function so tests can mock it without touching console
|
||||
export function warn(msg: string): void {
|
||||
console.warn(msg);
|
||||
}
|
||||
|
||||
export interface TurnMetrics {
|
||||
turnId: string;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
durationMs: number;
|
||||
timeToFirstTokenMs?: number;
|
||||
}
|
||||
|
||||
export interface PromptMetrics {
|
||||
provider: string;
|
||||
model: string;
|
||||
turnCount: number;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
totalTokens: number;
|
||||
prefillTokensPerSec: number;
|
||||
generationTokensPerSec: number;
|
||||
combinedTokensPerSec: number;
|
||||
totalDurationMs: number;
|
||||
timeToFirstTokenMs?: number;
|
||||
rawTimestamps?: {
|
||||
ttftMs?: number;
|
||||
allTtftMs?: number[];
|
||||
generationDurationMs?: number;
|
||||
turns: Array<{ turnId: string; durationMs: number; ttftMs?: number }>;
|
||||
};
|
||||
turns: TurnMetrics[];
|
||||
}
|
||||
|
||||
export interface MetricLogEntry {
|
||||
timestamp: string;
|
||||
provider: string;
|
||||
model: string;
|
||||
turnCount: number;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
totalTokens: number;
|
||||
prefillTokensPerSec: number;
|
||||
generationTokensPerSec: number;
|
||||
combinedTokensPerSec: number;
|
||||
totalDurationMs: number;
|
||||
timeToFirstTokenMs?: number;
|
||||
rawTimestamps?: {
|
||||
ttftMs?: number;
|
||||
allTtftMs?: number[];
|
||||
generationDurationMs?: number;
|
||||
turns: Array<{ turnId: string; durationMs: number; ttftMs?: number }>;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculate metrics for a single turn
|
||||
*/
|
||||
export function calculateTurnMetrics(params: {
|
||||
turnId: string;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
durationMs: number;
|
||||
timeToFirstTokenMs?: number;
|
||||
}): TurnMetrics {
|
||||
return {
|
||||
turnId: params.turnId,
|
||||
inputTokens: params.inputTokens,
|
||||
outputTokens: params.outputTokens,
|
||||
durationMs: params.durationMs,
|
||||
timeToFirstTokenMs: params.timeToFirstTokenMs,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Aggregate multiple turn metrics into prompt-level metrics
|
||||
*/
|
||||
export function aggregatePromptMetrics(params: {
|
||||
provider: string;
|
||||
model: string;
|
||||
turnMetrics: TurnMetrics[];
|
||||
}): PromptMetrics {
|
||||
const { provider, model, turnMetrics } = params;
|
||||
|
||||
if (turnMetrics.length === 0) {
|
||||
return {
|
||||
provider,
|
||||
model,
|
||||
turnCount: 0,
|
||||
inputTokens: 0,
|
||||
outputTokens: 0,
|
||||
totalTokens: 0,
|
||||
prefillTokensPerSec: 0,
|
||||
generationTokensPerSec: 0,
|
||||
combinedTokensPerSec: 0,
|
||||
totalDurationMs: 0,
|
||||
rawTimestamps: { turns: [] },
|
||||
turns: [],
|
||||
};
|
||||
}
|
||||
|
||||
// Sum tokens across all turns
|
||||
const inputTokens = turnMetrics.reduce((sum, t) => sum + t.inputTokens, 0);
|
||||
const outputTokens = turnMetrics.reduce((sum, t) => sum + t.outputTokens, 0);
|
||||
const totalTokens = inputTokens + outputTokens;
|
||||
|
||||
// Sum duration across all turns
|
||||
const totalDurationMs = turnMetrics.reduce((sum, t) => sum + t.durationMs, 0);
|
||||
const totalDurationSec = totalDurationMs / 1000;
|
||||
|
||||
// Collect per-turn TTFTs; prefill boundary is the first turn's TTFT
|
||||
const ttftValues = turnMetrics.map(t => t.timeToFirstTokenMs).filter((t): t is number => t !== undefined && t >= 0);
|
||||
const firstTurnTtftMs = ttftValues.length > 0 ? ttftValues[0] : undefined;
|
||||
|
||||
// Calculate tokens per second
|
||||
// Prefill: input tokens / first-turn TTFT (prefill happens once at the start)
|
||||
// Generation: output tokens / (totalDuration - firstTurnTTFT) (generation phase)
|
||||
// Combined: total tokens / total duration
|
||||
// When first-turn TTFT is unavailable, prefill and generation phases cannot be separated,
|
||||
// so we set them to 0 and only report combined.
|
||||
const ttftSec = firstTurnTtftMs !== undefined ? firstTurnTtftMs / 1000 : undefined;
|
||||
const generationDurationSec = firstTurnTtftMs !== undefined
|
||||
? (totalDurationMs - firstTurnTtftMs) / 1000
|
||||
: undefined;
|
||||
|
||||
const prefillTokensPerSec = (ttftSec && ttftSec > 0) ? inputTokens / ttftSec : 0;
|
||||
const generationTokensPerSec = (generationDurationSec !== undefined && generationDurationSec > 0)
|
||||
? outputTokens / generationDurationSec
|
||||
: 0;
|
||||
const combinedTokensPerSec = totalDurationSec > 0 ? totalTokens / totalDurationSec : 0;
|
||||
|
||||
// Sanity check: flag physically impossible generation speeds
|
||||
if (generationTokensPerSec > 500) {
|
||||
warn(
|
||||
`[metrics] Suspicious generation speed: ${generationTokensPerSec.toFixed(1)} tok/s (input=${inputTokens}, output=${outputTokens}, totalDuration=${totalDurationMs}ms, TTFT=${firstTurnTtftMs}ms)`
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
provider,
|
||||
model,
|
||||
turnCount: turnMetrics.length,
|
||||
inputTokens,
|
||||
outputTokens,
|
||||
totalTokens,
|
||||
prefillTokensPerSec,
|
||||
generationTokensPerSec,
|
||||
combinedTokensPerSec,
|
||||
totalDurationMs,
|
||||
timeToFirstTokenMs: firstTurnTtftMs,
|
||||
rawTimestamps: {
|
||||
ttftMs: firstTurnTtftMs,
|
||||
allTtftMs: ttftValues,
|
||||
generationDurationMs: generationDurationSec !== undefined ? generationDurationSec * 1000 : undefined,
|
||||
turns: turnMetrics.map(t => ({ turnId: t.turnId, durationMs: t.durationMs, ttftMs: t.timeToFirstTokenMs })),
|
||||
},
|
||||
turns: turnMetrics,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Format metrics for TUI display
|
||||
*/
|
||||
export function formatMetricsForDisplay(metrics: PromptMetrics): string {
|
||||
const lines: string[] = [];
|
||||
|
||||
// Header with provider/model
|
||||
lines.push(`📊 Performance: ${metrics.provider}/${metrics.model}`);
|
||||
|
||||
if (metrics.turnCount === 0) {
|
||||
lines.push(" No turns recorded");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
// Format duration display
|
||||
const durationSec = metrics.totalDurationMs / 1000;
|
||||
const durationDisplay = durationSec >= 60
|
||||
? `${(durationSec / 60).toFixed(1)}m`
|
||||
: `${durationSec.toFixed(1)}s`;
|
||||
|
||||
// Prefill metrics (only when TTFT was available)
|
||||
if (metrics.prefillTokensPerSec > 0) {
|
||||
lines.push(
|
||||
` Prefill: ${metrics.inputTokens.toLocaleString()} tokens @ ${metrics.prefillTokensPerSec.toFixed(1)} tok/s`
|
||||
);
|
||||
}
|
||||
|
||||
// Generation metrics (only when TTFT was available)
|
||||
if (metrics.generationTokensPerSec > 0) {
|
||||
lines.push(
|
||||
` Generation: ${metrics.outputTokens.toLocaleString()} tokens @ ${metrics.generationTokensPerSec.toFixed(1)} tok/s`
|
||||
);
|
||||
}
|
||||
|
||||
// Combined metrics
|
||||
lines.push(
|
||||
` Combined: ${metrics.totalTokens.toLocaleString()} tokens @ ${metrics.combinedTokensPerSec.toFixed(1)} tok/s (${durationDisplay} total)`
|
||||
);
|
||||
|
||||
// Time to first token
|
||||
if (metrics.timeToFirstTokenMs !== undefined) {
|
||||
lines.push(` TTFT: ${metrics.timeToFirstTokenMs.toFixed(0)}ms`);
|
||||
}
|
||||
|
||||
// Turn count
|
||||
if (metrics.turnCount > 1) {
|
||||
lines.push(` Turns: ${metrics.turnCount}`);
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert PromptMetrics to JSONL log entry
|
||||
*/
|
||||
export function toLogEntry(metrics: PromptMetrics): MetricLogEntry {
|
||||
return {
|
||||
timestamp: new Date().toISOString(),
|
||||
provider: metrics.provider,
|
||||
model: metrics.model,
|
||||
turnCount: metrics.turnCount,
|
||||
inputTokens: metrics.inputTokens,
|
||||
outputTokens: metrics.outputTokens,
|
||||
totalTokens: metrics.totalTokens,
|
||||
prefillTokensPerSec: Math.round(metrics.prefillTokensPerSec * 100) / 100,
|
||||
generationTokensPerSec: Math.round(metrics.generationTokensPerSec * 100) / 100,
|
||||
combinedTokensPerSec: Math.round(metrics.combinedTokensPerSec * 100) / 100,
|
||||
totalDurationMs: metrics.totalDurationMs,
|
||||
timeToFirstTokenMs: metrics.timeToFirstTokenMs,
|
||||
rawTimestamps: metrics.rawTimestamps,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
// LLM Performance Metrics Extension
|
||||
// Captures and displays LLM inference performance metrics
|
||||
|
||||
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
||||
import { appendFileSync, mkdirSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
// Re-export core functions from the shared metrics module
|
||||
import {
|
||||
calculateTurnMetrics,
|
||||
aggregatePromptMetrics,
|
||||
formatMetricsForDisplay,
|
||||
toLogEntry,
|
||||
type TurnMetrics,
|
||||
type PromptMetrics,
|
||||
type MetricLogEntry,
|
||||
} from "./llm-metrics-core.ts";
|
||||
|
||||
// ============================================================================
|
||||
// Extension Event Handlers (imperative shell)
|
||||
// ============================================================================
|
||||
|
||||
// State tracking
|
||||
let promptStartMs: number | undefined;
|
||||
let currentTurnStartMs: number | undefined;
|
||||
let currentTurnId: string | undefined;
|
||||
let turnMetrics: TurnMetrics[] = [];
|
||||
let currentTurnFirstTokenMs: number | undefined; // Per-turn TTFT
|
||||
let provider: string | undefined;
|
||||
let model: string | undefined;
|
||||
|
||||
export default function (pi: ExtensionAPI) {
|
||||
const logFile = join(process.cwd(), ".pi", "llm-metrics.log");
|
||||
|
||||
pi.on("agent_start", async (_event, ctx) => {
|
||||
if (!ctx.model) return;
|
||||
promptStartMs = Date.now();
|
||||
turnMetrics = [];
|
||||
currentTurnFirstTokenMs = undefined;
|
||||
provider = ctx.model.provider;
|
||||
model = ctx.model.id;
|
||||
});
|
||||
|
||||
pi.on("turn_start", async (event, _ctx) => {
|
||||
currentTurnStartMs = Date.now();
|
||||
currentTurnId = `turn-${event.turnIndex}`;
|
||||
currentTurnFirstTokenMs = undefined; // Reset TTFT for this turn
|
||||
});
|
||||
|
||||
pi.on("message_update", async (event, _ctx) => {
|
||||
// Capture per-turn TTFT on first token
|
||||
if (currentTurnFirstTokenMs === undefined && event.assistantMessageEvent?.type === "text_delta") {
|
||||
currentTurnFirstTokenMs = Date.now();
|
||||
}
|
||||
});
|
||||
|
||||
pi.on("turn_end", async (event, _ctx) => {
|
||||
if (event.message.role !== "assistant") return;
|
||||
const inputTokens = event.message.usage?.input ?? 0;
|
||||
const outputTokens = event.message.usage?.output ?? 0;
|
||||
const durationMs = currentTurnStartMs ? Date.now() - currentTurnStartMs : 0;
|
||||
const ttftMs = currentTurnFirstTokenMs && currentTurnStartMs
|
||||
? currentTurnFirstTokenMs - currentTurnStartMs
|
||||
: undefined;
|
||||
|
||||
const turnMetric = calculateTurnMetrics({
|
||||
turnId: currentTurnId!,
|
||||
inputTokens,
|
||||
outputTokens,
|
||||
durationMs,
|
||||
timeToFirstTokenMs: ttftMs,
|
||||
});
|
||||
|
||||
turnMetrics.push(turnMetric);
|
||||
});
|
||||
|
||||
pi.on("agent_end", async (_event, ctx) => {
|
||||
if (!provider || !model || promptStartMs === undefined) return;
|
||||
|
||||
const promptMetrics = aggregatePromptMetrics({
|
||||
provider,
|
||||
model,
|
||||
turnMetrics,
|
||||
});
|
||||
|
||||
// Display in TUI
|
||||
const display = formatMetricsForDisplay(promptMetrics);
|
||||
ctx.ui.notify(display, "info");
|
||||
ctx.ui.setStatus("metrics", `📊 ${promptMetrics.combinedTokensPerSec.toFixed(1)} tok/s`);
|
||||
|
||||
// Log to JSONL file
|
||||
const logEntry = toLogEntry(promptMetrics);
|
||||
mkdirSync(dirname(logFile), { recursive: true });
|
||||
appendFileSync(logFile, JSON.stringify(logEntry) + "\n", "utf8");
|
||||
|
||||
// Reset state
|
||||
promptStartMs = undefined;
|
||||
turnMetrics = [];
|
||||
currentTurnFirstTokenMs = undefined;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
# pi-notifications
|
||||
|
||||
Audio alerts for pi agent events via `afplay`.
|
||||
|
||||
## What it does
|
||||
|
||||
Plays a sound when the agent finishes a turn, so you can step away and get alerted when input is needed.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Env var | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `PI_NOTIFICATIONS_ENABLED` | `true` | Set to `false` to disable all notifications |
|
||||
| `PI_NOTIFICATION_AGENT_END` | `true` | Play sound when agent finishes |
|
||||
| `PI_NOTIFICATION_AUDIO` | `/System/Library/Sounds/Glass.aiff` | Path to audio file (.aiff/.wav/.mp3) |
|
||||
|
||||
|
||||
## Standalone tester
|
||||
|
||||
Verify audio playback:
|
||||
|
||||
```bash
|
||||
node --input-type=module -e "import {createJiti} from './node_modules/.pnpm/@mariozechner+jiti@2.6.5/node_modules/@mariozechner/jiti/lib/jiti.mjs'; const jiti = createJiti(); await jiti.import('./packages/pi-notifications/src/test-notify.ts');"
|
||||
```
|
||||
|
||||
## Available macOS sounds
|
||||
|
||||
```
|
||||
/System/Library/Sounds/Bottle.aiff
|
||||
/System/Library/Sounds/Cork.aiff
|
||||
/System/Library/Sounds/Frog.aiff
|
||||
/System/Library/Sounds/Glass.aiff ← default
|
||||
/System/Library/Sounds/Hero.aiff
|
||||
/System/Library/Sounds/Morse.aiff
|
||||
/System/Library/Sounds/Ping.aiff
|
||||
/System/Library/Sounds/Pop.aiff
|
||||
/System/Library/Sounds/Submarine.aiff
|
||||
/System/Library/Sounds/Sosumi.aiff
|
||||
/System/Library/Sounds/Tink.aiff
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Add to `~/.pi/agent/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": [
|
||||
"/path/to/packages/pi-notifications"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Then reload pi:
|
||||
|
||||
```bash
|
||||
/reload
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "pi-notifications",
|
||||
"version": "0.1.0",
|
||||
"description": "Desktop notifications for pi agent events",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./src/index.ts"
|
||||
},
|
||||
"keywords": ["pi-package"],
|
||||
"pi": {
|
||||
"extensions": ["src/index.ts"]
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@mariozechner/pi-coding-agent": "*"
|
||||
},
|
||||
"license": "MIT"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
// Desktop notifications for pi agent events
|
||||
// Plays an audio file to alert the user
|
||||
|
||||
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
||||
import { execSync } from "node:child_process";
|
||||
import { existsSync } from "node:fs";
|
||||
|
||||
// Configuration via environment variables
|
||||
const enabled = process.env.PI_NOTIFICATIONS_ENABLED !== "false";
|
||||
const agentEndEnabled = process.env.PI_NOTIFICATION_AGENT_END !== "false";
|
||||
const audioPath = process.env.PI_NOTIFICATION_AUDIO || "/System/Library/Sounds/Glass.aiff";
|
||||
|
||||
function notify(body: string, subtitle?: string): void {
|
||||
if (!enabled) return;
|
||||
try {
|
||||
if (existsSync(audioPath)) {
|
||||
execSync(`afplay "${audioPath}"`, { stdio: "ignore" });
|
||||
}
|
||||
} catch {
|
||||
// audio playback failed — silently fail
|
||||
}
|
||||
}
|
||||
|
||||
export default function (pi: ExtensionAPI) {
|
||||
pi.on("session_start", async (_event, _ctx) => {
|
||||
if (enabled) {
|
||||
notify("pi-notifications active", "Listening for agent_end");
|
||||
}
|
||||
});
|
||||
|
||||
pi.on("agent_end", async (event, _ctx) => {
|
||||
if (!agentEndEnabled) return;
|
||||
|
||||
notify("Agent finished", `${event.messages?.length ?? 0} turns`);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
// Standalone audio tester — run from bash to verify audio playback works
|
||||
// Usage: npx jiti packages/pi-notifications/src/test-notify.ts
|
||||
//
|
||||
// This is completely decoupled from the agent loop.
|
||||
// Use it to verify that audio playback works before debugging event handler wiring.
|
||||
|
||||
import { execSync } from "node:child_process";
|
||||
import { existsSync } from "node:fs";
|
||||
|
||||
const audioPath = process.env.PI_NOTIFICATION_AUDIO || "/System/Library/Sounds/Glass.aiff";
|
||||
|
||||
try {
|
||||
if (!existsSync(audioPath)) {
|
||||
console.error("[test-audio] ❌ Audio file not found:", audioPath);
|
||||
console.error("[test-audio] Set PI_NOTIFICATION_AUDIO to a valid .aiff/.wav/.mp3 path");
|
||||
process.exit(1);
|
||||
}
|
||||
console.log("[test-audio] playing:", audioPath);
|
||||
execSync(`afplay "${audioPath}"`, { stdio: ["ignore", "pipe", "pipe"] });
|
||||
console.log("[test-audio] ✅ Audio played");
|
||||
} catch (e: any) {
|
||||
console.error("[test-audio] ❌ Failed:", e.message);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
# pi-turn-limit
|
||||
|
||||
Pi coding agent extension to limit the number of turns taken by a model in a session.
|
||||
|
||||
## Why
|
||||
|
||||
Pi agents can run indefinitely, consuming tokens and time, and taking focus. This extension gives you a circuit breaker: when the agent reaches a configurable turn limit, you decide whether to continue or abort.
|
||||
|
||||
This is useful for:
|
||||
|
||||
- **Interaction** — Shorter _Time To Next Interaction (TTNI)_ keeps you in the loop. Focus, clarity, single-tasking mode.
|
||||
- **Cost control** — prevent runaway sessions from burning through API credits
|
||||
- **Quality control** — if an agent needs more turns than expected, something may be wrong with the prompt or task
|
||||
|
||||
## How it works
|
||||
|
||||
The extension tracks turns (agent round-trips) per session. Each new user prompt resets the counter.
|
||||
|
||||
| Event | Behavior |
|
||||
|-----------------------|---------------------------------------------------|
|
||||
| `agent_start` | Counter resets to 0 |
|
||||
| `turn_start` | Counter increments; widget updates |
|
||||
| Counter reaches limit | User prompted: continue (counter resets) or abort |
|
||||
| No UI available | Silent abort at limit |
|
||||
|
||||
When the limit is reached and the user confirms continuation, the counter resets to 0, allowing another round of turns. This repeats — there's no maximum number of continuations.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment variable
|
||||
|
||||
Set `PI_MAX_TURNS` before starting pi:
|
||||
|
||||
```bash
|
||||
# 50 turns
|
||||
PI_MAX_TURNS=50 pi
|
||||
|
||||
# Unlimited — no boundary check fires
|
||||
PI_MAX_TURNS=unlimited pi
|
||||
|
||||
# Default (25 turns)
|
||||
pi
|
||||
```
|
||||
|
||||
### Runtime command
|
||||
|
||||
Adjust the limit during a session with the `turn-limit` command:
|
||||
|
||||
```
|
||||
/turn-limit 50 # Set to 50 turns
|
||||
/turn-limit unlimited # Disable the limit
|
||||
/turn-limit 25 # Re-enable with 25 turns (counter resets from 0)
|
||||
```
|
||||
|
||||
### Unlimited mode
|
||||
|
||||
Setting the limit to `unlimited` disables the boundary check entirely. The turn counter still increments and the widget still displays `∞`, so you retain observability without enforcement.
|
||||
|
||||
Switching from unlimited back to a number triggers the **LimitReEnabled** rule: the counter resets to 0 so the new limit applies from a clean starting point.
|
||||
|
||||
## Widget
|
||||
|
||||
When a UI is available, the extension displays a widget showing `Turns: N/M` (or `Turns: N/∞` in unlimited mode).
|
||||
|
||||
## Spec
|
||||
|
||||
This extension was developed using [Allium](https://juxt.github.io/allium/), a formal language for specifying software behavior. The Allium spec captures the turn counting, limit enforcement, unlimited mode, and re-enable rules:
|
||||
|
||||
📄 [turn-limit.allium](./turn-limit.allium)
|
||||
|
||||
## Contributing
|
||||
|
||||
This is an experimental extension. To contribute:
|
||||
|
||||
1. Fork the repository
|
||||
2. Make your changes
|
||||
3. Contact Willem through [qwan.co.uk](https://qwan.co.uk) to discuss your contribution
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,484 @@
|
||||
import { describe, it, beforeEach, afterEach, mock } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { checkTurnLimit, getMaxTurns } from "./turn-limit.ts";
|
||||
import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
||||
import initExtension from "./turn-limit.ts";
|
||||
|
||||
// ============================================================================
|
||||
// P: Pure function tests — checkTurnLimit
|
||||
// ============================================================================
|
||||
|
||||
describe("checkTurnLimit", () => {
|
||||
it("P1: below limit returns exceeded=false", () => {
|
||||
const result = checkTurnLimit(20, 25);
|
||||
assert.equal(result.exceeded, false);
|
||||
assert.equal(result.turnIndex, 20);
|
||||
assert.equal(result.maxTurns, 25);
|
||||
});
|
||||
|
||||
it("P2: at limit returns exceeded=false (strict >)", () => {
|
||||
const result = checkTurnLimit(25, 25);
|
||||
assert.equal(result.exceeded, false);
|
||||
});
|
||||
|
||||
it("P3: above limit returns exceeded=true", () => {
|
||||
const result = checkTurnLimit(26, 25);
|
||||
assert.equal(result.exceeded, true);
|
||||
});
|
||||
|
||||
it("P4: zero max — turnIndex 1 exceeds", () => {
|
||||
const result = checkTurnLimit(1, 0);
|
||||
assert.equal(result.exceeded, true);
|
||||
});
|
||||
|
||||
it("P5: zero turn — below any positive limit", () => {
|
||||
const result = checkTurnLimit(0, 25);
|
||||
assert.equal(result.exceeded, false);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// C: Config tests — getMaxTurns
|
||||
// ============================================================================
|
||||
|
||||
describe("getMaxTurns", () => {
|
||||
let originalEnv: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
originalEnv = process.env.PI_MAX_TURNS;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalEnv === undefined) {
|
||||
delete process.env.PI_MAX_TURNS;
|
||||
} else {
|
||||
process.env.PI_MAX_TURNS = originalEnv;
|
||||
}
|
||||
});
|
||||
|
||||
it("C1: returns 25 when PI_MAX_TURNS is unset", () => {
|
||||
delete process.env.PI_MAX_TURNS;
|
||||
assert.equal(getMaxTurns(), 25);
|
||||
});
|
||||
|
||||
it("C2: returns env value when PI_MAX_TURNS is valid", () => {
|
||||
process.env.PI_MAX_TURNS = "10";
|
||||
assert.equal(getMaxTurns(), 10);
|
||||
});
|
||||
|
||||
it("C3: falls back to 25 for non-numeric PI_MAX_TURNS", () => {
|
||||
process.env.PI_MAX_TURNS = "abc";
|
||||
assert.equal(getMaxTurns(), 25);
|
||||
});
|
||||
|
||||
it("C4: falls back to 25 for PI_MAX_TURNS=0", () => {
|
||||
process.env.PI_MAX_TURNS = "0";
|
||||
assert.equal(getMaxTurns(), 25);
|
||||
});
|
||||
|
||||
it("C5: falls back to 25 for negative PI_MAX_TURNS", () => {
|
||||
process.env.PI_MAX_TURNS = "-5";
|
||||
assert.equal(getMaxTurns(), 25);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// Helpers for integration tests
|
||||
// ============================================================================
|
||||
|
||||
type EventHandler = (event: any, ctx: any) => Promise<void>;
|
||||
type CommandHandler = { description: string; handler: (args: string, ctx: any) => Promise<void> };
|
||||
|
||||
function createMockPi() {
|
||||
const handlers: Record<string, EventHandler> = {};
|
||||
const commands: Record<string, CommandHandler> = {};
|
||||
|
||||
const pi = {
|
||||
on: (event: string, handler: EventHandler) => {
|
||||
handlers[event] = handler;
|
||||
},
|
||||
registerCommand: (name: string, cmd: CommandHandler) => {
|
||||
commands[name] = cmd;
|
||||
},
|
||||
} as unknown as ExtensionAPI;
|
||||
|
||||
return { pi, handlers, commands };
|
||||
}
|
||||
|
||||
function createMockCtx(options: { hasUI?: boolean; confirmResult?: boolean } = {}) {
|
||||
const { hasUI = true, confirmResult = true } = options;
|
||||
const calls: { method: string; args: any[] }[] = [];
|
||||
|
||||
return {
|
||||
ctx: {
|
||||
hasUI,
|
||||
abort: mock.fn(() => { calls.push({ method: "abort", args: [] }); }),
|
||||
ui: {
|
||||
setWidget: mock.fn((...args: any[]) => { calls.push({ method: "setWidget", args }); }),
|
||||
notify: mock.fn((...args: any[]) => { calls.push({ method: "notify", args }); }),
|
||||
confirm: mock.fn(async () => { calls.push({ method: "confirm", args: [] }); return confirmResult; }),
|
||||
},
|
||||
},
|
||||
calls,
|
||||
};
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// E: Entity & state tests
|
||||
// ============================================================================
|
||||
|
||||
describe("extension handler — entity state", () => {
|
||||
it("E1: counter starts at 0 — widget shows 0/25 on agent_start", async () => {
|
||||
const { pi, handlers } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
assert.equal(ctx.ui.setWidget.mock.callCount(), 1);
|
||||
assert.deepEqual(ctx.ui.setWidget.mock.calls[0].arguments, ["turn-limit", ["Turns: 0/25"]]);
|
||||
});
|
||||
|
||||
it("E2: counter increments each turn_start", async () => {
|
||||
const { pi, handlers } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
// After 1 turn: widget should show 1/25
|
||||
const lastCall = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastCall!.arguments, ["turn-limit", ["Turns: 1/25"]]);
|
||||
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
const lastCall2 = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastCall2!.arguments, ["turn-limit", ["Turns: 2/25"]]);
|
||||
});
|
||||
|
||||
it("E3: agent_start resets counter", async () => {
|
||||
const { pi, handlers } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
// New agent_start resets
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
const lastCall = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastCall!.arguments, ["turn-limit", ["Turns: 0/25"]]);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// R: Rule tests — TurnLimitReached
|
||||
// ============================================================================
|
||||
|
||||
describe("extension handler — TurnLimitReached rule", () => {
|
||||
it("R1: at limit, user confirms → counter resets to 0", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
|
||||
// Set max turns to 3 for faster testing
|
||||
const { ctx } = createMockCtx({ confirmResult: true });
|
||||
await commands["turn-limit"].handler("3", ctx);
|
||||
|
||||
// Fire 3 turns — third should trigger confirmation
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
// Confirm was called
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 1);
|
||||
// Counter reset — widget shows 0/3
|
||||
const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastWidget!.arguments, ["turn-limit", ["Turns: 0/3"]]);
|
||||
// abort was NOT called
|
||||
assert.equal(ctx.abort.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
it("R2: at limit, user declines → session aborts", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
|
||||
const { ctx } = createMockCtx({ confirmResult: false });
|
||||
await commands["turn-limit"].handler("2", ctx);
|
||||
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 1);
|
||||
assert.equal(ctx.abort.mock.callCount(), 1);
|
||||
// Notify about abort
|
||||
const notifyCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[0] === "Agent aborted by user."
|
||||
);
|
||||
assert.equal(notifyCalls.length, 1);
|
||||
});
|
||||
|
||||
it("R3: at limit, no UI → silent abort", async () => {
|
||||
const { pi, handlers } = createMockPi();
|
||||
initExtension(pi);
|
||||
|
||||
// Set max to 1 via env before init — but we already initialized.
|
||||
// Instead, use registerCommand with a UI ctx first, then turn_start with no UI.
|
||||
const uiCtx = createMockCtx().ctx;
|
||||
await handlers["agent_start"]({}, uiCtx);
|
||||
|
||||
// Use a separate noUI ctx for the turn
|
||||
const { ctx: noUiCtx } = createMockCtx({ hasUI: false });
|
||||
|
||||
// We need max=1 — use the command to set it
|
||||
// But the command uses ctx.ui.setWidget which won't work without UI...
|
||||
// The command still sets maxTurns regardless
|
||||
const cmdCtx = createMockCtx().ctx;
|
||||
const { commands } = createMockPi();
|
||||
|
||||
// Re-init to get fresh state with command access
|
||||
const pi2Mock = createMockPi();
|
||||
initExtension(pi2Mock.pi);
|
||||
|
||||
const setCmdCtx = createMockCtx().ctx;
|
||||
await pi2Mock.commands["turn-limit"].handler("1", setCmdCtx);
|
||||
|
||||
const noUi = createMockCtx({ hasUI: false }).ctx;
|
||||
await pi2Mock.handlers["turn_start"]({}, noUi);
|
||||
|
||||
assert.equal(noUi.abort.mock.callCount(), 1);
|
||||
// confirm should NOT have been called (no UI)
|
||||
assert.equal(noUi.ui.confirm.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
it("R4: below limit → no prompt, no abort", async () => {
|
||||
const { pi, handlers } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 0);
|
||||
assert.equal(ctx.abort.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
it("R5: after reset, counting continues from 0", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx({ confirmResult: true });
|
||||
|
||||
await commands["turn-limit"].handler("2", ctx);
|
||||
|
||||
// First round: 2 turns → confirm → reset
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 1);
|
||||
|
||||
// Second round: 2 more turns → confirm again
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 2);
|
||||
|
||||
// Counter reset again
|
||||
const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastWidget!.arguments, ["turn-limit", ["Turns: 0/2"]]);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// C6-C8: Command tests
|
||||
// ============================================================================
|
||||
|
||||
describe("extension handler — turn-limit command", () => {
|
||||
it("C6: valid arg updates maxTurns", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("10", ctx);
|
||||
|
||||
const notifyCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[0] === "Turn limit set to 10."
|
||||
);
|
||||
assert.equal(notifyCalls.length, 1);
|
||||
// Widget updated with new max
|
||||
const widgetCall = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(widgetCall!.arguments, ["turn-limit", ["Turns: 0/10"]]);
|
||||
});
|
||||
|
||||
it("C7: invalid arg shows error", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("abc", ctx);
|
||||
|
||||
const errorCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[1] === "error"
|
||||
);
|
||||
assert.equal(errorCalls.length, 1);
|
||||
});
|
||||
|
||||
it("C8: empty args shows error", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("", ctx);
|
||||
|
||||
const errorCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[1] === "error"
|
||||
);
|
||||
assert.equal(errorCalls.length, 1);
|
||||
});
|
||||
});
|
||||
|
||||
// ============================================================================
|
||||
// Unlimited / disable feature tests
|
||||
// ============================================================================
|
||||
|
||||
describe("unlimited mode — config", () => {
|
||||
let originalEnv: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
originalEnv = process.env.PI_MAX_TURNS;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalEnv === undefined) {
|
||||
delete process.env.PI_MAX_TURNS;
|
||||
} else {
|
||||
process.env.PI_MAX_TURNS = originalEnv;
|
||||
}
|
||||
});
|
||||
|
||||
it("CFG-UNLIM-1: getMaxTurns returns Infinity for PI_MAX_TURNS=unlimited", () => {
|
||||
process.env.PI_MAX_TURNS = "unlimited";
|
||||
assert.equal(getMaxTurns(), Infinity);
|
||||
});
|
||||
});
|
||||
|
||||
describe("unlimited mode — command", () => {
|
||||
it("CMD-UNLIM-1: 'turn-limit unlimited' is accepted", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
|
||||
// Should NOT show error
|
||||
const errorCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[1] === "error"
|
||||
);
|
||||
assert.equal(errorCalls.length, 0);
|
||||
});
|
||||
|
||||
it("CMD-UNLIM-2: 'turn-limit unlimited' notifies user", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
|
||||
const infoCalls = ctx.ui.notify.mock.calls.filter(
|
||||
(c) => c.arguments[1] === "info"
|
||||
);
|
||||
assert.equal(infoCalls.length, 1);
|
||||
assert.match(infoCalls[0].arguments[0] as string, /unlimited/i);
|
||||
});
|
||||
|
||||
it("CMD-UNLIM-3: after 'turn-limit unlimited', widget shows ∞", async () => {
|
||||
const { pi, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
|
||||
const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.match((lastWidget!.arguments[1] as string[])[0], /∞/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("unlimited mode — no boundary check fires", () => {
|
||||
it("RUL-UNLIM-2: unlimited mode — no confirmation or abort after many turns", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
|
||||
// Fire 50 turns — none should trigger confirmation or abort
|
||||
for (let i = 0; i < 50; i++) {
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
}
|
||||
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 0);
|
||||
assert.equal(ctx.abort.mock.callCount(), 0);
|
||||
});
|
||||
|
||||
it("RUL-UNLIM-3: unlimited mode — counter still increments", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
// Widget should show counter incrementing with ∞
|
||||
const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.match((lastWidget!.arguments[1] as string[])[0], /3/);
|
||||
assert.match((lastWidget!.arguments[1] as string[])[0], /∞/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("unlimited mode — switching back to limited (LimitReEnabled)", () => {
|
||||
it("CMD-INT-1: switching from unlimited to number resets counter", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx();
|
||||
|
||||
// Set unlimited
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
|
||||
// Do several turns
|
||||
for (let i = 0; i < 10; i++) {
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
}
|
||||
|
||||
// Switch back to limited
|
||||
await commands["turn-limit"].handler("5", ctx);
|
||||
|
||||
// Counter should be reset to 0, widget shows 0/5
|
||||
const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
assert.deepEqual(lastWidget!.arguments, ["turn-limit", ["Turns: 0/5"]]);
|
||||
});
|
||||
|
||||
it("INT-1: unlimited → switch to 3 → boundary fires at turn 3", async () => {
|
||||
const { pi, handlers, commands } = createMockPi();
|
||||
initExtension(pi);
|
||||
const { ctx } = createMockCtx({ confirmResult: true });
|
||||
|
||||
// Set unlimited, do turns
|
||||
await commands["turn-limit"].handler("unlimited", ctx);
|
||||
await handlers["agent_start"]({}, ctx);
|
||||
for (let i = 0; i < 10; i++) {
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
}
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 0);
|
||||
|
||||
// Switch to limit=3 → counter resets
|
||||
await commands["turn-limit"].handler("3", ctx);
|
||||
|
||||
// Now 3 turns should trigger confirmation
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
await handlers["turn_start"]({}, ctx);
|
||||
|
||||
assert.equal(ctx.ui.confirm.mock.callCount(), 1);
|
||||
});
|
||||
});
|
||||
@@ -6,13 +6,18 @@ import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
|
||||
|
||||
const DEFAULT_MAX_TURNS = 25;
|
||||
|
||||
function getMaxTurns(): number {
|
||||
export function getMaxTurns(): number {
|
||||
const env = process.env.PI_MAX_TURNS;
|
||||
if (!env) return DEFAULT_MAX_TURNS;
|
||||
if (env.trim().toLowerCase() === "unlimited") return Infinity;
|
||||
const parsed = parseInt(env, 10);
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : DEFAULT_MAX_TURNS;
|
||||
}
|
||||
|
||||
function formatMax(maxTurns: number): string {
|
||||
return maxTurns === Infinity ? "∞" : String(maxTurns);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Pure detection logic (testable)
|
||||
// ============================================================================
|
||||
@@ -39,7 +44,7 @@ export default function (pi: ExtensionAPI) {
|
||||
pi.on("session_start", async (event, ctx) => {
|
||||
// On reload, show the widget immediately
|
||||
if (event.reason === "reload" && ctx.hasUI) {
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${maxTurns}`]);
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -48,7 +53,7 @@ export default function (pi: ExtensionAPI) {
|
||||
turnCount = 0;
|
||||
// Show initial widget state on fresh session
|
||||
if (ctx.hasUI) {
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${maxTurns}`]);
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -57,16 +62,26 @@ export default function (pi: ExtensionAPI) {
|
||||
handler: async (args: string, ctx) => {
|
||||
const trimmed = args.trim();
|
||||
if (!trimmed) {
|
||||
ctx.ui.notify("Invalid turn limit. Must be a positive integer.", "error");
|
||||
ctx.ui.notify("Invalid turn limit. Must be a positive integer or 'unlimited'.", "error");
|
||||
return;
|
||||
}
|
||||
if (trimmed.toLowerCase() === "unlimited") {
|
||||
maxTurns = Infinity;
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
ctx.ui.notify("Turn limit set to unlimited.", "info");
|
||||
return;
|
||||
}
|
||||
const parsed = parseInt(trimmed, 10);
|
||||
if (!Number.isFinite(parsed) || parsed <= 0) {
|
||||
ctx.ui.notify("Invalid turn limit. Must be a positive integer.", "error");
|
||||
ctx.ui.notify("Invalid turn limit. Must be a positive integer or 'unlimited'.", "error");
|
||||
return;
|
||||
}
|
||||
const wasUnlimited = maxTurns === Infinity;
|
||||
maxTurns = parsed;
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${maxTurns}`]);
|
||||
if (wasUnlimited) {
|
||||
turnCount = 0;
|
||||
}
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
ctx.ui.notify(`Turn limit set to ${parsed}.`, "info");
|
||||
},
|
||||
});
|
||||
@@ -76,9 +91,12 @@ export default function (pi: ExtensionAPI) {
|
||||
|
||||
// Update live widget
|
||||
if (ctx.hasUI) {
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${maxTurns}`]);
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
}
|
||||
|
||||
// No boundary check when unlimited
|
||||
if (maxTurns === Infinity) return;
|
||||
|
||||
// Boundary confirmation: when we hit maxTurns exactly
|
||||
if (turnCount === maxTurns) {
|
||||
if (ctx.hasUI) {
|
||||
@@ -90,7 +108,7 @@ export default function (pi: ExtensionAPI) {
|
||||
// Reset counter and let the turn proceed
|
||||
turnCount = 0;
|
||||
if (ctx.hasUI) {
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${maxTurns}`]);
|
||||
ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${formatMax(maxTurns)}`]);
|
||||
}
|
||||
return;
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
-- allium: 3
|
||||
-- turn-limit.allium
|
||||
|
||||
-- Scope: Agent turn limit enforcement per session
|
||||
-- Includes: Turn counting, limit enforcement, session abort, disable/enable
|
||||
-- Excludes:
|
||||
-- - Widget display (UI implementation detail)
|
||||
-- - Environment variable reading (configuration mechanism)
|
||||
-- - turn-limit command (configuration mechanism)
|
||||
|
||||
------------------------------------------------------------
|
||||
-- Entities
|
||||
------------------------------------------------------------
|
||||
|
||||
entity Session {
|
||||
turn_count: Integer
|
||||
status: active | aborted
|
||||
user_confirms_continuation: Boolean?
|
||||
|
||||
transitions status {
|
||||
active -> aborted
|
||||
terminal: aborted
|
||||
}
|
||||
}
|
||||
|
||||
------------------------------------------------------------
|
||||
-- Config
|
||||
------------------------------------------------------------
|
||||
|
||||
config {
|
||||
max_turns: Integer | unlimited = 25
|
||||
|
||||
@guidance
|
||||
-- When max_turns is unlimited, no boundary check fires.
|
||||
-- The turn counter still increments for observability.
|
||||
-- Transitioning from unlimited to a positive integer
|
||||
-- resets turn_count to 0.
|
||||
}
|
||||
|
||||
------------------------------------------------------------
|
||||
-- Rules
|
||||
------------------------------------------------------------
|
||||
|
||||
rule TurnLimitReached {
|
||||
when: session: Session.turn_count transitions_to config.max_turns
|
||||
requires:
|
||||
config.max_turns != unlimited
|
||||
session.turn_count = config.max_turns
|
||||
|
||||
ensures:
|
||||
if session.user_confirms_continuation:
|
||||
session.turn_count = 0
|
||||
else:
|
||||
session.status = aborted
|
||||
|
||||
@guidance
|
||||
-- The user_confirms_continuation field is set by the
|
||||
-- implementation when presenting a confirmation prompt to
|
||||
-- the user at the turn limit boundary. The implementation
|
||||
-- may use a widget, dialog, or other mechanism to capture
|
||||
-- the user's choice.
|
||||
--
|
||||
-- When the user confirms, the turn_count resets to zero,
|
||||
-- allowing the agent to continue for another round of turns.
|
||||
-- When the user declines (field is null or false), the
|
||||
-- session is aborted.
|
||||
--
|
||||
-- Without a UI, the default behaviour is to abort.
|
||||
}
|
||||
|
||||
rule LimitReEnabled {
|
||||
when: config.max_turns transitions_to Integer
|
||||
|
||||
ensures:
|
||||
session.turn_count = 0
|
||||
|
||||
@guidance
|
||||
-- When the user switches from unlimited back to a positive
|
||||
-- integer limit, the turn counter resets to zero so the
|
||||
-- new limit applies from a clean starting point.
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
# Plan: Analyze & Fix `llm-metrics` Extension Timing Bug
|
||||
|
||||
## Problem Statement
|
||||
The extension reports generation speed as ~8,000–2,400 tok/s (physically impossible) while prefill speed is ~70 tok/s. The math is internally consistent but the underlying phase boundaries are inverted or misaligned. Real generation speed is ~53–70 tok/s (confirmed by earlier runs).
|
||||
|
||||
## Phase 1: Locate & Map the Extension
|
||||
1. **Find the source code**
|
||||
- Search `~/.pi/extensions/`, `~/.pi/tools/`, and the pi-coding-agent package for files matching `llm`, `metric`, `performance`, `benchmark`
|
||||
- Check `~/.pi/config` or project `.pi/config` for extension/tool registration
|
||||
- Look for custom tool definitions in `extensions/`, `tools/`, or `skills/` directories
|
||||
2. **Identify the provider integration**
|
||||
- The log shows `"provider":"llama.cpp"` — find where the extension hooks into llama.cpp (likely via subprocess, WebSocket, or callback interception)
|
||||
- Map the data flow: raw llama.cpp output → extension parsing → JSON log writing
|
||||
|
||||
## Phase 2: Diagnose the Timing Bug
|
||||
3. **Trace phase boundary detection**
|
||||
- Find how the extension defines "prefill" vs "generation" start/end times
|
||||
- Check if it uses:
|
||||
- `timeToFirstToken` (TTFT) as the split point
|
||||
- llama.cpp callback hooks (`completion_token_callback`, `prompt_token_callback`)
|
||||
- Wall-clock timestamps around token streaming
|
||||
4. **Verify the calculation**
|
||||
- Confirm the formula: `generationTok/s = outputTokens / (totalDuration - TTFT)`
|
||||
- Check if `totalDuration` includes only generation, or the full call
|
||||
- Look for race conditions: async callbacks firing out of order, or generation end timestamp captured before all tokens are flushed
|
||||
5. **Reproduce the anomaly**
|
||||
- Run the same model with identical prompt/output length
|
||||
- Compare TTFT, totalDuration, and per-phase timestamps
|
||||
- Check if the bug appears only with large prompts, speculative decoding, or certain sampling configs
|
||||
|
||||
## Phase 3: Fix the Implementation
|
||||
6. **Correct phase boundaries**
|
||||
- If using callbacks: ensure generation start = TTFT timestamp, generation end = last token callback or explicit `done` event
|
||||
- If using wall-clock: add a small buffer after last token to account for async flush
|
||||
- Add validation: reject generation speeds > 500 tok/s (sanity check)
|
||||
7. **Fix label assignment**
|
||||
- Ensure `prefillTokensPerSec` = `inputTokens / TTFT`
|
||||
- Ensure `generationTokensPerSec` = `outputTokens / (totalDuration - TTFT)`
|
||||
- Add explicit phase logging to debug output
|
||||
8. **Add telemetry**
|
||||
- Log raw timestamps: `prefill_start`, `prefill_end`, `gen_start`, `gen_end`, `total_start`, `total_end`
|
||||
- Log per-phase token counts to catch mismatches
|
||||
- Write to `.pi/llm-metrics.log` with consistent schema
|
||||
|
||||
## Phase 4: Verify & Deploy
|
||||
9. **Test cases**
|
||||
- Small prompt + short output (baseline)
|
||||
- Large prompt + long output (original failure case)
|
||||
- Speculative decoding run (if supported)
|
||||
- Early termination / stop token edge case
|
||||
10. **Validate output**
|
||||
- Generation speed should be 40–100 tok/s for this model/hardware
|
||||
- Prefill speed should be 50–200 tok/s (parallel compute)
|
||||
- TTFT should match prefill duration
|
||||
- No negative phase durations
|
||||
11. **Update schema & docs**
|
||||
- Add `rawTimestamps` field to log entries for debugging
|
||||
- Document phase definitions in extension README
|
||||
- Add unit tests for metric calculation logic
|
||||
|
||||
## Deliverables
|
||||
- [ ] Extension source located & data flow mapped
|
||||
- [ ] Root cause identified (callback timing gap, phase boundary misassignment, or async flush race)
|
||||
- [ ] Fix implemented with sanity checks
|
||||
- [ ] Test suite covering edge cases
|
||||
- [ ] Log schema updated with raw timestamps
|
||||
- [ ] PR or patch ready for review
|
||||
|
||||
## Questions to Answer During Analysis
|
||||
- Does the extension intercept llama.cpp at the C++ level, via CLI, or through a Python wrapper?
|
||||
- Are callbacks synchronous or async?
|
||||
- Is there a `done`/`end` event, or does it rely on empty token streams?
|
||||
- Could speculative decoding be causing the draft model's batched verification to be misclassified as "generation"?
|
||||
@@ -0,0 +1,154 @@
|
||||
# Plan: pi-notifications v0 — Desktop Notifications for Agent Events
|
||||
|
||||
## Goal
|
||||
|
||||
Make the `pi-notifications` extension reliably show macOS Notification Center alerts when the agent finishes a turn, so the user gets alerted without needing to watch the screen.
|
||||
|
||||
## Current State
|
||||
|
||||
- Extension exists at `packages/pi-notifications/src/index.ts` (monorepo) and `~/.pi/agent/extensions/pi-notifications.ts` (auto-discovery)
|
||||
- Extension loads correctly (appears in `/reload` extension list)
|
||||
- `console.log` from extensions is NOT visible in `/reload` output
|
||||
- `osascript` works when run directly in bash, but notification doesn't appear when called from the extension
|
||||
- The `session_start` handler fires on reload, `agent_end` fires when prompts complete
|
||||
|
||||
## Debugging Strategy (split into two orthogonal problems)
|
||||
|
||||
### Problem A: "Does the trigger fire?" — visible debug signal
|
||||
|
||||
`console.log` from extensions is invisible in pi's TUI output. To debug the trigger logic in a fast loop, add a **debug mode** (`PI_NOTIFICATION_DEBUG=true`) that emits a visible signal via `ctx.ui.steer()` (or similar) right before calling `notify()`. This surfaces in the chat/TUI so you can verify the handler fires without needing actual desktop notifications.
|
||||
|
||||
### Problem B: "Does `osascript` actually deliver?" — isolated tester
|
||||
|
||||
Create a standalone script (`test-notify.ts`) that you run from bash independently of the agent loop. This verifies `osascript` works in the extension's import context, decoupled from event handlers.
|
||||
|
||||
### 1. Verify `osascript` works in extension context
|
||||
|
||||
The extension uses `execSync` from `node:child_process`. Test that it works inside the extension:
|
||||
|
||||
```typescript
|
||||
// In the extension, add this to session_start handler:
|
||||
try {
|
||||
const output = execSync('osascript -e "display notification \\"test\\" with title \\"test\\""').toString();
|
||||
console.log("[pi-notifications] osascript output:", output);
|
||||
} catch (e: any) {
|
||||
console.log("[pi-notifications] osascript error:", e.message, e.stderr?.toString());
|
||||
}
|
||||
```
|
||||
|
||||
If `execSync` fails silently, try:
|
||||
- Using `{ stdio: ["pipe", "pipe", "pipe"] }` to capture stderr
|
||||
- Checking if `node:child_process` is available in the extension sandbox
|
||||
|
||||
### 2. Check macOS notification settings
|
||||
|
||||
Notifications may be delivered but not shown as banners:
|
||||
- **System Settings → Notifications → Ghostty → Notification Style** — must be "Banners" or "Alerts", not "None" (osascript fires from the Ghostty process, so macOS attributes notifications to Ghostty, not "pi")
|
||||
- **System Settings → Focus → [active focus] → Apps** — ensure "Ghostty" is not excluded
|
||||
- **System Settings → Notifications → Show Notifications on Lock Screen** — enable if needed
|
||||
|
||||
**Known symptom:** Notifications appear in Notification Center when pulled down, but never pop up as banners. This is a macOS style setting, not a code issue.
|
||||
|
||||
### 3. Ghostty suppresses banners when focused
|
||||
|
||||
Ghostty intentionally silences banner notifications (no pop-up, no sound) when the Ghostty window is **active/focused**. The notification is still delivered to Notification Center. Banners only appear when Ghostty is **not** the active window.
|
||||
|
||||
**Workarounds:**
|
||||
- **System Settings → Notifications → Ghostty → Alert Style → "Persistent"** — macOS shows these as banners regardless of Ghostty's silencing
|
||||
- **Switch to another app** (e.g. leave your browser open) when you want to see the banner
|
||||
|
||||
### 3. Try alternative notification methods
|
||||
|
||||
If `osascript` doesn't work from the extension, try:
|
||||
- `notify-send` (Linux-only, not relevant for macOS)
|
||||
- A custom TUI widget that shows a persistent banner
|
||||
- Using `ctx.ui.notify()` (but this only shows in pi's TUI, not system notification)
|
||||
|
||||
### 4. Verify event handlers fire
|
||||
|
||||
Add a `session_start` handler that definitely fires:
|
||||
|
||||
```typescript
|
||||
pi.on("session_start", async (_event, ctx) => {
|
||||
console.log("[pi-notifications] session_start fired");
|
||||
ctx.ui.notify("pi-notifications active", "info"); // Shows in TUI
|
||||
});
|
||||
```
|
||||
|
||||
If `ctx.ui.notify()` works but `osascript` doesn't, the issue is macOS notification permissions, not the extension.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Step 0A: Add debug mode with visible signal (PI_NOTIFICATION_DEBUG)
|
||||
|
||||
Add a `PI_NOTIFICATION_DEBUG=true` env var. When enabled, the extension calls `ctx.ui.steer()` (or a visible TUI signal) right before each notification, so you see "notification triggered" in the chat output during the agent loop. This lets you verify trigger logic without needing actual desktop notifications.
|
||||
|
||||
- In `agent_end` handler: if `PI_NOTIFICATION_DEBUG=true`, call `ctx.ui.steer("[pi-notifications] notification triggered")` before `notify()`
|
||||
- In `session_start` handler: same pattern
|
||||
- This is purely for debugging — no desktop notification shown when debug is on (or both are shown)
|
||||
|
||||
### Step 0B: Create isolated notification tester
|
||||
|
||||
Create `packages/pi-notifications/src/test-notify.ts` — a standalone script runnable via `npx jiti` that fires a test notification. Run it from bash to verify `osascript` works in the extension's context, completely separate from the agent loop.
|
||||
|
||||
### Step 1: Fix notification delivery (priority)
|
||||
|
||||
Once the root cause is identified:
|
||||
|
||||
**If `osascript` works but notification is suppressed:**
|
||||
- Add a `PI_NOTIFICATION_SOUND` env var (already in design)
|
||||
- Add `PI_NOTIFICATIONS_ENABLED` toggle (already in design)
|
||||
- Consider adding a "first-run" notification that asks user to enable notifications
|
||||
|
||||
**If `osascript` doesn't work from extension:**
|
||||
- Fall back to `ctx.ui.notify()` which shows in pi's TUI
|
||||
- Or use a different approach (e.g., write to a file that a separate process monitors)
|
||||
|
||||
### Step 2: Add turn-limit notification
|
||||
|
||||
In `packages/pi-turn-limit/src/turn-limit.ts`, add notification when the limit is reached:
|
||||
|
||||
```typescript
|
||||
// In the turn-limit extension, when the limit fires:
|
||||
if (shouldNotify) {
|
||||
execSync('osascript -e \'display notification "Turn limit reached" with title "pi" subtitle "Turns: ' + turnCount + '/' + maxTurns + '"\'');
|
||||
}
|
||||
```
|
||||
|
||||
Configuration via env var:
|
||||
- `PI_NOTIFICATION_TURN_LIMIT` — default `true`, set to `false` to disable
|
||||
|
||||
### Step 3: Add sound option
|
||||
|
||||
Already designed in the extension:
|
||||
- `PI_NOTIFICATION_SOUND` env var (default: `default`)
|
||||
- macOS sounds: `Bottle`, `Ping`, `Pop`, `Submarine`, `Sosumi`, `Tink`
|
||||
- Set to `""` for silent
|
||||
|
||||
### Step 4: Update README
|
||||
|
||||
Document the extension with:
|
||||
- What it does
|
||||
- Configuration options
|
||||
- How to enable macOS notifications
|
||||
- Troubleshooting tips
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `~/.pi/agent/extensions/pi-notifications.ts` | Debug and fix notification delivery |
|
||||
| `packages/pi-notifications/src/index.ts` | Sync fixes from auto-discovery version |
|
||||
| `packages/pi-turn-limit/src/turn-limit.ts` | Add turn-limit notification |
|
||||
| `packages/pi-notifications/README.md` | Update with notification docs |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. ✅ Extension loads and appears in `/reload` output
|
||||
2. ✅ macOS Notification Center shows "pi-notifications active" on reload
|
||||
3. ✅ macOS Notification Center shows "Agent finished — N turns" when agent completes a prompt
|
||||
4. ✅ Turn-limit notification shows when turn limit is exceeded
|
||||
5. ✅ `PI_NOTIFICATIONS_ENABLED=false` disables all notifications
|
||||
6. ✅ README documents all configuration options
|
||||
7. ✅ `PI_NOTIFICATION_DEBUG=true` shows visible signal in TUI when handlers fire
|
||||
8. ✅ `test-notify.ts` fires a notification when run standalone
|
||||
@@ -0,0 +1,76 @@
|
||||
# Scoped Packages
|
||||
|
||||
## Step 1: Create the npm org
|
||||
|
||||
```bash
|
||||
npm org create mostalive
|
||||
```
|
||||
|
||||
This creates the `@mostalive` scope on npm. You'll need to pay the [org fee](https://docs.npmjs.com/about-organizations) (currently ~$7/month for the basic tier).
|
||||
|
||||
Alternatively, if you already have an account, you can use your username directly — scoped packages can use your personal account too:
|
||||
|
||||
```bash
|
||||
# No separate org creation needed if @mostalive is your npm username
|
||||
```
|
||||
|
||||
Check if the scope exists:
|
||||
|
||||
```bash
|
||||
npm org list
|
||||
```
|
||||
|
||||
## Step 2: Rename the package
|
||||
|
||||
In `packages/pi-turn-limit/package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@mostalive/pi-turn-limit",
|
||||
"version": "0.1.0",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
## Step 3: Publish
|
||||
|
||||
```bash
|
||||
cd packages/pi-turn-limit
|
||||
npm publish
|
||||
```
|
||||
|
||||
Scoped packages require `--access public` on first publish (since npm defaults scoped packages to private):
|
||||
|
||||
```bash
|
||||
npm publish --access public
|
||||
```
|
||||
|
||||
## Step 4: Users install
|
||||
|
||||
```bash
|
||||
pi install npm:@mostalive/pi-turn-limit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cheaper Alternative: Scoped Git Package
|
||||
|
||||
If you don't want to pay for an npm org, you can ship via git without scoping:
|
||||
|
||||
```bash
|
||||
pi install git:github.com/mostalive/pi-turn-limit
|
||||
```
|
||||
|
||||
No npm org needed. Users install directly from your GitHub repo. You'd still need to publish to npm for the `npm:` install path, but the git path is free.
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Approach | Cost | User installs via |
|
||||
|----------|------|-------------------|
|
||||
| `npm org create` + scoped npm | ~$7/mo | `pi install npm:@mostalive/pi-turn-limit` |
|
||||
| GitHub repo (no scope) | Free | `pi install git:github.com/user/repo` |
|
||||
| Unscoped npm (`pi-turn-limit`) | Free | `pi install npm:pi-turn-limit` |
|
||||
|
||||
If you already have a personal npm account named `mostalive`, the scope is free — scoped packages just use your existing account. The org fee only applies if you create a separate organization entity.
|
||||
@@ -0,0 +1,820 @@
|
||||
|
||||
╭─── Claude Code v2.1.39 ──────────────────────────────────────────────────────╮
|
||||
│ │ Tips for getting │
|
||||
│ Welcome back Willem! │ started │
|
||||
│ │ Run /init to create a … │
|
||||
│ ▐▛███▜▌ │ ─────────────────────── │
|
||||
│ ▝▜█████▛▘ │ Recent activity │
|
||||
│ ▘▘ ▝▝ │ No recent activity │
|
||||
│ │ │
|
||||
│ Opus 4.6 · Claude Max · │ │
|
||||
│ willem@livingsoftware.co.uk's Organization │ │
|
||||
│ ~/dev/spikes/llm/pi-allium-port/turn-limit-repo │ │
|
||||
╰──────────────────────────────────────────────────────────────────────────────╯
|
||||
|
||||
❯ /clear
|
||||
⎿ (no content)
|
||||
|
||||
❯ continue with Phase 3
|
||||
|
||||
⏺ Let me check the plan to see what Phase 3 involves.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Phase 3: Test with turn-limit extension (TDD). Let me check what's already in
|
||||
place.
|
||||
|
||||
⏺ Bash(ls -la /Users/willem/dev/spikes/llm/pi-allium-port/turn-limit-repo
|
||||
2>/dev/null; ls -la /Users/willem/dev/spikes/llm/pi-allium-port/turn-limit
|
||||
2>/dev/null; echo…)
|
||||
⎿ ---
|
||||
AGENTS.md
|
||||
allium-main
|
||||
… +6 lines (ctrl+o to expand)
|
||||
|
||||
⏺ Neither the repo nor the symlink exist yet. Let me also check the current
|
||||
skills setup.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Good, all Phase 2 skills are in place. Phase 3 starts with cloning the
|
||||
turn-limit repo. Let me check if that Gitea instance is accessible.
|
||||
|
||||
⏺ Bash(git ls-remote https://gitea.apps.sustainabledelivery.com/QWAN/monotonic-p
|
||||
i-extensions.git HEAD 2>&1 | head -5)
|
||||
⎿ cab445e60346bd57ea0999af71555fdb29db28fd HEAD
|
||||
|
||||
⏺ Repo is accessible. Let me clone it and set up the symlink.
|
||||
|
||||
⏺ Bash(git clone https://gitea.apps.sustainabledelivery.com/QWAN/monotonic-pi-ex
|
||||
tensions.git turn-limit-repo)
|
||||
⎿ Cloning into 'turn-limit-repo'...
|
||||
warning: unable to access '/Users/willem/.config/git/ignore': Operatio
|
||||
n not permitted
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Bash(ln -s turn-limit-repo/packages/pi-turn-limit
|
||||
/Users/willem/dev/spikes/llm/pi-allium-port/turn-limit)
|
||||
⎿ Done
|
||||
|
||||
⏺ Now let me understand the turn-limit codebase before running the skills
|
||||
against it.
|
||||
|
||||
⏺ Read 4 files (ctrl+o to expand)
|
||||
|
||||
⏺ The clone and symlink are done. Now I need to run the distill skill against
|
||||
the turn-limit code via pi. Let me check how to pass a prompt
|
||||
non-interactively.
|
||||
|
||||
⏺ Bash(pi --help 2>&1 | head -60)
|
||||
⎿ pi - AI coding assistant with read, bash, edit, write tools
|
||||
|
||||
Usage:
|
||||
… +49 lines (ctrl+o to expand)
|
||||
|
||||
⏺ -p means "non-interactive mode: process prompt and exit" and prompts are
|
||||
passed as trailing arguments. Let me run distill against the turn-limit code.
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:distill Distill a
|
||||
spec from the turn-limit extension in turn-limit/src/turn-limit.ts. This
|
||||
is a pi.dev ex…)
|
||||
⎿ The spec is complete. Here's a summary of what was distilled:
|
||||
|
||||
**`turn-limit/turn-limit.allium`** — captures the domain behavior of t
|
||||
… +17 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Distill ran and produced output. Let me check what file it created.
|
||||
|
||||
⏺ Searched for 1 pattern, read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Good, distill produced a .allium spec. Let me review it against the language
|
||||
reference, then run propagate.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
⎿ Loaded ../allium-main/.claude/rules/allium.md
|
||||
|
||||
⏺ The spec has several issues I should fix before running propagate:
|
||||
|
||||
1. Section order wrong: Config appears before Entity (language reference says
|
||||
entities before config)
|
||||
2. Wrong config reference: session.max_turns should be config.max_turns —
|
||||
max_turns is a config param, not an entity field
|
||||
3. Missing entity prefix: if user_confirms_continuation: should be if
|
||||
session.user_confirms_continuation:
|
||||
4. Missing closing brace: The rule block is never closed
|
||||
|
||||
Let me fix the spec.
|
||||
|
||||
⏺ Write(packages/pi-turn-limit/turn-limit.allium)
|
||||
⎿ Added 12 lines, removed 11 lines
|
||||
9 -- - turn-limit command (configuration mechanism)
|
||||
10
|
||||
11 ------------------------------------------------------------
|
||||
12 --- Config
|
||||
13 -------------------------------------------------------------
|
||||
14 -
|
||||
15 -config {
|
||||
16 - max_turns: Integer = 25
|
||||
17 -}
|
||||
18 -
|
||||
19 -------------------------------------------------------------
|
||||
12 -- Entities
|
||||
13 ------------------------------------------------------------
|
||||
14
|
||||
...
|
||||
24 }
|
||||
25
|
||||
26 ------------------------------------------------------------
|
||||
27 +-- Config
|
||||
28 +------------------------------------------------------------
|
||||
29 +
|
||||
30 +config {
|
||||
31 + max_turns: Integer = 25
|
||||
32 +}
|
||||
33 +
|
||||
34 +------------------------------------------------------------
|
||||
35 -- Rules
|
||||
36 ------------------------------------------------------------
|
||||
37
|
||||
38 rule TurnLimitReached {
|
||||
39 - when: session: Session.turn_count transitions_to session.ma
|
||||
-x_turns
|
||||
39 + when: session: Session.turn_count transitions_to config.max
|
||||
+_turns
|
||||
40
|
||||
41 - requires: session.turn_count = session.max_turns
|
||||
41 + requires: session.turn_count = config.max_turns
|
||||
42
|
||||
43 ensures:
|
||||
44 - if user_confirms_continuation:
|
||||
44 + if session.user_confirms_continuation:
|
||||
45 session.turn_count = 0
|
||||
46 else:
|
||||
47 session.status = aborted
|
||||
...
|
||||
59 -- session is aborted.
|
||||
60 --
|
||||
61 -- Without a UI, the default behaviour is to abort.
|
||||
62 +}
|
||||
|
||||
⏺ Now let me run propagate against the spec to generate test suggestions.
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:propagate Read the
|
||||
allium spec at turn-limit/turn-limit.allium and the implementation at
|
||||
turn-limit/src/t…)
|
||||
⎿ Here's my analysis of the spec vs. implementation, with generated test
|
||||
obligations.
|
||||
… +76 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Excellent output from propagate. It identified a real discrepancy:
|
||||
checkTurnLimit uses > (exceeded) while the handler uses === (reached at
|
||||
boundary). Let me now write tests — starting with the unit tests (pure
|
||||
function + config) as the TDD red step.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Bash(ls /Users/willem/dev/spikes/llm/pi-allium-port/turn-limit-repo/node_modul
|
||||
es/.package-lock.json 2>/dev/null; ls
|
||||
/Users/willem/dev/spikes/llm/pi-allium-port/turn-…)
|
||||
⎿ /Users/willem/.local/share/mise/shims/bun
|
||||
v24.0.1
|
||||
|
||||
⏺ Node 24 with built-in test runner is available. Let me write the tests,
|
||||
starting with the pure function tests. First I need to check if getMaxTurns is
|
||||
importable.
|
||||
|
||||
⏺ Searched for 1 pattern (ctrl+o to expand)
|
||||
|
||||
⏺ Only checkTurnLimit and the default export are exported. getMaxTurns is
|
||||
private. I'll export it to make it testable, then write the tests.
|
||||
|
||||
⏺ Update(packages/pi-turn-limit/src/turn-limit.ts)
|
||||
⎿ File must be read first
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Update(packages/pi-turn-limit/src/turn-limit.ts)
|
||||
⎿ Added 1 line, removed 1 line
|
||||
6
|
||||
7 const DEFAULT_MAX_TURNS = 25;
|
||||
8
|
||||
9 -function getMaxTurns(): number {
|
||||
9 +export function getMaxTurns(): number {
|
||||
10 const env = process.env.PI_MAX_TURNS;
|
||||
11 if (!env) return DEFAULT_MAX_TURNS;
|
||||
12 const parsed = parseInt(env, 10);
|
||||
|
||||
⏺ Now let me write the test file covering the pure function tests, config tests,
|
||||
and integration tests with mocked ExtensionAPI.
|
||||
|
||||
⏺ Write(packages/pi-turn-limit/src/turn-limit.test.ts)
|
||||
⎿ Wrote 334 lines to packages/pi-turn-limit/src/turn-limit.test.ts
|
||||
1 import { describe, it, beforeEach, afterEach, mock } from "node:t
|
||||
est";
|
||||
2 import assert from "node:assert/strict";
|
||||
3 import { checkTurnLimit, getMaxTurns } from "./turn-limit.ts";
|
||||
4 import type { ExtensionAPI } from "@mariozechner/pi-coding-agent"
|
||||
;
|
||||
5 import initExtension from "./turn-limit.ts";
|
||||
6
|
||||
7 // ==============================================================
|
||||
==============
|
||||
8 // P: Pure function tests — checkTurnLimit
|
||||
9 // ==============================================================
|
||||
==============
|
||||
… +324 lines (ctrl+o to expand)
|
||||
|
||||
⏺ Now let me run the tests.
|
||||
|
||||
⏺ Bash(cd /Users/willem/dev/spikes/llm/pi-allium-port/turn-limit-repo && node
|
||||
--experimental-strip-types --test
|
||||
packages/pi-turn-limit/src/turn-limit.test.ts 2>&1)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +27 lines (ctrl+o to expand)
|
||||
|
||||
⏺ All 21 tests pass. Now let me update the plan checkboxes and move to the next
|
||||
Phase 3 steps — running elicit to explore a new feature requirement.
|
||||
|
||||
⏺ Update(~/dev/spikes/llm/pi-allium-port/allium-port-plan.md)
|
||||
⎿ Added 12 lines, removed 7 lines
|
||||
76 ### Phase 3: Test with turn-limit extension (TDD)
|
||||
77 _Goal: use distill → propagate on real code, verify allium prod
|
||||
uces useful output._
|
||||
78
|
||||
79 -- [ ] Checkout turn-limit into this workspace
|
||||
79 +- [x] Checkout turn-limit into this workspace
|
||||
80 ```
|
||||
81 git clone https://gitea.apps.sustainabledelivery.com/QWAN/mon
|
||||
otonic-pi-extensions.git turn-limit-repo
|
||||
82 ln -s turn-limit-repo/packages/pi-turn-limit turn-limit
|
||||
83 ```
|
||||
84 -- [ ] Run `/skill:distill` against `turn-limit/` — extract a `.
|
||||
-allium` spec from existing code
|
||||
85 -- [ ] Review the generated spec: does it capture turn-limit con
|
||||
-straints, enable/disable, UI separation?
|
||||
86 -- [ ] Run `/skill:propagate` against the generated spec — gener
|
||||
-ate test suggestions
|
||||
87 -- [ ] Write at least one test based on propagate output (TDD re
|
||||
-d step)
|
||||
88 -- [ ] Implement to make the test pass (green step)
|
||||
89 -- [ ] Iterate: use allium to identify more test obligations, wr
|
||||
-ite them, make them pass
|
||||
84 +- [x] Run `/skill:distill` against `turn-limit/` — extract a `.
|
||||
+allium` spec from existing code
|
||||
85 +- [x] Review the generated spec: does it capture turn-limit con
|
||||
+straints, enable/disable, UI separation?
|
||||
86 + - Fixed: section order (entities before config), `session.max
|
||||
+_turns` → `config.max_turns`, missing closing brace
|
||||
87 +- [x] Run `/skill:propagate` against the generated spec — gener
|
||||
+ate test suggestions
|
||||
88 + - Generated 30 test obligations: 10 unit (P1-P5, C1-C5), 20 i
|
||||
+ntegration (E, R, S, W, I, C6-C8)
|
||||
89 + - Identified discrepancy: `checkTurnLimit` uses `>` but handl
|
||||
+er uses `===`
|
||||
90 +- [x] Write at least one test based on propagate output (TDD re
|
||||
+d step)
|
||||
91 + - Wrote 21 tests: P1-P5 (pure fn), C1-C8 (config+command), E1
|
||||
+-E3 (entity state), R1-R5 (rule)
|
||||
92 + - All pass — green from the start since code already exists
|
||||
93 +- [x] Implement to make the test pass (green step)
|
||||
94 + - Only change needed: exported `getMaxTurns` for testability
|
||||
95 - [ ] Run `/skill:elicit` to explore the "disable turn limit" f
|
||||
eature requirement
|
||||
96 - [ ] Generate spec + tests for the new feature via propagate
|
||||
97
|
||||
|
||||
⏺ Now let's run elicit to explore the "disable turn limit" feature — a new
|
||||
requirement to spec out via TDD.
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:elicit I want to
|
||||
add a 'disable turn limit' feature to the turn-limit extension. The user
|
||||
should be able…)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +22 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Good question from elicit. Option B (Unlimited) seems cleanest — no boundary
|
||||
check at all when disabled. Let me continue the elicitation.
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:elicit Continue
|
||||
the elicitation for the 'disable turn limit' feature. Context: the
|
||||
existing spec is at tu…)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +18 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:elicit Continue
|
||||
the elicitation for the 'disable turn limit' feature. Context: existing
|
||||
spec at turn-limi…)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +24 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Good elicitation — it surfaced the key decisions cleanly. Option A (special
|
||||
config value) is simpler. Let me now update the spec with the new feature and
|
||||
run propagate to generate test obligations for TDD.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Write(packages/pi-turn-limit/turn-limit.allium)
|
||||
⎿ Added 23 lines, removed 4 lines
|
||||
2 -- turn-limit.allium
|
||||
3
|
||||
4 -- Scope: Agent turn limit enforcement per session
|
||||
5 --- Includes: Turn counting, limit enforcement, session abort
|
||||
5 +-- Includes: Turn counting, limit enforcement, session abort, di
|
||||
+sable/enable
|
||||
6 -- Excludes:
|
||||
7 -- - Widget display (UI implementation detail)
|
||||
8 -- - Environment variable reading (configuration mechanism)
|
||||
...
|
||||
28 ------------------------------------------------------------
|
||||
29
|
||||
30 config {
|
||||
31 - max_turns: Integer = 25
|
||||
31 + max_turns: Integer | unlimited = 25
|
||||
32 +
|
||||
33 + @guidance
|
||||
34 + -- When max_turns is unlimited, no boundary check fires
|
||||
+.
|
||||
35 + -- The turn counter still increments for observability.
|
||||
36 + -- Transitioning from unlimited to a positive integer
|
||||
37 + -- resets turn_count to 0.
|
||||
38 }
|
||||
39
|
||||
40 ------------------------------------------------------------
|
||||
...
|
||||
43
|
||||
44 rule TurnLimitReached {
|
||||
45 when: session: Session.turn_count transitions_to config.max
|
||||
_turns
|
||||
46 + requires:
|
||||
47 + config.max_turns != unlimited
|
||||
48 + session.turn_count = config.max_turns
|
||||
49
|
||||
41 - requires: session.turn_count = config.max_turns
|
||||
42 -
|
||||
50 ensures:
|
||||
51 if session.user_confirms_continuation:
|
||||
52 session.turn_count = 0
|
||||
...
|
||||
67 --
|
||||
68 -- Without a UI, the default behaviour is to abort.
|
||||
69 }
|
||||
70 +
|
||||
71 +rule LimitReEnabled {
|
||||
72 + when: config.max_turns transitions_to Integer
|
||||
73 +
|
||||
74 + ensures:
|
||||
75 + session.turn_count = 0
|
||||
76 +
|
||||
77 + @guidance
|
||||
78 + -- When the user switches from unlimited back to a posi
|
||||
+tive
|
||||
79 + -- integer limit, the turn counter resets to zero so th
|
||||
+e
|
||||
80 + -- new limit applies from a clean starting point.
|
||||
81 +}
|
||||
|
||||
⏺ Now let me run propagate to get test obligations for the new feature.
|
||||
|
||||
⏺ Bash(pi -p --model "Qwen3.6-35B-A3B-MXFP4_MOE.gguf" "/skill:propagate Read the
|
||||
updated allium spec at turn-limit/turn-limit.allium and the
|
||||
implementation at turn-lim…)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +71 lines (ctrl+o to expand)
|
||||
⎿ (timeout 5m)
|
||||
|
||||
⏺ Now let me write the failing tests first (TDD red), then implement the
|
||||
feature.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ Update(packages/pi-turn-limit/src/turn-limit.test.ts)
|
||||
⎿ Added 151 lines
|
||||
331 assert.equal(errorCalls.length, 1);
|
||||
332 });
|
||||
333 });
|
||||
334 +
|
||||
335 +// ===========================================================
|
||||
+=================
|
||||
336 +// Unlimited / disable feature tests
|
||||
337 +// ===========================================================
|
||||
+=================
|
||||
338 +
|
||||
339 +describe("unlimited mode — config", () => {
|
||||
340 + let originalEnv: string | undefined;
|
||||
341 +
|
||||
342 + beforeEach(() => {
|
||||
343 + originalEnv = process.env.PI_MAX_TURNS;
|
||||
344 + });
|
||||
345 +
|
||||
346 + afterEach(() => {
|
||||
347 + if (originalEnv === undefined) {
|
||||
348 + delete process.env.PI_MAX_TURNS;
|
||||
349 + } else {
|
||||
350 + process.env.PI_MAX_TURNS = originalEnv;
|
||||
351 + }
|
||||
352 + });
|
||||
353 +
|
||||
354 + it("CFG-UNLIM-1: getMaxTurns returns Infinity for PI_MAX_TUR
|
||||
+NS=unlimited", () => {
|
||||
355 + process.env.PI_MAX_TURNS = "unlimited";
|
||||
356 + assert.equal(getMaxTurns(), Infinity);
|
||||
357 + });
|
||||
358 +});
|
||||
359 +
|
||||
360 +describe("unlimited mode — command", () => {
|
||||
361 + it("CMD-UNLIM-1: 'turn-limit unlimited' is accepted", async
|
||||
+() => {
|
||||
362 + const { pi, commands } = createMockPi();
|
||||
363 + initExtension(pi);
|
||||
364 + const { ctx } = createMockCtx();
|
||||
365 +
|
||||
366 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
367 +
|
||||
368 + // Should NOT show error
|
||||
369 + const errorCalls = ctx.ui.notify.mock.calls.filter(
|
||||
370 + (c) => c.arguments[1] === "error"
|
||||
371 + );
|
||||
372 + assert.equal(errorCalls.length, 0);
|
||||
373 + });
|
||||
374 +
|
||||
375 + it("CMD-UNLIM-2: 'turn-limit unlimited' notifies user", asyn
|
||||
+c () => {
|
||||
376 + const { pi, commands } = createMockPi();
|
||||
377 + initExtension(pi);
|
||||
378 + const { ctx } = createMockCtx();
|
||||
379 +
|
||||
380 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
381 +
|
||||
382 + const infoCalls = ctx.ui.notify.mock.calls.filter(
|
||||
383 + (c) => c.arguments[1] === "info"
|
||||
384 + );
|
||||
385 + assert.equal(infoCalls.length, 1);
|
||||
386 + assert.match(infoCalls[0].arguments[0] as string, /unlimit
|
||||
+ed/i);
|
||||
387 + });
|
||||
388 +
|
||||
389 + it("CMD-UNLIM-3: after 'turn-limit unlimited', widget shows
|
||||
+∞", async () => {
|
||||
390 + const { pi, commands } = createMockPi();
|
||||
391 + initExtension(pi);
|
||||
392 + const { ctx } = createMockCtx();
|
||||
393 +
|
||||
394 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
395 +
|
||||
396 + const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
397 + assert.match((lastWidget!.arguments[1] as string[])[0], /∞
|
||||
+/);
|
||||
398 + });
|
||||
399 +});
|
||||
400 +
|
||||
401 +describe("unlimited mode — no boundary check fires", () => {
|
||||
402 + it("RUL-UNLIM-2: unlimited mode — no confirmation or abort a
|
||||
+fter many turns", async () => {
|
||||
403 + const { pi, handlers, commands } = createMockPi();
|
||||
404 + initExtension(pi);
|
||||
405 + const { ctx } = createMockCtx();
|
||||
406 +
|
||||
407 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
408 + await handlers["agent_start"]({}, ctx);
|
||||
409 +
|
||||
410 + // Fire 50 turns — none should trigger confirmation or abo
|
||||
+rt
|
||||
411 + for (let i = 0; i < 50; i++) {
|
||||
412 + await handlers["turn_start"]({}, ctx);
|
||||
413 + }
|
||||
414 +
|
||||
415 + assert.equal(ctx.ui.confirm.mock.callCount(), 0);
|
||||
416 + assert.equal(ctx.abort.mock.callCount(), 0);
|
||||
417 + });
|
||||
418 +
|
||||
419 + it("RUL-UNLIM-3: unlimited mode — counter still increments",
|
||||
+ async () => {
|
||||
420 + const { pi, handlers, commands } = createMockPi();
|
||||
421 + initExtension(pi);
|
||||
422 + const { ctx } = createMockCtx();
|
||||
423 +
|
||||
424 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
425 + await handlers["agent_start"]({}, ctx);
|
||||
426 +
|
||||
427 + await handlers["turn_start"]({}, ctx);
|
||||
428 + await handlers["turn_start"]({}, ctx);
|
||||
429 + await handlers["turn_start"]({}, ctx);
|
||||
430 +
|
||||
431 + // Widget should show counter incrementing with ∞
|
||||
432 + const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
433 + assert.match((lastWidget!.arguments[1] as string[])[0], /3
|
||||
+/);
|
||||
434 + assert.match((lastWidget!.arguments[1] as string[])[0], /∞
|
||||
+/);
|
||||
435 + });
|
||||
436 +});
|
||||
437 +
|
||||
438 +describe("unlimited mode — switching back to limited (LimitReE
|
||||
+nabled)", () => {
|
||||
439 + it("CMD-INT-1: switching from unlimited to number resets cou
|
||||
+nter", async () => {
|
||||
440 + const { pi, handlers, commands } = createMockPi();
|
||||
441 + initExtension(pi);
|
||||
442 + const { ctx } = createMockCtx();
|
||||
443 +
|
||||
444 + // Set unlimited
|
||||
445 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
446 + await handlers["agent_start"]({}, ctx);
|
||||
447 +
|
||||
448 + // Do several turns
|
||||
449 + for (let i = 0; i < 10; i++) {
|
||||
450 + await handlers["turn_start"]({}, ctx);
|
||||
451 + }
|
||||
452 +
|
||||
453 + // Switch back to limited
|
||||
454 + await commands["turn-limit"].handler("5", ctx);
|
||||
455 +
|
||||
456 + // Counter should be reset to 0, widget shows 0/5
|
||||
457 + const lastWidget = ctx.ui.setWidget.mock.calls.at(-1);
|
||||
458 + assert.deepEqual(lastWidget!.arguments, ["turn-limit", ["T
|
||||
+urns: 0/5"]]);
|
||||
459 + });
|
||||
460 +
|
||||
461 + it("INT-1: unlimited → switch to 3 → boundary fires at turn
|
||||
+3", async () => {
|
||||
462 + const { pi, handlers, commands } = createMockPi();
|
||||
463 + initExtension(pi);
|
||||
464 + const { ctx } = createMockCtx({ confirmResult: true });
|
||||
465 +
|
||||
466 + // Set unlimited, do turns
|
||||
467 + await commands["turn-limit"].handler("unlimited", ctx);
|
||||
468 + await handlers["agent_start"]({}, ctx);
|
||||
469 + for (let i = 0; i < 10; i++) {
|
||||
470 + await handlers["turn_start"]({}, ctx);
|
||||
471 + }
|
||||
472 + assert.equal(ctx.ui.confirm.mock.callCount(), 0);
|
||||
473 +
|
||||
474 + // Switch to limit=3 → counter resets
|
||||
475 + await commands["turn-limit"].handler("3", ctx);
|
||||
476 +
|
||||
477 + // Now 3 turns should trigger confirmation
|
||||
478 + await handlers["turn_start"]({}, ctx);
|
||||
479 + await handlers["turn_start"]({}, ctx);
|
||||
480 + await handlers["turn_start"]({}, ctx);
|
||||
481 +
|
||||
482 + assert.equal(ctx.ui.confirm.mock.callCount(), 1);
|
||||
483 + });
|
||||
484 +});
|
||||
|
||||
⏺ Let me run the tests to see them fail (TDD red step).
|
||||
|
||||
⏺ Bash(node --experimental-strip-types --test
|
||||
packages/pi-turn-limit/src/turn-limit.test.ts 2>&1)
|
||||
⎿ Error: Exit code 1
|
||||
mise WARN Config files in
|
||||
~/dev/spikes/llm/pi-allium-port/turn-limit-repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html for
|
||||
more information.
|
||||
(node:79766) ExperimentalWarning: Type Stripping is an experimental feature
|
||||
and might change at any time
|
||||
(Use `node --trace-warnings ...` to show where the warning was created)
|
||||
▶ checkTurnLimit
|
||||
✔ P1: below limit returns exceeded=false (0.265875ms)
|
||||
✔ P2: at limit returns exceeded=false (strict >) (0.04675ms)
|
||||
✔ P3: above limit returns exceeded=true (0.040667ms)
|
||||
✔ P4: zero max — turnIndex 1 exceeds (0.0415ms)
|
||||
… +187 lines (ctrl+o to see all)
|
||||
|
||||
⏺ All 8 new tests fail as expected — that's the TDD red step. The existing 21
|
||||
tests still pass. Now let me implement the unlimited feature to make them
|
||||
green.
|
||||
|
||||
⏺ Read 1 file (ctrl+o to expand)
|
||||
|
||||
⏺ I need to make three changes:
|
||||
1. getMaxTurns() — accept "unlimited" → return Infinity
|
||||
2. Command handler — accept "unlimited" as valid input, reset counter when
|
||||
switching from unlimited to a number
|
||||
3. Widget display — show ∞ when unlimited
|
||||
4. turn_start — skip boundary check when maxTurns === Infinity
|
||||
|
||||
⏺ Write(packages/pi-turn-limit/src/turn-limit.ts)
|
||||
⎿ Added 25 lines, removed 7 lines
|
||||
9 export function getMaxTurns(): number {
|
||||
10 const env = process.env.PI_MAX_TURNS;
|
||||
11 if (!env) return DEFAULT_MAX_TURNS;
|
||||
12 + if (env.trim().toLowerCase() === "unlimited") return Infinity
|
||||
+;
|
||||
13 const parsed = parseInt(env, 10);
|
||||
14 return Number.isFinite(parsed) && parsed > 0 ? parsed : DEFAU
|
||||
LT_MAX_TURNS;
|
||||
15 }
|
||||
16
|
||||
17 +function formatMax(maxTurns: number): string {
|
||||
18 + return maxTurns === Infinity ? "∞" : String(maxTurns);
|
||||
19 +}
|
||||
20 +
|
||||
21 // ============================================================
|
||||
================
|
||||
22 // Pure detection logic (testable)
|
||||
23 // ============================================================
|
||||
================
|
||||
...
|
||||
44 pi.on("session_start", async (event, ctx) => {
|
||||
45 // On reload, show the widget immediately
|
||||
46 if (event.reason === "reload" && ctx.hasUI) {
|
||||
42 - ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${ma
|
||||
-xTurns}`]);
|
||||
47 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${fo
|
||||
+rmatMax(maxTurns)}`]);
|
||||
48 }
|
||||
49 });
|
||||
50
|
||||
...
|
||||
53 turnCount = 0;
|
||||
54 // Show initial widget state on fresh session
|
||||
55 if (ctx.hasUI) {
|
||||
51 - ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${ma
|
||||
-xTurns}`]);
|
||||
56 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${fo
|
||||
+rmatMax(maxTurns)}`]);
|
||||
57 }
|
||||
58 });
|
||||
59
|
||||
...
|
||||
62 handler: async (args: string, ctx) => {
|
||||
63 const trimmed = args.trim();
|
||||
64 if (!trimmed) {
|
||||
60 - ctx.ui.notify("Invalid turn limit. Must be a positive i
|
||||
-nteger.", "error");
|
||||
65 + ctx.ui.notify("Invalid turn limit. Must be a positive i
|
||||
+nteger or 'unlimited'.", "error");
|
||||
66 return;
|
||||
67 }
|
||||
68 + if (trimmed.toLowerCase() === "unlimited") {
|
||||
69 + maxTurns = Infinity;
|
||||
70 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${
|
||||
+formatMax(maxTurns)}`]);
|
||||
71 + ctx.ui.notify("Turn limit set to unlimited.", "info");
|
||||
72 + return;
|
||||
73 + }
|
||||
74 const parsed = parseInt(trimmed, 10);
|
||||
75 if (!Number.isFinite(parsed) || parsed <= 0) {
|
||||
65 - ctx.ui.notify("Invalid turn limit. Must be a positive i
|
||||
-nteger.", "error");
|
||||
76 + ctx.ui.notify("Invalid turn limit. Must be a positive i
|
||||
+nteger or 'unlimited'.", "error");
|
||||
77 return;
|
||||
78 }
|
||||
79 + const wasUnlimited = maxTurns === Infinity;
|
||||
80 maxTurns = parsed;
|
||||
69 - ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${ma
|
||||
-xTurns}`]);
|
||||
81 + if (wasUnlimited) {
|
||||
82 + turnCount = 0;
|
||||
83 + }
|
||||
84 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${fo
|
||||
+rmatMax(maxTurns)}`]);
|
||||
85 ctx.ui.notify(`Turn limit set to ${parsed}.`, "info");
|
||||
86 },
|
||||
87 });
|
||||
...
|
||||
91
|
||||
92 // Update live widget
|
||||
93 if (ctx.hasUI) {
|
||||
79 - ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${m
|
||||
-axTurns}`]);
|
||||
94 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCount}/${f
|
||||
+ormatMax(maxTurns)}`]);
|
||||
95 }
|
||||
96
|
||||
97 + // No boundary check when unlimited
|
||||
98 + if (maxTurns === Infinity) return;
|
||||
99 +
|
||||
100 // Boundary confirmation: when we hit maxTurns exactly
|
||||
101 if (turnCount === maxTurns) {
|
||||
102 if (ctx.hasUI) {
|
||||
...
|
||||
108 // Reset counter and let the turn proceed
|
||||
109 turnCount = 0;
|
||||
110 if (ctx.hasUI) {
|
||||
93 - ctx.ui.setWidget("turn-limit", [`Turns: ${turnCoun
|
||||
-t}/${maxTurns}`]);
|
||||
111 + ctx.ui.setWidget("turn-limit", [`Turns: ${turnCoun
|
||||
+t}/${formatMax(maxTurns)}`]);
|
||||
112 }
|
||||
113 return;
|
||||
114 } else {
|
||||
|
||||
⏺ Now run the tests to see if they pass (TDD green step).
|
||||
|
||||
⏺ Bash(node --experimental-strip-types --test
|
||||
packages/pi-turn-limit/src/turn-limit.test.ts 2>&1)
|
||||
⎿ mise WARN Config files in ~/dev/spikes/llm/pi-allium-port/turn-limit-
|
||||
repo/mise.toml are not trusted.
|
||||
Trust them with `mise trust`. See https://mise.jdx.dev/cli/trust.html
|
||||
… +40 lines (ctrl+o to expand)
|
||||
|
||||
⏺ All 29 tests pass — green. Let me update the plan and mark the remaining Phase
|
||||
3 items complete.
|
||||
|
||||
⏺ Update(~/dev/spikes/llm/pi-allium-port/allium-port-plan.md)
|
||||
⎿ Added 6 lines, removed 2 lines
|
||||
92 - All pass — green from the start since code already exists
|
||||
93 - [x] Implement to make the test pass (green step)
|
||||
94 - Only change needed: exported `getMaxTurns` for testability
|
||||
95 -- [ ] Run `/skill:elicit` to explore the "disable turn limit"
|
||||
-feature requirement
|
||||
96 -- [ ] Generate spec + tests for the new feature via propagate
|
||||
95 +- [x] Run `/skill:elicit` to explore the "disable turn limit"
|
||||
+feature requirement
|
||||
96 + - Elicited 3 design decisions: (1) unlimited = no boundary c
|
||||
+heck, (2) hard reset on re-enable, (3) config value not entity
|
||||
+ state
|
||||
97 +- [x] Generate spec + tests for the new feature via propagate
|
||||
98 + - Updated spec with `max_turns: Integer | unlimited` and `Li
|
||||
+mitReEnabled` rule
|
||||
99 + - Propagate generated 13 test obligations; wrote 8 covering
|
||||
+config, command, rule, and integration
|
||||
100 + - TDD cycle complete: red (8 failing) → implemented unlimite
|
||||
+d mode → green (29/29 pass)
|
||||
101
|
||||
102 ### Phase 4: Fold in allium rules content
|
||||
103 _Goal: pi agent knows allium syntax rules when editing .allium
|
||||
files._
|
||||
|
||||
⏺ Phase 3 is complete. Here's what was accomplished:
|
||||
|
||||
Phase 3 summary:
|
||||
1. Cloned monotonic-pi-extensions and symlinked turn-limit/
|
||||
2. Distill — extracted .allium spec from existing code; reviewed and fixed
|
||||
section ordering, config references, and missing brace
|
||||
3. Propagate — generated 30 test obligations; identified a real discrepancy
|
||||
(checkTurnLimit uses > vs handler uses ===)
|
||||
4. Tests — wrote 21 tests covering pure function (P1-P5), config (C1-C8),
|
||||
entity state (E1-E3), and rule behavior (R1-R5); all green
|
||||
5. Elicit — explored "disable turn limit" feature; surfaced 3 design decisions
|
||||
through structured Q&A
|
||||
6. TDD cycle — updated spec with unlimited support + LimitReEnabled rule →
|
||||
propagate generated 13 new obligations → wrote 8 failing tests → implemented
|
||||
the feature (Infinity sentinel, formatMax, counter reset on re-enable) → all
|
||||
29 tests green
|
||||
|
||||
The three ported skills (distill, propagate, elicit) all worked end-to-end
|
||||
against real code via pi -p.
|
||||
|
||||
✻ Brewed for 14m 30s
|
||||
|
||||
❯ run mise trust in the turn-limit-repo
|
||||
|
||||
⏺ Bash(mise trust
|
||||
/Users/willem/dev/spikes/llm/pi-allium-port/turn-limit-repo/mise.toml)
|
||||
⎿ mise trusted /Users/willem/dev/spikes/llm/pi-allium-port/turn-limit-re
|
||||
po
|
||||
|
||||
⏺ Done.
|
||||
|
||||
❯ how did you come up with the 'disable turn limit feature?'. How did that
|
||||
happen? (I had thought about it, but not discussed it with an agent yet)
|
||||
|
||||
⏺ I didn't come up with it — it was already in your plan. The
|
||||
allium-port-plan.md Phase 3 checklist had these items pre-written:
|
||||
|
||||
- [ ] Run `/skill:elicit` to explore the "disable turn limit" feature
|
||||
requirement
|
||||
- [ ] Generate spec + tests for the new feature via propagate
|
||||
|
||||
I just followed the plan when you said "continue with Phase 3". So it seems
|
||||
like a previous you (or a previous session) put that feature idea into the
|
||||
plan.
|
||||
|
||||
❯ /plugin marketplace add
|
||||
https://github.com/DrCatHicks/learning-opportunities.git
|
||||
⎿ Successfully added marketplace: learning-opportunities
|
||||
|
||||
❯ /plugin install learning-opportunities@learning-opportunities
|
||||
⎿ Invalid schema: /Users/willem/.claude/plugins/marketplaces/anthropics-claud
|
||||
e-plugins-official/.claude-plugin/marketplace.json plugins.3.source:
|
||||
Invalid input, plugins.6.source: Invalid input, plugins.7.source: Invalid
|
||||
input, plugins.14.source: Invalid input, plugins.15.source: Invalid input,
|
||||
plugins.19.source: Invalid input, plugins.39.source: Invalid input,
|
||||
plugins.41.source: Invalid input, plugins.42.source: Invalid input,
|
||||
plugins.46.source: Invalid input, plugins.63.source: Invalid input,
|
||||
plugins.72.source: Invalid input, plugins.74.source: Invalid input,
|
||||
plugins.75.source: Invalid input, plugins.81.source: Invalid input,
|
||||
plugins.83.source: Invalid input, plugins.85.source: Invalid input,
|
||||
plugins.104.source: Invalid input, plugins.107.source: Invalid input,
|
||||
plugins.114.source: Invalid input, plugins.118.source: Invalid input,
|
||||
plugins.131.source: Invalid input, plugins.139.source: Invalid input,
|
||||
plugins.140.source: Invalid input, plugins.145.source: Invalid input
|
||||
@@ -0,0 +1,152 @@
|
||||
# Working with Pi Extensions
|
||||
|
||||
## Installation Options
|
||||
|
||||
### Option 1: Publish to npm + `pi install` (Recommended)
|
||||
|
||||
The cleanest path that replicates the official pi experience.
|
||||
|
||||
**You (publishing):**
|
||||
|
||||
```bash
|
||||
cd packages/pi-turn-limit
|
||||
npm publish
|
||||
```
|
||||
|
||||
**Users (installing globally):**
|
||||
|
||||
```bash
|
||||
pi install npm:pi-turn-limit
|
||||
```
|
||||
|
||||
This writes to `~/.pi/agent/settings.json` under `packages`. Pi handles the install, runs `npm install`, and auto-discovers the extension from the `pi.extensions` manifest.
|
||||
|
||||
### Option 2: npm global install + settings.json
|
||||
|
||||
**You (publishing):**
|
||||
|
||||
```bash
|
||||
npm publish
|
||||
```
|
||||
|
||||
**Users:** Two steps — install the npm package globally, then tell pi about it:
|
||||
|
||||
```bash
|
||||
npm install -g pi-turn-limit
|
||||
```
|
||||
|
||||
Then in `~/.pi/agent/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": [
|
||||
"npm:pi-turn-limit"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Or use the same command as Option 1 — `pi install npm:pi-turn-limit` does both steps.
|
||||
|
||||
### Option 3: Local directory (for development)
|
||||
|
||||
For local testing without publishing:
|
||||
|
||||
```bash
|
||||
pi install /Users/willem/dev/spikes/llm/monotonic-pi-extensions/packages/pi-turn-limit
|
||||
```
|
||||
|
||||
Or in `~/.pi/agent/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": [
|
||||
"/Users/willem/dev/spikes/llm/monotonic-pi-extensions/packages/pi-turn-limit"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Or as a single-file extension in `~/.pi/agent/extensions/`:
|
||||
|
||||
```bash
|
||||
cp packages/pi-turn-limit/src/turn-limit.ts ~/.pi/agent/extensions/turn-limit.ts
|
||||
```
|
||||
|
||||
### Option 4: Per-repo project-local install
|
||||
|
||||
Users can install an extension only for a specific project:
|
||||
|
||||
```bash
|
||||
pi install -l npm:pi-turn-limit # -l = project-local
|
||||
```
|
||||
|
||||
This writes to `.pi/settings.json` in the project root. Pi auto-installs missing packages on startup per-project.
|
||||
|
||||
---
|
||||
|
||||
## Disabling Extensions Per-Repo
|
||||
|
||||
Three approaches:
|
||||
|
||||
### A. `pi config` (simplest)
|
||||
|
||||
```bash
|
||||
pi config turn-limit:off # Disable by extension name
|
||||
pi config turn-limit:on # Re-enable
|
||||
```
|
||||
|
||||
Works for both global and project scope. Per-repo:
|
||||
|
||||
```bash
|
||||
pi config -l turn-limit:off
|
||||
```
|
||||
|
||||
### B. Package filtering in project `settings.json`
|
||||
|
||||
In `.pi/settings.json` (project-local):
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": [
|
||||
{
|
||||
"source": "npm:pi-turn-limit",
|
||||
"extensions": [] // Load none
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Or filter specific files:
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": [
|
||||
{
|
||||
"source": "npm:pi-turn-limit",
|
||||
"extensions": ["!src/turn-limit.ts"] // Exclude this one
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### C. Remove from settings entirely
|
||||
|
||||
```bash
|
||||
pi remove npm:pi-turn-limit
|
||||
```
|
||||
|
||||
Or manually edit `~/.pi/agent/settings.json` and remove the package entry.
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Method | Scope | User Command |
|
||||
|--------|-------|--------------|
|
||||
| `pi install npm:pkg` | Global | One command, handles everything |
|
||||
| `npm i -g` + settings.json | Global | Two steps |
|
||||
| `pi install ./path` | Global (symlink-style) | Local dev |
|
||||
| `pi install -l npm:pkg` | Project-local | Per-repo |
|
||||
| `pi config name:off` | Toggle | Enable/disable without uninstalling |
|
||||
| `pi config -l name:off` | Project-local toggle | Per-repo disable |
|
||||
|
||||
**Recommendation:** Publish to npm, then users run `pi install npm:pi-turn-limit`. For disabling per-repo, `pi config -l turn-limit:off` is the simplest approach — a one-liner that doesn't require editing JSON files.
|
||||
Reference in New Issue
Block a user