91 lines
4.2 KiB
Markdown
91 lines
4.2 KiB
Markdown
# Test Output Parsers
|
|
|
|
## Motivation
|
|
|
|
The test-status widget needs to decide pass/fail and show a meaningful summary.
|
|
Relying on output regex alone for the pass/fail decision is brittle — different
|
|
test frameworks, locales, custom reporters, or unusual test names can produce
|
|
false positives or negatives. Using the process exit code as the canonical
|
|
signal is universal and correct across all tools.
|
|
|
|
This document outlines a modular architecture where pass/fail is decided by exit
|
|
code, and output parsing is purely cosmetic enrichment.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────┐
|
|
│ Decision (exit code) │ ← single source of truth for pass/fail
|
|
│ - Always runs │
|
|
│ - Produces: pass/fail bool │
|
|
└──────────┬──────────────────┘
|
|
│ feeds status icon
|
|
▼
|
|
┌─────────────────────────────┐
|
|
│ Display │
|
|
│ - Icon from exit code │
|
|
│ - Summary from parser │
|
|
└──────────┬──────────────────┘
|
|
│ enriched by
|
|
▼
|
|
┌─────────────────────────────┐
|
|
│ Parser Registry │ ← extensible, tool-agnostic
|
|
│ - Tries parsers in order │
|
|
│ - First match wins │
|
|
│ - Each parser has: │
|
|
│ • detect(output) → bool │
|
|
│ • summarize(output) │
|
|
│ → str | null │
|
|
│ - Fallback: generic text │
|
|
└─────────────────────────────┘
|
|
```
|
|
|
|
## Parser Module Interface
|
|
|
|
Each parser is a self-contained module that exports two functions:
|
|
|
|
| Function | Signature | Role |
|
|
|---|---|---|
|
|
| `detect` | `(output: string) => boolean` | Returns `true` if this tool's output is present. Checks for distinctive markers (e.g., vitest's "RUN" banner, pytest's `===` separator, cargo test's "running" lines). |
|
|
| `summarize` | `(output: string) => string \| null` | Extracts a one-line human-readable summary (e.g., "3 passed (242ms)", "2 failed, 1 passed"). Returns `null` if no summary can be extracted. |
|
|
|
|
## Lifecycle
|
|
|
|
1. A tool execution completes (any bash command).
|
|
2. Check if the command is a known test command (current behaviour).
|
|
3. **Decision**: Read `event.details?.exitCode` — `0` means pass, anything else
|
|
means fail. This is the single source of truth.
|
|
4. **Display — icon**: Set widget icon (● green or ● red) based on exit code.
|
|
5. **Display — summary**: Walk the parser registry. For each parser, call
|
|
`detect(output)`. On first match, call `summarize(output)` for the text. If
|
|
no parser matches or `summarize` returns `null`, fall back to a generic
|
|
string ("All tests passed" / "Tests failed").
|
|
|
|
## Adding a New Tool
|
|
|
|
To add support for a new test framework, create a new file in the parsers
|
|
directory that implements the two-function interface. Register it in the
|
|
registry list. Zero changes to the core decision or display logic.
|
|
|
|
Examples of tool-specific markers for `detect`:
|
|
|
|
| Tool | Detect marker |
|
|
|---|---|
|
|
| Vitest | `" RUN "` banner |
|
|
| Jest | `"PASS "` / `"FAIL "` prefixes |
|
|
| pytest | `"=== test session starts ==="` |
|
|
| Go test | `"ok "` or `"FAIL "` line prefixes |
|
|
| cargo test | `"running "` followed by digit |
|
|
| Mocha | `"passing"` / `"failing"` in summary |
|
|
| RSpec | `"Finished in "` |
|
|
|
|
## Benefits Over Current Approach
|
|
|
|
| Aspect | Current (inline regex) | Proposed (modular parsers) |
|
|
|---|---|---|
|
|
| Pass/fail signal | Regex on output — brittle | Exit code — universal & correct |
|
|
| Output parsing | Inline, single format | Registry of modular parsers |
|
|
| Adding new tools | Edit the one function, risk regressions | Add a new file, register it |
|
|
| False negatives | Possible (word "failed" in test name) | Eliminated (exit code is canonical) |
|
|
| False positives | Possible (weird output formatting) | Eliminated |
|
|
| Parsing failure impact | Can break pass/fail (wrong icon) | Only loses cosmetic detail (correct icon preserved) | |