yak(mutation-testing-with-muex-blogex-pure-app-code-1341): mutation testing with muex (blogex + pure app code)

doc/mutation-testing.md

@@ -0,0 +1,163 @@
+# Mutation testing with muex
+
+Mutation testing is a **sensor, not a gate**. It deliberately breaks
+production code one mutant at a time and re-runs the suite: a mutant *killed*
+by a test is evidence the test pins that behaviour; a mutant that *survives*
+is a question for a Lens session. Runs always pass `--fail-at 0`, so a survivor
+is reported and triaged, never a failing build.
+
+Nothing here is wired into `VERIFY_SENSORS`. `make check`, `make verify`,
+coverage floors and `.githooks/pre-push` are unchanged by this yak.
+
+## How to run it
+
+Both apps declare two aliases in `mix.exs`; both read a plain-text scope file
+at run time, so the file stays the single source of truth.
+
+| Command | Scope | Blogex wall clock (2026-10-28) |
+|---------|-------|--------------------------------|
+| `mix mutate.fast` | `mutation_scope_fast.txt` (3 modules) | ~3 s |
+| `mix mutate` | `mutation_scope.txt` (8 modules) | ~4 s |
+
+The app declares the same aliases but both scope files are empty (see
+"Scope: app" below), so `mix mutate` there raises a clear Mix error instead of
+running a sensor over nothing.
+
+Extra arguments pass through to muex, so a one-off target is explicit:
+
+```bash
+cd blogex
+mix mutate --files lib/blogex/seo.ex          # one module, ad hoc
+mix mutate.fast --no-optimize                 # full mutant list, slower
+```
+
+The aliases pin `--fail-at 0` first, so an ad-hoc flag cannot turn the sensor
+into a gate. `mix mutate` does not enable `--cover`: muex runs each mutant's
+suite under `MIX_ENV=test`, and coverage would let a code-deleting mutant trip
+the project's coverage floor and abort the run.
+
+## Scope: blogex
+
+| In scope (`mutation_scope.txt`) | Why |
+|---------------------------------|-----|
+| `lib/blogex/link_validator.ex` | Pure link parsing/validation; the yak's example module. |
+| `lib/blogex/seo.ex` | Pure sitemap/JSON-LD/OpenGraph string building. |
+| `lib/blogex/feed.ex` | Pure feed generation. |
+| `lib/blogex/post.ex` | Pure post parsing/visibility helpers. |
+| `lib/blogex/external_links.ex` | Pure HTML rewriting for external links. |
+| `lib/blogex/registry.ex` | Module-registry lookup; no I/O. |
+| `lib/blogex/not_found_error.ex` | Exception definition. |
+| `lib/blogex/link_error.ex` | Exception definition. |
+
+| Out of scope | Why |
+|--------------|-----|
+| `lib/blogex.ex` | Facade + moduledoc: delegation and docs, no logic to mutate. |
+| `lib/blogex/blog.ex` | A `defmacro __using__` that expands `NimblePublisher`; mutating macro bodies produces noise, not behaviour. |
+| `lib/blogex/router.ex` | Phoenix router DSL (`use`, `scope`, `get`); macro-generated. |
+| `lib/blogex/components.ex` | Phoenix component DSL (`use`, `attr`, `slot`, `~H`). |
+| `lib/blogex/layout.ex` | LiveView layout DSL. |
+
+`mutation_scope_fast.txt` is a subset (`link_validator`, `seo`, `post`) so the
+inner loop stays seconds long; a pin in
+`test/mutation_scope_test.exs` fails if it ever drifts outside the audited
+scope. Another pin fails if any scoped path stops existing, so a rename cannot
+silently leave the sensor reporting a score for a reduced set.
+
+## Mutators
+
+The aliases pass an explicit curated set (muex 0.12 CLI names):
+
+```
+arithmetic, boolean, comparison, conditional, function_call, literal, return_value
+```
+
+Macro/`use`-based mutators are deliberately absent. The scoped modules are plain
+functions, and the macro-heavy modules are out of scope; macro mutations produce
+false-positive survivors rather than behavioural questions. The `arithmetic`,
+`boolean`, `comparison` and `conditional` families are the behavioural core;
+`function_call`, `literal` and `return_value` catch removed calls, changed
+constants and stubbed return values.
+
+## Baseline (blogex, 2026-10-28)
+
+Command: `mix mutate` (default muex optimization, `--fail-at 0`).
+Result: **19 mutants, 12 killed, 0 survived, 0 invalid, 7 equivalent — 100%.**
+Wall clock: 3.5 s. The run that found the survivor below was 11 killed / 1
+survived (91.67%) before its pin landed.
+
+| Module | Mutants | Killed | Survived | Equivalent |
+|--------|--------:|-------:|---------:|-----------:|
+| `link_validator.ex` | 7 | 4 | 0 | 3 |
+| `external_links.ex` | 6 | 5 | 0 | 1 |
+| `registry.ex` | 4 | 1 | 0 | 3 |
+| `post.ex` | 1 | 1 | 0 | 0 |
+| `seo.ex` | 1 | 1 | 0 | 0 |
+| `feed.ex` | 0 | 0 | 0 | 0 |
+| `not_found_error.ex` | 0 | 0 | 0 | 0 |
+| `link_error.ex` | 0 | 0 | 0 | 0 |
+
+muex's file analyser gives `not_found_error.ex` a complexity score of 0 and
+skips it under the default filter. The scope file still lists it: it is pure
+code and stays audited if it grows logic. `feed.ex` is analysed but the curated
+mutator set generates nothing for it.
+
+### Survivors found and killed
+
+| Mutant | Resolution |
+|--------|------------|
+| `link_validator.ex:201` — `Boolean: and to or` in `validate_body/3`'s guard `when is_binary(body) and is_atom(blog)` | Genuine: the mutant let a string blog id through the guard. Pinned by two `FunctionClauseError` tests in `test/blogex/link_validator_test.exs`; `mix mutate.fast` then reports 0 survivors. |
+
+## False-positive classes
+
+These are not gaps in behaviour; do not weaken a test to "kill" them.
+
+- **Commutative argument swaps.** `function_call` mutates `a and b` to
+  `b and a`, `a == b` to `b == a`, and `a || b` to `b || a`. These compile to
+  the same behaviour, and muex's trivial-compiler-equivalence check marks them
+  `equivalent` (7 of the 19 baseline mutants). They are noise from the
+  mutator, not survivors.
+- **Removed log lines.** A removed `Logger` call changes no return value; the
+  suite cannot observe it. Such a mutant is documented here instead of adding
+  a log-capture assertion that pins an implementation detail.
+- **`and`/`or` in guards is *not* automatically a false positive.** The
+  baseline survivor above was a real guard-contract gap; only commutative
+  rewrites are equivalent.
+- **Invalid mutants.** Mutated code that does not compile is counted
+  `invalid`, not `survived`, and excluded from the score.
+
+## Deeper runs
+
+muex's default `balanced` optimization reduced 1304 generated mutants to 12
+tested ones (99.1%) on the blogex full scope. That is what keeps the sensor in
+seconds, but it is sampled, not exhaustive. `mix mutate --no-optimize` tests
+everything; on the three-module fast scope it took ~2 min 15 s and produced
+434 mutants (329 killed, 19 survived, 83 invalid, 3 equivalent) before the
+guard pin. Use it when triaging a specific module, not in a loop.
+
+Re-baseline after a yak that changes the scoped modules (notably the Boundary
+work, which adds `use` lines inside them); do not compare scores across such a
+change.
+
+## Scope: app
+
+**Deliberately empty; no app baseline was run.** The candidate pure modules
+are `Firehose.Accounts.User`, `Firehose.Accounts.Scope` and
+`Firehose.Accounts.UserToken`, but every test that exercises them lives in
+`test/firehose/accounts_test.exs`, which uses `Firehose.DataCase`, fixtures and
+`Repo`; the SQL sandbox needs PostgreSQL. The yak's Q45 decision was: use those
+modules only after confirming their tests need no database, otherwise leave the
+app scope empty. They need one, so the scope is empty.
+
+Consequences, all pinned by `app/test/mutation_scope_test.exs` and
+`app/test/mutation_config_test.exs`:
+
+- Both app scope files list nothing, so `mix mutate` / `mix mutate.fast` raise
+  `mutation scope … is empty; refusing to run the mutation sensor over nothing`.
+- Ad-hoc `--files` arguments do not bypass the guard: the scope is validated
+  before muex arguments are built.
+- No scope entry may ever begin with `lib/firehose_web` (controllers,
+  LiveViews, components).
+
+There is no app mutation baseline to report. Do not add one until the scoped
+modules have tests that run without the database; change the scope file and
+these pins in the same commit.