firehose/doc/mutation-testing.md

Mutation testing with muex

Mutation testing is a sensor, not a gate. It deliberately breaks production code one mutant at a time and re-runs the suite: a mutant killed by a test is evidence the test pins that behaviour; a mutant that survives is a question for a Lens session. Runs always pass --fail-at 0, so a survivor is reported and triaged, never a failing build.

Nothing here is wired into VERIFY_SENSORS. make check, make verify, coverage floors and .githooks/pre-push are unchanged by this yak.

How to run it

Both apps declare two aliases in mix.exs; both read a plain-text scope file at run time, so the file stays the single source of truth.

Command	Scope	Blogex wall clock (2026-10-28)
mix mutate.fast	mutation_scope_fast.txt (3 modules)	~3 s
mix mutate	mutation_scope.txt (8 modules)	~4 s

The app declares the same aliases but both scope files are empty (see "Scope: app" below), so mix mutate there raises a clear Mix error instead of running a sensor over nothing.

Extra arguments pass through to muex, so a one-off target is explicit:

cd blogex
mix mutate --files lib/blogex/seo.ex          # one module, ad hoc
mix mutate.fast --no-optimize                 # full mutant list, slower

The aliases pin --fail-at 0 first, so an ad-hoc flag cannot turn the sensor into a gate. mix mutate does not enable --cover: muex runs each mutant's suite under MIX_ENV=test, and coverage would let a code-deleting mutant trip the project's coverage floor and abort the run.

Scope: blogex

In scope (mutation_scope.txt)	Why
lib/blogex/link_validator.ex	Pure link parsing/validation; the yak's example module.
lib/blogex/seo.ex	Pure sitemap/JSON-LD/OpenGraph string building.
lib/blogex/feed.ex	Pure feed generation.
lib/blogex/post.ex	Pure post parsing/visibility helpers.
lib/blogex/external_links.ex	Pure HTML rewriting for external links.
lib/blogex/registry.ex	Module-registry lookup; no I/O.
lib/blogex/not_found_error.ex	Exception definition.
lib/blogex/link_error.ex	Exception definition.

Out of scope	Why
lib/blogex.ex	Facade + moduledoc: delegation and docs, no logic to mutate.
lib/blogex/blog.ex	A defmacro __using__ that expands NimblePublisher; mutating macro bodies produces noise, not behaviour.
lib/blogex/router.ex	Phoenix router DSL (use, scope, get); macro-generated.
lib/blogex/components.ex	Phoenix component DSL (use, attr, slot, ~H).
lib/blogex/layout.ex	LiveView layout DSL.

mutation_scope_fast.txt is a subset (link_validator, seo, post) so the inner loop stays seconds long; a pin in test/mutation_scope_test.exs fails if it ever drifts outside the audited scope. Another pin fails if any scoped path stops existing, so a rename cannot silently leave the sensor reporting a score for a reduced set.

Mutators

The aliases pass an explicit curated set (muex 0.12 CLI names):

arithmetic, boolean, comparison, conditional, function_call, literal, return_value

Macro/use-based mutators are deliberately absent. The scoped modules are plain functions, and the macro-heavy modules are out of scope; macro mutations produce false-positive survivors rather than behavioural questions. The arithmetic, boolean, comparison and conditional families are the behavioural core; function_call, literal and return_value catch removed calls, changed constants and stubbed return values.

Baseline (blogex, 2026-10-28)

Command: mix mutate (default muex optimization, --fail-at 0). Result: 19 mutants, 12 killed, 0 survived, 0 invalid, 7 equivalent — 100%. Wall clock: 3.5 s. The run that found the survivor below was 11 killed / 1 survived (91.67%) before its pin landed.

Module	Mutants	Killed	Survived	Equivalent
link_validator.ex	7	4	0	3
external_links.ex	6	5	0	1
registry.ex	4	1	0	3
post.ex	1	1	0	0
seo.ex	1	1	0	0
feed.ex	0	0	0	0
not_found_error.ex	0	0	0	0
link_error.ex	0	0	0	0

muex's file analyser gives not_found_error.ex a complexity score of 0 and skips it under the default filter. The scope file still lists it: it is pure code and stays audited if it grows logic. feed.ex is analysed but the curated mutator set generates nothing for it.

Survivors found and killed

Mutant	Resolution
link_validator.ex:201 — Boolean: and to or in validate_body/3's guard when is_binary(body) and is_atom(blog)	Genuine: the mutant let a string blog id through the guard. Pinned by two FunctionClauseError tests in test/blogex/link_validator_test.exs; mix mutate.fast then reports 0 survivors.

False-positive classes

These are not gaps in behaviour; do not weaken a test to "kill" them.

• Commutative argument swaps. function_call mutates a and b to b and a, a == b to b == a, and a || b to b || a. These compile to the same behaviour, and muex's trivial-compiler-equivalence check marks them equivalent (7 of the 19 baseline mutants). They are noise from the mutator, not survivors.
• Removed log lines. A removed Logger call changes no return value; the suite cannot observe it. Such a mutant is documented here instead of adding a log-capture assertion that pins an implementation detail.
• and/or in guards is not automatically a false positive. The baseline survivor above was a real guard-contract gap; only commutative rewrites are equivalent.
• Invalid mutants. Mutated code that does not compile is counted invalid, not survived, and excluded from the score.

Deeper runs

muex's default balanced optimization reduced 1304 generated mutants to 12 tested ones (99.1%) on the blogex full scope. That is what keeps the sensor in seconds, but it is sampled, not exhaustive. mix mutate --no-optimize tests everything; on the three-module fast scope it took ~2 min 15 s and produced 434 mutants (329 killed, 19 survived, 83 invalid, 3 equivalent) before the guard pin. Use it when triaging a specific module, not in a loop.

Re-baseline after a yak that changes the scoped modules (notably the Boundary work, which adds use lines inside them); do not compare scores across such a change.

Scope: app

Deliberately empty; no app baseline was run. The candidate pure modules are Firehose.Accounts.User, Firehose.Accounts.Scope and Firehose.Accounts.UserToken, but every test that exercises them lives in test/firehose/accounts_test.exs, which uses Firehose.DataCase, fixtures and Repo; the SQL sandbox needs PostgreSQL. The yak's Q45 decision was: use those modules only after confirming their tests need no database, otherwise leave the app scope empty. They need one, so the scope is empty.

Consequences, all pinned by app/test/mutation_scope_test.exs and app/test/mutation_config_test.exs:

• Both app scope files list nothing, so mix mutate / mix mutate.fast raise mutation scope … is empty; refusing to run the mutation sensor over nothing.
• Ad-hoc --files arguments do not bypass the guard: the scope is validated before muex arguments are built.
• No scope entry may ever begin with lib/firehose_web (controllers, LiveViews, components).

There is no app mutation baseline to report. Do not add one until the scoped modules have tests that run without the database; change the scope file and these pins in the same commit.