feat: track post-install mutation efficacy with deterministic recurrence windows

[karngyan]

Claim

Once a mutation is installed, the only honest question left is empirical: does the failure signature it addresses recur less after installed_at? This adds a deterministic, model-free, observational answer — a versioned efficacy/0.1 report comparing two equal-length windows anchored on the install timestamp. No causal claim, no model call, and no lifecycle coupling (efficacy never advances or retires a mutation). It is the sibling of the shadow crate in shape and spirit, closing the loop from "installed" to "did it help".

What's here

• ADR 0015 — observational-not-causal stance, symmetric window rule, exact verdict thresholds, the matching rule (below), why no lifecycle coupling, and privacy.
• Spec docs/specs/efficacy/0.1 — result.schema.json + valid/+invalid/ fixtures, exercised by round-trip tests.
• Migration 0003 (schema v3) — STRICT mutation_efficacy + mutation_efficacy_evidence, mirroring the shadow tables. History is intentional (many reports per mutation over time). The evidence-removal trigger deletes only the report when a cited event is removed — never the candidate — precisely because efficacy gates no lifecycle state.
• New crate autophagy-efficacy — pure, deterministic evaluation (model.rs/evaluate.rs/validate.rs), unit-tested. The store gathers occurrences + coverage; the CLI wires them — that wiring is the crate's executable vertical slice.
• Store — register_efficacy (content-hash immutable like register_shadow, but no state transition), list_efficacy, efficacy_occurrences, efficacy_status_summary.
• CLI — autophagy mutations efficacy <id> (evaluate + register + print, with --now for a reproducible/backdated clock); mutations show gains a latest-efficacy line; status gains a one-line installed-mutation efficacy summary (only when installations exist). Humanized text (percent, relative time, per-week rates); full report under --output json.

Selector-form gotcha (resolved empirically)

Against a throwaway copy of the real DB (69,400 events; 34,911 indexed signatures): both registered candidates carry failure/v1|…|exit:1 selectors, while event_signatures holds operation/v1|… rows exclusively. The failure grammar never appears in the index, so a failure selector cannot be matched against it directly.

The matching rule (failure_signature_recurrence) therefore decomposes the selector into its operation signature operation/<v>|<tool>|<command> plus an exit code, and counts events that are indexed under that operation signature and are tool.failed and carry the matching exit code. (Command text can contain |, so the failure form is parsed by trimming the trailing |exit:<code>.) For the go build candidate the operation signature resolves to 56 indexed rows — 28 tool.called, 25 tool.completed (exit 0), and exactly 3 tool.failed (exit 1), the three the finding cites.

Coverage diagnostics (GOTCHA 2)

The index is partial by construction, so the report always states total_failures vs classifiable_failures and never silently undercounts: coverage below 50% forces insufficient_data (partial_index_coverage).

Real-data evidence

Drove one candidate on the DB copy through challenge → replay (12 scenarios, passed) → shadow (12 observations, 100% precision, passed) → install into a scratch git repo, producing a real installed_at = 2026-07-18T20:54:57Z.

Evaluated at the real install instant, the post-window is 0 days → honest insufficient_data. Evaluated at a later clock (same real installed_at, --now 2026-11-10), the three historical failures land in the pre-window and none recur:

eff_74cd633d…  improved · 3 → 0 failures (0.2/wk → 0.0/wk) · -100.0% · 80.4% classifiable   114d window   model_used=false

mutations show (audit tail):

… installation ins_56ec8ffe…  installed  .claude/skills/autophagy-a87b7c83aa9b/SKILL.md
14s ago  efficacy eff_74cd633d…  improved · 3 → 0 failures (0.2/wk → 0.0/wk) · -100.0% · 80.4% classifiable

status:

Lessons    1 active · 1 awaiting review
           1 installed · 1 improved · 0 regressed · 0 unchanged · 0 inconclusive

The report cites the exact three go build failure event_ids, and the 80.4% coverage figure (304 of 425 in-span tool.failed indexed) shows the partial-index diagnostic surfacing rather than hiding.

Test coverage

• autophagy-efficacy: 9 tests — each verdict, threshold edges, coverage, content-id determinism, backwards-clock rejection, and schema/fixture round-trip.
• autophagy-store: 6 tests — matching rule + coverage, span bounding + unparsed selectors, append-only/immutable/lifecycle-free registration, inconsistent-registration rejection, evidence-removal cascade (report dropped, candidate kept), latest-verdict status summary.
• autophagy-cli: full lifecycle test extended through mutations efficacy (valid report, model_used=false, registered, no state change) and refusal once retired.
• Migration adoption/schema tests updated to v3.

Privacy

Strictly subtractive on derived data. The report cites exact event_ids and integer counts only — never command text or event content. Occurrences are matched through the same redaction-approved signature index that already gates search, so efficacy can only see what redaction already approved. No new source text is persisted; nothing leaves the machine.

Quality gate

mise run check is fully green (ci + docs + fmt + lint + test).

🤖 Generated with Claude Code