Skip to content

fix: report selector/index signature-grammar mismatch in efficacy#46

Merged
karngyan merged 1 commit into
mainfrom
fix/efficacy-grammar-mismatch
Jul 18, 2026
Merged

fix: report selector/index signature-grammar mismatch in efficacy#46
karngyan merged 1 commit into
mainfrom
fix/efficacy-grammar-mismatch

Conversation

@karngyan

Copy link
Copy Markdown
Contributor

Claim

autophagy mutations efficacy gave a misleading verdict when an installed mutation's trigger selector uses an older signature grammar than the index it is evaluated against.

The mutation mut_b13f2878…'s selector is failure/v1|shell|cd zuzoto && go build ./... 2>&1|exit:1, but the showcase database's event_signatures index was re-minted to grammar v2 by reindex --index-tool-input. The v1 operation key (operation/v1|…) then matches zero rows even though the 8 pre-install failures exist — they are simply indexed under v2. The report read 0 → 0 failures · no prior baseline · needs more observed occurrences, as if the failures never happened.

The honest statement is that the selector's grammar is older than the index's and the measurement cannot see those events. A v1 selector is not silently translated to v2 — the v2 normalization is not derivable from the v1 string.

Change

  • New insufficient_reasons variant selector_grammar_mismatch (efficacy result v0.1, additive enum member). It fires when a selector's grammar version is older than the newest grammar present in the index and the index holds no rows of that older grammar. A partial/skipped reindex that still carries the old grammar does not trip it (the selector can still match those rows).
  • autophagy-efficacy: EfficacyObservations gains index_grammar_versions; evaluate parses each selector's grammar version and compares.
  • autophagy-store: new read-only Store::signature_grammar_versions() (no migration).
  • CLI: renders the actionable sentence selector uses signature grammar v1 but the index is v2 — re-propose from current findings to measure efficacy, suppressing the consequential "needs …" reasons.
  • Docs: result.schema.json enum value, a real valid/selector_grammar_mismatch.json fixture (generated from the reindexed showcase DB), spec README, and an addendum to ADR 0015 justifying why an additive enum member is not a version bump.

Evidence

Real-data verification on the reindexed showcase database (AUTOPHAGY_CONFIG_DIR=$(mktemp -d)), --now 2026-11-16T00:00:00Z. The index holds only v2 signatures (34,911 rows, zero v1); the installed mutation's selector is v1.

Before

insufficient data · 0 → 0 failures (0.0/wk → 0.0/wk) · no prior baseline · 80.5% classifiable · needs more observed occurrences

After

insufficient data · 0 → 0 failures (0.0/wk → 0.0/wk) · no prior baseline · 80.5% classifiable · selector uses signature grammar v1 but the index is v2 — re-propose from current findings to measure efficacy

insufficient_reasons in the JSON report is now ["selector_grammar_mismatch", "sparse_occurrences"].

New unit tests: v1 selector + v2 index → mismatch reason; matching v2 grammars unaffected; an old selector still present in a mixed index is not flagged; store grammar-version enumeration. mise run check is fully green.

Privacy

None. The report continues to cite exact event_ids and counts only — never command text or event content. The new store query reads grammar-version tokens from the existing redaction-approved event_signatures projection; nothing new is persisted and nothing leaves the machine.

🤖 Generated with Claude Code

A mutation whose trigger selector uses signature grammar v1
(failure/v1|shell|…) evaluated against a database whose event_signatures
index was re-minted to grammar v2 (reindex --index-tool-input) reported
"insufficient data · 0 → 0 failures · no prior baseline · needs more
observed occurrences". The pre-install failures exist — they are indexed
under v2 — so the v1 operation key matches zero rows, and the report was
misleading rather than honest.

Efficacy now detects when a selector's grammar is older than the newest
grammar present in the index and the index holds no rows of that older
grammar, and emits the insufficient-data reason selector_grammar_mismatch
with the text "selector uses signature grammar v1 but the index is v2 —
re-propose from current findings to measure efficacy". A v1 selector is
never silently translated to v2 (v2 normalization is not derivable from
the v1 string).

Additive efficacy/0.1 enum variant; existing fixtures unchanged. Recorded
as an addendum to ADR 0015.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@karngyan
karngyan merged commit 88434eb into main Jul 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant