Skip to content

feat: add shadow-draft exporter and smooth the mutation pipeline#38

Merged
karngyan merged 1 commit into
mainfrom
feat/mutation-pipeline-ergonomics
Jul 17, 2026
Merged

feat: add shadow-draft exporter and smooth the mutation pipeline#38
karngyan merged 1 commit into
mainfrom
feat/mutation-pipeline-ergonomics

Conversation

@karngyan

Copy link
Copy Markdown
Contributor

Claim

The mutation lifecycle (propose → challenge → replay-draft → annotate → replay → shadow → install) was impassable for a normal user at the shadow step (no Shadow Suite exporter — you had to hand-author shadow-suite/0.1 JSON from the schema) and needlessly rough elsewhere. This PR makes the whole path passable and self-guiding without weakening any zero-permission, determinism, or evidence-traceability invariant.

Four fixes:

  1. mutations shadow-draft — mirrors replay-draft: mutations shadow-draft <ID> --suite <PATH> [--context-events N] [--force]. Extraction lives in a new extract_observation_draft in autophagy-shadow (the natural downstream home, mirroring replay's extract_review_draft). It emits a schema-valid shadow-suite/0.1: supporting evidence → observation drafted intervention_would_help: true, counterexamples → false, each with a human-readable note, exact source event ids, and observed trigger selectors. Observation ids (shd_<hex>) are deterministic SHA-256 of inputs (no clock, no randomness), and source event ids are unique across the suite (the contract requires it). Read-only: it reads the DB and writes the one file you name — no state transition.
  2. Flag harmonizationreplay gains canonical --suite; the former --scenarios stays accepted as a hidden alias. shadow gets the same --suite/--observations pair. replay-draft/shadow-draft and their evaluators now refer to one file by one name.
  3. Manifest-free deterministic synthesis — when the effective provider is deterministic and no --manifest/config path is given, synthesis falls back to a built-in reference manifest instead of erroring. An explicit --manifest still wins and validates strictly. Other providers still require a manifest, and that error now points at a complete example (docs/specs/synthesis/0.3/manifest/valid/claude_cli.json). Validation is not weakened.
  4. Actionable replay error — the unreviewed-outcomes failure now truncates the id list (first 3 + "and N more") and appends the exact remedy plus the suite path: edit <suite path> and set counterfactual_outcome to "expected_result" or "contradiction" for each listed scenario.

Plus one-line next-step hints to stderr (text mode only, never JSON stdout) after propose, challenge, replay-draft, replay (on pass), shadow-draft, and shadow (on pass), using real ids and paths.

Evidence

New tests:

  • autophagy-shadowevidence_extraction_produces_stable_schema_valid_shadow_draft: the draft is schema-valid against docs/specs/shadow/0.1/suite.schema.json, deterministic across repeated runs and reversed event order, has both would-help and no-op coverage, shd_ ids, notes, globally-unique source event ids, and is a legal input to evaluate.
  • autophagy-clideterministic_synthesis_needs_no_manifest: mutations synthesize with no --manifest reports provider=deterministic, manifest_path=builtin://deterministic, registers candidates; a non-deterministic provider without a manifest errors with the example-manifest pointer.
  • autophagy-climutation_pipeline_ergonomics_are_smooth_end_to_end: propose/challenge/replay/shadow next-step hints appear on stderr in text mode and never on JSON stdout; replay --suite works; shadow-draft emits schema-valid deterministic JSON, refuses to clobber without --force, overwrites identically with --force; shadow --suite passes.
  • autophagy-cli → existing milestone_demo_… updated: the unreviewed-replay error now asserts the scenario ids, the remedy text, and the suite path; the --scenarios alias is exercised.

mise run check — green (ci + docs + fmt + lint + test), 216s, all workspace tests pass.

Pipeline walked end-to-end on a demo DB (new ergonomics)

### import
11 lines · 11 events · 11 inserted · …

### synthesize (deterministic, NO --manifest)
provider=deterministic · model=deterministic-reference · model_used=false · network_used=false · remote_endpoint_allowed=false
mut_6b51ef…a0ee  Prevent repeated command failure: shell: mise run check  3 evidence  zero permissions  candidate

### propose (text — next-step hint on stderr)
mut_6b51ef…a0ee  …  candidate
next: autophagy mutations challenge mut_6b51ef…a0ee --check coincidence-considered --check sessions-comparable --check trigger-observable --check legitimate-uses-bounded --check equivalent-searched --check counterexamples-reviewed

### challenge
mut_6b51ef…a0ee  candidate -> challenged  changed=true
next: autophagy mutations replay-draft mut_6b51ef…a0ee --suite replay-suite.json

### shadow-draft (NEW — read-only, no transition)
mut_6b51ef…a0ee  4 observations · 3 intervention · 1 no-op  /tmp/…/shadow-review.json
next: autophagy mutations shadow mut_6b51ef…a0ee --suite /tmp/…/shadow-review.json (after confirming intervention_would_help for each observation)
# shadow-review.json → {"spec_version":"shadow-suite/0.1", …, "observation_id":"shd_7799f1…", "intervention_would_help":true, "note":"Draft observation from 1 exact supporting event(s) … Confirm the real outcome …"}

### replay-draft, then replay WITHOUT annotating (actionable error)
mut_6b51ef…a0ee  4 scenarios · 3 intervention · 1 no-op · 3 unreviewed  /tmp/…/replay-review.json
error: replay suite has 3 unreviewed scenario(s) (rps_7eb2bf…, rps_ac2ee7…, rps_bbbe12…); edit /tmp/…/replay-review.json and set counterfactual_outcome to "expected_result" or "contradiction" for each listed scenario

### replay against an annotated (passing) suite — canonical --suite
rpl_a14d24…  passed=true  3 success · 2 no-op · 0 contradiction · 0 false intervention  mutation_executed=false
next: autophagy mutations shadow-draft mut_6b51ef…a0ee --suite shadow-suite.json
# `--scenarios` alias still accepted — identical result

### shadow against an annotated (passing) suite — canonical --suite
shr_a5fd96…  passed=true  precision=10000 bps · recall=10000 bps · 0 false positives  mutation_applied=false
next: autophagy mutations install mut_6b51ef…a0ee --repository <repo> --target claude-code --confirm-permissions repo-skill-write

### install (dry-run preview)
ins_302b94…  .claude/skills/autophagy-6b51ef819f54/SKILL.md  dry-run  f0d84287…  claude_code_repo_skill

JSON mode confirmed clean: --output json mutations propose → stdout parses as JSON with no next:, stderr is 0 bytes.

Privacy implications

shadow-draft writes exactly one local file at the user-named --suite path. Its contents (AEP event ids and observed trigger selectors) are already stored locally in the user's database; nothing new is collected, nothing leaves the machine, no model or network is used, and no state transition or permission is granted by drafting. Default path stays local-only and offline-capable.

🤖 Generated with Claude Code

https://claude.ai/code/session_015LY7xtercMc3NtLeX2z4YV

Make the propose → challenge → replay-draft → replay → shadow-draft →
shadow → install lifecycle passable without hand-authoring contract JSON.

- Add `mutations shadow-draft <ID> --suite <PATH> [--context-events N]
  [--force]`, mirroring `replay-draft`. Extraction lives in
  autophagy-shadow's new `extract_observation_draft`, deriving a
  schema-valid `shadow-suite/0.1` from the mutation's own supporting
  evidence (drafted `intervention_would_help: true`) and counterexamples
  (drafted `false`). Observation ids (`shd_…`) are deterministic SHA-256
  of inputs — no clock, no randomness — and source event ids are unique
  across the suite. Read-only: it writes one user-named file and performs
  no state transition.
- Harmonize replay's flag: `--suite` is canonical; `--scenarios` stays as
  a hidden alias. Shadow gains the same `--suite`/`--observations` pair.
- Manifest-free deterministic synthesis: the built-in `deterministic`
  provider falls back to a built-in reference manifest when no
  `--manifest`/config path is given. Explicit manifests still win and
  validate strictly; other providers still require one, and that error
  now points at a complete example manifest.
- Actionable replay error: the unreviewed-scenarios failure now truncates
  the id list (first 3 + "and N more") and appends the exact remedy plus
  the suite path to edit.
- One-line next-step hints to stderr (text mode only, never JSON stdout)
  after propose, challenge, replay-draft, replay, shadow-draft, and
  shadow, using real ids and paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@karngyan
karngyan merged commit 4bc83ec into main Jul 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant