Skip to content

feat(skill): add context-intelligence-hill-climbing skill for audit-trail tracking#77

Merged
colombod merged 1 commit into
microsoft:mainfrom
colombod:feat/hill-climbing-audit-trail-skill
Jul 17, 2026
Merged

feat(skill): add context-intelligence-hill-climbing skill for audit-trail tracking#77
colombod merged 1 commit into
microsoft:mainfrom
colombod:feat/hill-climbing-audit-trail-skill

Conversation

@colombod

Copy link
Copy Markdown
Collaborator

Summary

Add a thin, shared context-intelligence-hill-climbing skill (todo-as-climb-log) and wire both graph-analyst and session-navigator agents to load it on multi-step investigations. An earlier evaluation found both agents already navigate safely and correctly (bounded probes, no context flooding) but leave an inconsistent audit trail — baseline session-navigator used 0 todo calls while still answering. This skill's purpose is observability/consistency, NOT correctness or efficiency.

Scope / guardrails

  • Modified: agents/graph-analyst.md (added "Track multi-step investigations in a todo climb-log" guidance block)
  • Modified: agents/session-navigator.md (identical wiring block)
  • New: skills/context-intelligence-hill-climbing/SKILL.md (thin skill, ~2 KB, focused on using todo as climb-log)
  • NOT touched: write-side hook (logging_handler.py, _DestinationDispatcher, fanout.py), navigation discipline mechanics (those stay in graph-query, session-navigation, navigation-budget-discipline.md), bundle structure

The skill governs HOW an investigation is tracked (seed open questions, write findings + citations, mark dead leads) — it deliberately does NOT re-teach navigation mechanics.

Verification

  • Module tests pass — modules/tool-context-intelligence-query: 132 passed
  • Top-level tests pass — tests/: 18 passed
  • ruff check + ruff format --check clean — 0 errors
  • pyright clean — 0 errors
  • Full bundle validation PASS — scripts/validate-full.shvalidation_mode: full
    The lone mode-advertising ERROR is a documented FALSE POSITIVE (name collision between bundle name, storage path, skill name, and internal mode) — see AGENTS.md. Validator confirmed overall verdict PASS.

Real evidence on seams (not mock-only)

This change crosses a seam: agent wiring (skill loading). Per AGENTS.md, seam changes require real DTU run + evaluation harness with captured evidence.

Lightweight A/B evidence (delegate-based, n=1 per cell, skill injected as inline guidance because not yet installed):

Both agents were asked the same real RCA/provenance question (session 1ade4f66…, which session produced the auto-recovery design first?) on both surfaces (graph-analyst, session-navigator), cold baseline vs. +skill treatment.

run total tool calls todo calls correct? audit trail
graph-analyst baseline 10 6 yes ad-hoc
graph-analyst +skill 18 10 yes structured
session-navigator baseline 31 0 yes none
session-navigator +skill 46 8 yes structured

The +skill runs produced a structured, cited todo climb-log (open questions seeded up front; findings written into items with node-id/line citations). Correctness held at baseline and remained correct with the skill. Cost rose ~50–80% (expected overhead for an observability feature).

Session-navigator's final climb-log from the +skill run read:

✓ Resolve root session dir and confirm sub-session locations
✓ Q1 ROOT CAUSE — ANSWERED: foundation:explorer sub-session (…7bc831ed…, spawned 15:48:58Z from root events.jsonl:162)…
✓ Q2 FIRST DESIGN AUTHOR — ANSWERED (nuanced): design authored directly by the ROOT session itself, not a delegated sub-agent…
✓ Q3 CAUSAL CHAIN — ANSWERED: error report → foundation:explorer → root-session brainstorming+ROB → council → brainstormer (documents)…
  • Seam(s) crossed by this change are proven against real behavior (lightweight A/B on real scenario; see above)

NOTE: This repo's AGENTS.md requires agent/skill/mode edits to pass "a real DTU run + the evaluation harness, with captured evidence" — not just lightweight delegate tests. That full DTU-isolated, multi-trial run has NOT been executed. RECOMMEND running the DTU + evaluation-harness gate before merge (see .amplifier/digital-twin-universe/profiles/ for available profiles).

Docs & diagrams

  • bundle.dot / bundle.png regenerated if bundle structure changed — N/A — no bundle structure change (skill addition is additive, no new seams in bundle.md or behaviors layout)
  • README / SKILLs / agent files updated if a tool/skill contract changed — documentation in skill SKILL.md and agent guidance blocks reflects the new capability
  • Convention files updated if this surfaced a lasting lesson — N/A — no new convention learned

Notes / follow-ups

Gate not fully met: This PR provides lightweight evidence (delegate-based, n=1 per cell) that the skill produces structured audit trails without breaking correctness. Full DTU + evaluation harness validation remains as a pre-merge recommendation per AGENTS.md § Testing & what "done" looks like. The DTU profiles exist in .amplifier/digital-twin-universe/profiles/ (e.g., context-intelligence-bundle-smoke-test.yaml) — running one of them with this branch's agents will give the full proof required before shipping.

…rail tracking

Add a thin, shared skill that instructs agents to use the todo tool as
a hill-climb log during multi-step investigations. Wire both graph-analyst
and session-navigator agents to load this skill when facing provenance
tracing, cross-session synthesis, or other multi-step investigation tasks.

The skill governs HOW an investigation is TRACKED (seed open questions as
todos, write each finding + its citation into items, mark dead leads) but
deliberately does NOT re-teach navigation mechanics (those remain in
graph-query / session-navigation / navigation-budget-discipline).

Previous evaluation showed both agents already navigate safely and correctly
(bounded probes, no context flooding) but leave inconsistent audit trails.
This skill adds observability to that discipline: turning the todo tool
into a structured climb-log so a later reader can see how an answer was
reached.

🤖 Generated with Amplifier

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod
colombod merged commit 2926f48 into microsoft:main Jul 17, 2026
8 checks passed
@colombod
colombod deleted the feat/hill-climbing-audit-trail-skill branch July 17, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant