docs(plan): eval authoring restructure — promptfoo superset (DRAFT)

[christso]

Summary

Draft design plan (no code changes) for re-aligning AgentV's eval authoring format with promptfoo — keeping snake_case — and borrowing runner/analytics from Margin-Lab/evals and transcripts/agentic-graders from vercel-labs/agent-eval.

Plan doc: docs/plans/promptfoo-aligned-eval-restructure.md

North-star

AgentV's eval contract is a strict superset of (snake_cased) promptfoo: any promptfoo config, mechanically snake_cased, is a valid AgentV eval with equivalent semantics; AgentV accepts more on top (bare-string asserts, workspace, gate, agentic judges, multi-turn, …).

Governing decisions (§2, resolved)

• Prefer promptfoo naming where functionally equivalent; keep AgentV only where semantics are better (gate > scalar threshold; repeat: {count, strategy, early_exit} > repeat: int; workspace/repo materialization; agentic judge).
• Hard deprecation (major version): removed keys (assertions, composite, eval_cases, grader name-as-metric, ${{ ENV }}) hard-error; one-shot codemod migrates.
• targets re-canonicalized as first-class system-under-test. Verified against promptfoo source + history: promptfoo targets is a 2024-05 redteam alias for the canonical providers. AgentV elevates targets (fits agent-eval) and demotes provider/apiId to the backend-kind vocabulary. It does not accept a top-level providers alias (that would overload AgentV's existing backend provider/sub-provider term); the promptfoo importer rewrites top-level providers: → targets: instead.
• Templating: {{ var }} (nunjucks, eval-time) + ${ENV} (docker/k8s style, config-time, replaces ${{ ENV }}) — no sigil collision.
• Optional test id: split stable derived test_id (identity/filter/compare) from display label (id → description → vars → Test #n).
• Bare-string assert shorthand kept (better semantics: N criteria → one judge call), desugars to a batched llm-rubric. One-way compat (promptfoo ⊆ AgentV) is the design.
• LLM-judge types consolidated into one promptfoo-named llm-rubric (folds AgentV's rubrics + agentic llm-grader in as optional fields).

Borrows

• margin-lab: instance-as-scheduling-unit, lease/heartbeat worker pool, infra-only retry, pass@k analytics via one pure Build().
• vercel-agent-eval: two-layer transcript (raw + normalized canonical tool_name enum + inlined summary), evidence-by-path agentic judge, judge pinning.

Open (implementation sequencing, non-blocking — §8)

1. Which exotic assertion types ship working vs accepted-but-stubbed at launch.
2. Redteam: accept-and-ignore the redteam: key to preserve superset parsing vs hard carve-out.
3. javascript/python desugar to code-grader vs distinct types.

Review asks

• Confirm the §2 contract decisions.
• Weigh in on the three §8 sequencing calls (leanings noted in the doc).

🤖 Generated with Claude Code

---

Implementation hand-off (beads)

Tracked under epic av-kfik. Workers have no context of the design conversation — read docs/plans/promptfoo-aligned-eval-restructure.md (this PR) + docs/adr/0015 first; each bead is self-contained and cites its plan section.

Ready now (no blockers):

• av-kfik.1 (P0) — Write authoring-contract + output-contract ADRs (anchor)
• av-kfik.3 (P1) — Group A: delete deprecated back-compat aliases (independent, parallel)

Unlocked after the schema anchor (av-kfik.2 ← .1):

• av-kfik.2 (P0) — Snake_cased promptfoo eval schema (Zod)
• av-kfik.4 — Templating: nunjucks vars + ${ENV} config env ← .2
• av-kfik.5 — prompts × targets matrix + instance expansion ← .2
• av-kfik.6 — Re-canonicalize targets; provider=backend; registry promptfoo shape ← .2
• av-kfik.7 — Assertion vocabulary + llm-rubric consolidation + grader execution ← .2
• av-kfik.8 — Two-layer transcript + canonical tool_name enum + summary ← .2
• av-kfik.9 — Datasets: file:// loading + __expected DSL ← .2
• av-kfik.10 — Runner: worker pool + reset workspace pool + --rerun-failed ← .5
• av-kfik.11 — Grading contract: assertion_results/summary/verdict/score + agentic judge ← .7
• av-kfik.12 — Output/artifact contract: queryable summary.json + .internal/ + timing→metrics merge ← .1, .5
• av-kfik.13 — Multi-turn: split execution/evaluation; drop window_size (ADR-0015) ← .4, .7
• av-kfik.14 — Extensions/workspace/agent-rules slice (PR docs(plans): add promptfoo-compatible extensions plan #1592 + amendments A1–A6) ← .2, .4
• av-kfik.15 — Hard-deprecation codemod for existing eval files ← .2, .6, .7, .13, .12
• av-kfik.16 — Docs + examples + live provider/grader dogfood ← .10, .11, .12, .14

Use bd ready to find unblocked work; bd show <id> for the full task spec + acceptance.

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	0bff279
Status:	✅  Deploy successful!
Preview URL:	https://2020d8e0.agentv.pages.dev
Branch Preview URL:	https://plan-promptfoo-aligned-eval.agentv.pages.dev

View logs

[christso]

Progress update on Group A cleanup / grader artifact wording:

• Pushed 5df2530f to fix the prior CI failures: Biome formatting plus regenerated skills-data/agentv-eval-writer/references/eval.schema.json.
• Updated ADR-0017 to clarify that per-criterion/per-aspect rows are generic AgentV grader assertion evidence, not g-eval-specific. All graders can emit assertions[]; multi-aspect graders emit multiple rows; grading.json keeps both flat assertions[] and nested graders[].assertions[].
• Local verification passed: bun run lint; bun test packages/core/test/evaluation/validation/eval-schema-sync.test.ts; bun test apps/cli/test/commands/eval/artifact-writer.test.ts; bun --filter @agentv/core test (2107 pass).

GitHub CI is running on head 5df2530f.

[christso]

Progress update on Group A alias cleanup:

• Pushed 90b662b3 removing deprecated exported aliases: EvalCase, OutputMessage, PASS_THRESHOLD, and getAgentvHome.
• Dropped the special results_by_project config deprecation path; it now falls through as an ordinary unexpected field. Dashboard docs now point directly to projects[].results.
• Removed the stale required_min_score row from the eval-writer rubric reference card.

Local verification passed: bun run lint; bun --filter @agentv/core typecheck; bun --filter @agentv/core build; focused paths/config/grader/parser tests; bun run typecheck.

GitHub CI is green on head 90b662b3.

[christso]

Progress checkpoint on head 76178419:

• Removed the programmatic TypeScript expected_output convenience alias from evaluate() inputs; expectedOutput remains the TS API, while YAML/wire expected_output remains the first-class golden/reference answer.
• Added hard-deprecation checks for inline tests and conversation turns that still pass expected_output to evaluate().
• Tightened ADR-0017 wording for the owner clarification: artifact assertion rows are the generic AgentV grader evidence channel. Every grader returns assertions[]; deterministic graders usually emit one row, while multi-aspect graders emit one row per criterion/aspect. This is not g-eval-specific.

Local verification:

• bunx biome format --write docs/adr/0017-output-artifact-and-workspace-resolver-contract.md packages/core/src/evaluation/evaluate.ts packages/core/test/evaluation/evaluate-enhanced.test.ts packages/core/test/evaluation/evaluate-programmatic-api.test.ts
• bun test packages/core/test/evaluation/evaluate-enhanced.test.ts packages/core/test/evaluation/evaluate-programmatic-api.test.ts (25 pass)
• bun --filter @agentv/core typecheck
• bun --filter @agentv/core build
• bun run lint
• bun run typecheck

CI is running for head 76178419.

[christso]

CI follow-up: head 76178419 is green (Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, Cloudflare Pages all SUCCESS).

[christso]

Progress checkpoint on head 97a95244:

• Removed built-in target provider alias sprawl from Group A: azure-openai, google, google-gemini, codex-cli, copilot, copilot_sdk, pi, claude-code, cc-mirror, bedrock, and vertex no longer canonicalize to built-in providers.
• Kept canonical providers: azure, gemini, codex, copilot-cli, copilot-sdk, pi-coding-agent, pi-cli, claude/claude-cli/claude-sdk.
• Preserved discovered-provider fallback for .agentv/providers/<kind>.ts; removed aliases now validate as unknown provider names instead of built-in aliases.
• Removed stale cc-mirror target docs and updated Copilot docs to use provider: copilot-cli.

Local verification:

• bunx biome format --write apps/web/src/content/docs/docs/targets/coding-agents.mdx packages/core/src/evaluation/providers/targets.ts packages/core/src/evaluation/providers/types.ts packages/core/src/evaluation/validation/targets-validator.ts packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts
• bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts (77 pass)
• bun --filter @agentv/core typecheck
• bun --filter @agentv/core build
• bun run lint
• bun run typecheck

CI is running for head 97a95244.

[christso]

CI follow-up: head 97a95244 is green (Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, Cloudflare Pages all SUCCESS).

[christso]

Follow-up clarification pushed in cee6a8d.\n\n- Tightened ADR-0016 and the restructure plan to state that artifact assertion rows are the generic AgentV grader contract, not a g-eval-specific behavior.\n- Clarified the terminology: every grader returns assertions[]; deterministic graders usually emit one row; multi-aspect graders emit one row per authored check/result unit; structured g-eval emits one row per criterion because criteria are one such multi-aspect unit.\n- Verification: git diff --check.\n- GitHub checks are green on cee6a8d: Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, Cloudflare Pages.

[christso]

Progress update for Group A cleanup: pushed d74a7da.\n\n- Removed target-level log_format/log_output_format alias support and now reject those fields with stream_log guidance.\n- Kept canonical stream_log and wired it into Copilot/Pi/Claude logger format resolution; stream_log: raw maps to JSON event logs, stream_log: summary maps to summary logs, and stream_log: false disables those stream loggers.\n- Updated target validation so provider-specific legacy log_format settings no longer pass as known settings.\n- Local verification: bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts; bun --filter @agentv/core typecheck; bun --filter @agentv/core build; bun run lint; bun run typecheck; git diff --check.\n- GitHub checks are green on d74a7da: Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, Cloudflare Pages.

[christso]

Progress update for av-kfik.3: pushed e27b6aa0 (chore(evals): remove numeric required thresholds).

What changed:

• Removed the legacy required: 0.7 threshold shim from grader parsing and orchestration. required is now boolean-only; custom thresholds use required: true + min_score.
• Updated core/programmatic/SDK types, eval validation schema, generated eval.schema.json, docs, examples, and the eval-writer skill reference.
• Numeric required values now fail parser loading with migration guidance and validator warnings point to min_score.

Local verification:

• bun test packages/core/test/evaluation/loaders/grader-parser.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts packages/core/test/evaluation/orchestrator.test.ts packages/sdk/test/grader-helpers.test.ts (285 pass)
• bun --filter @agentv/core typecheck
• bun --filter @agentv/sdk typecheck
• bun --filter @agentv/sdk build
• bun run lint
• bun run typecheck
• git diff --check
• jq empty skills-data/agentv-eval-writer/references/eval.schema.json

GitHub CI on head e27b6aa0 is green: Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, and Cloudflare Pages all succeeded.

[christso]

Small follow-up for av-kfik.3: pushed eba1442e (chore(targets): align azure api_format removal error).

This just aligns the runtime Azure target error with the validator/docs wording: Azure api_format now consistently says the field has been removed. api_format remains supported for the non-Azure target types that still need it, including OpenAI/Codex/Copilot local-proxy configs.

Local verification:

• bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/validation/targets-validator.test.ts (80 pass)
• bun --filter @agentv/core typecheck
• bun run lint
• git diff --check

GitHub CI on head eba1442e is green: Build, Typecheck, Lint, Test, Check Links, Validate Marketplace, Validate Evals, and Cloudflare Pages all succeeded.

[christso]

Live dogfood update for av-kfik.3 at head eba1442e.

• Ran a 1-case live eval through the local OpenAI-compatible endpoint using base_url=http://127.0.0.1:10531/v1, provider: codex / api_format: responses for the agent target, and provider: openai / api_format: chat for the grader target. Env refs were used in YAML for base URL, API key, and model; command-level values were LOCAL_OPENAI_PROXY_API_KEY=dummy and LOCAL_OPENAI_PROXY_MODEL=gpt-5.4-mini.
• Command: bun apps/cli/src/cli.ts eval run /tmp/agentv-av-kfik3-dogfood/dogfood.eval.yaml --targets /tmp/agentv-av-kfik3-dogfood/targets.yaml --target local-codex-agent --workers 1
• Result: PASS, 1/1, score 0.90. Run bundle: .agentv/results/2026-07-02T07-47-33-344Z.
• Artifact check: grading.json.assertions[] has 2 rows matching the two authored score-range criteria, and grading.json.graders[0].assertions[] has the same 2 rows. index.jsonl.scores[0].type is llm-grader, with scores[0].assertions[] also carrying the 2 rows and details.raw_scores showing boolean-required=8, min-score-threshold=10.
• Conclusion: assertion/artifact rows are the generic AgentV grader evidence contract, not G-Eval-specific. Score-range rubrics emit one row per criterion; deterministic and code graders emit whatever assertion rows their checks/scripts return.

Private evidence pushed to agentv-private:evidence/av-kfik3-live-dogfood-2026-07-02 at commit aa7f3a5.

[christso]

av-kfik.2 schema/validation slice pushed in 8b261b2.\n\nChanges:\n- Zod eval schema accepts snake_cased promptfoo-shaped prompts/targets/default_test/assert/scenarios/derived_metrics/output_path/env/nunjucks_filters/extensions/evaluate_options fields.\n- Tests can omit id and use prompts/provider_output; canonical assert is accepted at suite/default_test/test levels.\n- Validator keeps top-level providers as an unknown-field warning, not a live alias for targets.\n- Regenerated skills-data/agentv-eval-writer/references/eval.schema.json.\n\nVerification:\n- bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts\n- bun --filter @agentv/core typecheck\n- bun --filter @agentv/core lint\n- bun run lint\n- git diff --check\n\nBoundary: runtime parser/execution for promptfoo assert/g-eval remains in av-kfik.7+ follow-up work; this commit is schema/validation only.

[christso]

av-kfik.1 ADR polish pushed in 916eccd.\n\nChange: removed the contradictory duplicate standalone Accepted line from both superseded ADR-0013 files, so the status blocks now clearly read Accepted-then-superseded by ADR-0016.\n\nVerification:\n- bun run lint\n- git diff --check\n\nNo runtime impact.