Skip to content

feat(eval): add lifecycle extensions and agent rules#1607

Merged
christso merged 2 commits into
mainfrom
feat/av-kfik-14-extensions
Jul 2, 2026
Merged

feat(eval): add lifecycle extensions and agent rules#1607
christso merged 2 commits into
mainfrom
feat/av-kfik-14-extensions

Conversation

@christso

@christso christso commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Recovery

Recovers closed PR #1605 after its stacked base branch feat/av-kfik-4-templating was deleted following #1597 squash-merge.

  • Original preserved head: b7af94a60ca4a153aec6baf57baf6705d6f05600
  • Recovered head on main: ca0019330e76101e94ea422d4a5fcca2a32f64e8
  • git range-diff 61bcec47..recover/pr1605-pre-rebase origin/main..HEAD reports both lifecycle commits as patch-equivalent.
  • Attempted to reopen feat(eval): add lifecycle extensions and agent rules #1605, but GitHub rejected it because the deleted base branch prevents changing PR state.

Summary

  • Adds promptfoo-compatible AgentV lifecycle extensions: beforeAll, beforeEach, afterEach, and afterAll.
  • Adds file://...:<hook> extension execution and the built-in agentv:agent-rules staging extension.
  • Keeps workspace.repos first-class: repos are materialized before extension hooks, and repo acquisition is not moved into extension semantics.
  • Exposes staged agent_rules_paths through provider context, result metadata, prepare manifests, generated schema, docs, and examples.
  • Preserves the feat(eval): add lifecycle extensions and agent rules #1605 review fixes from b7af94a6: conversation-mode teardown/result metadata, slot-scoped pooled beforeAll extension state, and beforeEach baseline refresh when hooks mutate files without returning state.

Validation

  • bun test packages/core/test/evaluation/extensions.test.ts packages/core/test/evaluation/orchestrator.test.ts packages/core/test/evaluation/workspace/setup.test.ts packages/core/test/evaluation/workspace/file-changes.test.ts packages/core/test/evaluation/suite-level-input.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts apps/cli/test/commands/prepare/prepare.test.ts (262 passing)
  • bun run --cwd packages/core typecheck
  • bun run --cwd packages/core lint
  • bun run --cwd packages/core build
  • bun run --cwd apps/cli build

Prior #1605 validation preserved for review context:

Private evidence:

  • agentv-private branch: evidence/av-kfik-14-extensions
  • Original evidence commit: 36424ef
  • Review-fix evidence commit: 1c16dc4, folder review-fix-dogfood/

Scope Notes

  • Legacy workspace.hooks command execution remains available for existing suites and reset policy; new docs and migrated examples route executable setup through top-level extensions.
  • Hard removal of remaining legacy command hooks is left out of this slice because examples and tests still exercise existing behavior.

Post-Deploy Monitoring & Validation

  • Watch CI for eval schema generation, core/CLI typecheck, and CLI prepare tests.
  • For early adopters, inspect run index.jsonl metadata for agent_rules_paths and provider request metadata in failed evals involving extensions.
  • Healthy signal: evals using workspace.repos plus agentv:agent-rules show staged paths under .agentv/agent-rules/** after workspace materialization.
  • Failure signal: setup failures mentioning agentv:agent-rules requires a materialized workspace, stale prepared agent_rules_paths, or missing file:// export functions.
  • Rollback trigger: extension-enabled evals fail before provider invocation despite valid workspace materialization.
  • Validation window: first CI run and first manual extension-enabled eval after merge; owner: AgentV maintainers.

@christso christso marked this pull request as ready for review July 2, 2026 14:26
@christso christso merged commit 4901729 into main Jul 2, 2026
8 checks passed
@christso christso deleted the feat/av-kfik-14-extensions branch July 2, 2026 14:27
@christso christso restored the feat/av-kfik-14-extensions branch July 2, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant