feat(eval): add lifecycle extensions and agent rules#1607
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recovery
Recovers closed PR #1605 after its stacked base branch
feat/av-kfik-4-templatingwas deleted following #1597 squash-merge.b7af94a60ca4a153aec6baf57baf6705d6f05600main:ca0019330e76101e94ea422d4a5fcca2a32f64e8git range-diff 61bcec47..recover/pr1605-pre-rebase origin/main..HEADreports both lifecycle commits as patch-equivalent.Summary
beforeAll,beforeEach,afterEach, andafterAll.file://...:<hook>extension execution and the built-inagentv:agent-rulesstaging extension.workspace.reposfirst-class: repos are materialized before extension hooks, and repo acquisition is not moved into extension semantics.agent_rules_pathsthrough provider context, result metadata, prepare manifests, generated schema, docs, and examples.b7af94a6: conversation-mode teardown/result metadata, slot-scoped pooled beforeAll extension state, and beforeEach baseline refresh when hooks mutate files without returning state.Validation
bun test packages/core/test/evaluation/extensions.test.ts packages/core/test/evaluation/orchestrator.test.ts packages/core/test/evaluation/workspace/setup.test.ts packages/core/test/evaluation/workspace/file-changes.test.ts packages/core/test/evaluation/suite-level-input.test.ts packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts apps/cli/test/commands/prepare/prepare.test.ts(262 passing)bun run --cwd packages/core typecheckbun run --cwd packages/core lintbun run --cwd packages/core buildbun run --cwd apps/cli buildPrior #1605 validation preserved for review context:
bun run lintbun run typecheckb7af94a6: https://github.com/EntityProcess/agentv/actions/runs/28592955569http://127.0.0.1:10531/v1: PASS, 1/1, mean score 100%.gpt-5.3-codex-spark: PASS, 1/1, mean 100%.Private evidence:
agentv-privatebranch:evidence/av-kfik-14-extensions36424ef1c16dc4, folderreview-fix-dogfood/Scope Notes
workspace.hookscommand execution remains available for existing suites and reset policy; new docs and migrated examples route executable setup through top-levelextensions.Post-Deploy Monitoring & Validation
index.jsonlmetadata foragent_rules_pathsand provider request metadata in failed evals involving extensions.workspace.reposplusagentv:agent-rulesshow staged paths under.agentv/agent-rules/**after workspace materialization.agentv:agent-rules requires a materialized workspace, stale preparedagent_rules_paths, or missingfile://export functions.