feat(core): adopt nunjucks eval templating

[christso]

Summary

AgentV eval files can now use Nunjucks as the shared templating language for per-case data and config-load environment values. Authors can render {{ vars.foo }}, loops, filters, and custom nunjucks_filters in templated eval fields, while config env interpolation moves to {{ env.VAR }} / {{ env.VAR | default("x") }} without colliding with runtime shell $VAR or ${VAR} in CLI target commands.

This also covers promptfoo-style row expansion for top-level string-array vars and renders chat-array prompt strings before parsing them into message arrays. The docs and examples now show the new vars and env namespaces.

Follow-up fix c0b8f47c makes target secret fields accept whole-value {{ env.* }} templates while still rejecting inline templated secrets such as Bearer {{ env.API_KEY }}.

Related: Bead av-kfik.4

Validation

• git diff --check
• bun --filter @agentv/core typecheck
• bun --filter @agentv/core lint
• bun run lint
• bun --filter @agentv/core build
• bun --filter @agentv/core test (2128 pass)
• bun run validate:examples (62 valid)
• bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/interpolation.test.ts packages/core/test/evaluation/interpolation-integration.test.ts packages/core/test/evaluation/suite-level-input.test.ts (132 pass)
• bun run --cwd packages/core typecheck
• bun run --cwd packages/core lint
• bun run --cwd packages/core build
• agentv validate against private dogfood eval + targets YAML
• Live provider/grader dogfood against http://127.0.0.1:10531/v1, model gpt-5.3-codex-spark: PASS 4/4, mean score 100%
• GitHub CI green on c0b8f47c

Private evidence: EntityProcess/agentv-private:evidence/av-kfik4-templating at d945e7a, folder dogfood/target-env-templates/.

Post-Deploy Monitoring & Validation

No additional production monitoring required. This is local/CI eval-loading behavior with no deployed service surface; validation should come from CI, the live dogfood run, and a release smoke using an eval file with {{ vars.* }}, custom filters, array expansion, chat-array prompts, and {{ env.* }} config values.

Failure signals: eval YAML that previously validated now fails during config loading, target secret fields unexpectedly reject whole-value {{ env.* }} references, shell command targets unexpectedly lose $VAR / ${VAR} literals, or custom filter imports fail in the packaged CLI runtime. Rollback trigger: revert this PR before release if CI or live dogfood finds a templating regression.

---

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	c654e97
Status:	✅  Deploy successful!
Preview URL:	https://36b0ed96.agentv.pages.dev
Branch Preview URL:	https://feat-av-kfik-4-templating.agentv.pages.dev

View logs

[christso]

Dogfood gate completed and pushed in c0b8f47c (fix(eval): resolve env templates in target secrets).

The live dogfood initially exposed a real gap: target secret fields rejected {{ env.LOCAL_OPENAI_PROXY_API_KEY }} because the target resolver only recognized legacy ${{ VARIABLE }} secret references. The fix lets secret fields resolve whole-value {{ env.* }} templates while still rejecting inline templated secrets such as Bearer {{ env.API_KEY }}.

Validation now passed:

• bun test packages/core/test/evaluation/providers/targets.test.ts packages/core/test/evaluation/interpolation.test.ts packages/core/test/evaluation/interpolation-integration.test.ts packages/core/test/evaluation/suite-level-input.test.ts (132 pass)
• bun run --cwd packages/core typecheck
• bun run --cwd packages/core lint
• bun run --cwd packages/core build
• git diff --check
• agentv validate against the dogfood eval and targets YAML
• Live provider/grader dogfood against http://127.0.0.1:10531/v1, model gpt-5.3-codex-spark: PASS 4/4, mean score 100%

Private evidence: EntityProcess/agentv-private:evidence/av-kfik4-templating at d945e7a, folder dogfood/target-env-templates/. It includes the initial failing bundle and final passing bundle.

[christso]

Independent review findings

I reviewed PR #1597 at c0b8f47c against origin/main, including the Nunjucks env/vars templating paths, target secret {{ env.* }} behavior, docs/examples, and the posted dogfood evidence. CI is green and the PR includes live dogfood evidence, but I found two issues to address before merge.

1. P1 - Secret fields can still contain literal/composed secrets through whole {{ env.* }} templates

   The secret-field guard only checks WHOLE_ENV_TEMPLATE_PATTERN (packages/core/src/evaluation/providers/targets.ts:823) before rendering (packages/core/src/evaluation/providers/targets.ts:2343). That regex accepts any template that starts with {{ env. and ends at a later }}, so secret fields can still contain literal material inside a filter or multiple env outputs while passing the "whole value" guard. I verified both of these resolve today:

   api_key: '{{ env.OPENAI_API_KEY | default("hardcoded-secret") }}'
   api_key: '{{ env.OPENAI_API_KEY }}{{ env.OPENAI_API_KEY_2 }}'

   The first one accepts a committed fallback secret when the env var is missing; the second renders to a composed secret (onetwo). The current regression test only covers the Bearer {{ env.OPENAI_API_KEY }} prefix case (packages/core/test/evaluation/providers/targets.test.ts:612), so this bypass is untested. For secret fields, tighten the accepted shape to a single env lookup (or an explicitly safe parsed subset) and add tests for default-filter literals, adjacent env refs, and extra template output.

2. P2 - Public eval-file docs still teach the removed ${{ VAR }} env syntax

   apps/web/src/content/docs/docs/evaluation/eval-files.mdx:410 still says all eval strings support ${{ VAR }}, and the example at lines 416-422 plus behavior bullets at 427-429 still use ${{ ... }}. This conflicts with the new behavior and the new unit test that leaves legacy ${{ HOME }} unchanged. Users copying this section will get literal ${{ ... }} text instead of env interpolation. Update that whole Environment Variable Interpolation block to {{ env.VAR }} / {{ env.VAR | default("x") }} and adjust the inline partial-interpolation example accordingly.

[christso]

Verified c654e971 against the independent review blockers:

• Secret target fields now only accept a single whole {{ env.VAR }} env template for the Nunjucks env path; inline/prefixed output, adjacent env refs, and default-filter literal fallbacks are covered by regression tests.
• Public eval-file docs now teach {{ env.VAR }} syntax and no longer use the removed ${{ VAR }} syntax in the eval-file Environment Variable Interpolation section.

Focused local validation passed:

• bun test packages/core/test/evaluation/providers/targets.test.ts
• bun run --cwd packages/core typecheck
• bun run --cwd packages/core lint
• git diff --check

GitHub CI for head c654e9713002a4e27a20b204d3939bc2929c143b completed successfully.