diff --git a/examples/optimization/eval_optimize_loop/DESIGN.md b/examples/optimization/eval_optimize_loop/DESIGN.md new file mode 100644 index 00000000..6a1fb2bb --- /dev/null +++ b/examples/optimization/eval_optimize_loop/DESIGN.md @@ -0,0 +1,31 @@ +# 方案设计说明(Eval-Optimize Loop) + +**失败归因**:对每条失败 case 依据框架 metric 结果做规则归因,聚成六类——工具调用 +错误、工具参数错误、知识召回不足、格式不符合要求、LLM rubric 不达标、最终回复 +不匹配。轨迹失败先比调用名字多重集:名字不同判调用错误,名字同而参数异判参数 +错误,漏调知识工具时并报召回不足;期望回答可解析为 JSON 而实际不能则判格式违规。 +每条归因附证据与中文解释,规则未覆盖的失败按 metric 兜底映射,保证每个失败 case +至少有一条可解释理由;根因按「轨迹在上游」的优先级选取。 + +**接受策略**:候选须连过六道可配置闸门——验证集通过率与平均分双阈值提升、不得 +新增 hard fail、保护 case 不得退化、过拟合守卫、成本预算、时长预算,全部通过才 +接受;拒绝理由按严重度点名最关键的闸门,并列出同时未过的其它闸门。 + +**防过拟合**:优化器只见弱指标(黑盒模式禁用轨迹与召回)与自己那份调参集; +pipeline 一律用独立验证集加完整验收套件复评,出现「训练集提升且验证集退化」即判 +过拟合并拒绝,保护 case 与新增失败两道闸门再兜底。示例内置「泄漏调参集」场景: +优化器视角一路变好,独立复评当场揭穿,守卫必将其拒绝。 + +**产物审计**:每轮候选 prompt、接受理由、成本、耗时、种子与配置快照由优化器 +落盘 rounds/ 等目录;pipeline 另存基线与候选的逐 case 记录、归因明细、闸门配置 +快照,报告以 JSON 与 Markdown 双格式输出,整条链路离线确定可复现、可追溯审计。 + +--- + +**English abstract** — The loop evaluates baseline on train+val with the full metric +suite, clusters failures into six explainable types, runs GEPA-based prompt +optimization, then re-evaluates the candidate on an independent validation set with +per-case deltas. Six configurable gates (improvement, no new hard fails, protected +cases, overfit guard, cost and duration budgets) must all pass before acceptance, +and every round leaves reproducible audit artifacts (candidates, reasons, cost, +seed, config snapshots) in JSON and Markdown reports. diff --git a/examples/optimization/eval_optimize_loop/README.en.md b/examples/optimization/eval_optimize_loop/README.en.md new file mode 100644 index 00000000..86e29a15 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/README.en.md @@ -0,0 +1,243 @@ +# Eval-Optimize Loop — Automated Evaluate · Attribute · Optimize · Regress · Audit + +[中文版](README.md) | Design note: [DESIGN.md](DESIGN.md) + +> **Runs with zero API keys**: `python run_pipeline.py --scenario all` — all three +> scenarios finish end-to-end in under a minute. + +## 1 · Problem & Design Goals + +`AgentOptimizer` can produce prompts with higher scores, but "higher score" does not +mean "safe to ship": + +- the optimizer may only see **weak metrics** (black-box mode has no tool trajectory / + knowledge-recall signal); +- if the optimizer's tuning set leaks from the training distribution it will + **overfit** without noticing; +- without per-case comparison you cannot tell whether the gain was paid for by + breaking previously-passing key cases; +- without audit artifacts, an improved prompt still cannot pass a production review. + +This example wires `AgentEvaluator` and `AgentOptimizer` into a **reproducible +six-stage closed loop** that answers one question: **is this candidate prompt worth +accepting?** + +``` +① baseline eval ② failure attribution ③ optimization +train+val × 4 metrics → clustering into 6 → AgentOptimizer (GEPA) +(per-case score/traj) failure types (2 TargetPrompt fields) + │ +⑥ audit artifacts ⑤ acceptance gates ④ candidate validation +report json+md ← 6 configurable gates ← independent re-eval +per-round cand/cost (all must pass) per-case delta +``` + +### The minimal demo + +A "City Info Assistant": distance conversion (`convert_distance` tool), city +introductions (`knowledge_search` tool + source citation), identity questions (no +tool). The baseline prompt has three defects (unnormalized unit, no knowledge +retrieval, non-JSON output); 6 committed cases (3 train + 3 val) expose all of them. +Three built-in scenarios cover the three canonical outcomes: + +| Scenario | Candidate proposed by the optimizer | Independent re-eval | Gate decision | +| --- | --- | --- | --- | +| `success` | fixes all three defects | train 1/3→3/3, val 1/3→3/3 | ✅ ACCEPT | +| `no_effect` | copy-editing only (directives unchanged) | all unchanged | ❌ REJECT (insufficient improvement) | +| `overfit` | memorizes training samples | train 1/3→3/3, **val 1/3→0/3** | ❌ REJECT (overfit guard) | + +## 2 · Terminology + +| Term | Meaning | +| --- | --- | +| Acceptance suite | `data/eval_config.json`: trajectory + exact response + rubric + knowledge recall (4 metrics), used for regression eval | +| Optimizer's weak metric | `optimizer.json` has only `final_response_avg_score`: the SDK forbids trajectory/recall metrics in black-box `call_agent` mode — this information gap is part of the overfitting story | +| Leaked tuning set | `data/optimizer_probe.evalset.json`: same distribution as train; fed to the optimizer as its "validation set" only in the overfit scenario | +| Protected cases | `protected_cases` in `pipeline.json`: whitelist of key cases; any regression rejects the candidate | +| Directive DSL | the `` comment block in prompts; the fake agent parses it, so prompt edits change behavior offline for real | +| Trace mode | `evalMode: "trace"` in the evalset: evaluate/attribute pre-recorded traces without running the agent | + +## 3 · Running + +### 3.1 Zero-dependency run (no env vars / API keys) + +```bash +# default scenario: success +python examples/optimization/eval_optimize_loop/run_pipeline.py + +# all three scenarios (recommended first run) +python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all + +# additionally evaluate/attribute the pre-recorded baseline traces (trace mode) +python examples/optimization/eval_optimize_loop/run_pipeline.py --baseline-from-trace + +# write the best candidate back to loop_agent/prompts/ when gates pass (mutates sources! +# with --scenario all the write is deferred until every scenario has finished) +python examples/optimization/eval_optimize_loop/run_pipeline.py --apply + +# validate a report against the schema contract +python examples/optimization/eval_optimize_loop/run_pipeline.py --check sample_output/success/optimization_report.json +``` + +Tests: + +```bash +python -m pytest examples/optimization/eval_optimize_loop/tests -q +``` + +### 3.2 Output layout + +``` +runs/-/ +├── optimization_report.json # structured report: baseline / candidate / delta / attribution / gate decision +├── optimization_report.md # human-readable verdict with all the evidence +├── baseline_eval.json # stage ① raw per-case records +├── candidate_eval.json # stage ④ raw per-case records +├── attribution.json # stage ② findings +├── pipeline_config.snapshot.json # gate/seed config snapshot of this run +└── optimize/ # stage ③ native SDK audit directory + ├── result.json summary.txt run.log config.snapshot.json + ├── rounds/round_001.json … # per-round candidate prompts, acceptance reason, cost, duration + └── baseline_prompts/ best_prompts/ +``` + +The committed `sample_output/` holds the three reports from `--scenario all`; to +regenerate: + +```bash +python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all --output /tmp/regen +# then copy /tmp/regen/-*/optimization_report.{json,md} into sample_output// +``` + +## 4 · Inputs / Outputs + +| File | Role | +| --- | --- | +| `data/train.evalset.json` | 3 training cases (reflection minibatch source) | +| `data/val.evalset.json` | 3 independent validation cases (final referee for regression) | +| `data/optimizer_probe.evalset.json` | 3 leaked tuning cases (fed to the optimizer only in `overfit`) | +| `data/trace_baseline.evalset.json` | 2 trace-mode cases (pre-recorded baseline failures) | +| `data/eval_config.json` | acceptance metric suite (4 metrics, fake judge) | +| `optimizer.json` / `configs/optimizer.*.json` | optimizer configs (scenarios differ only in `reflection_lm.model_name`) | +| `pipeline.json` | gate thresholds / protected cases / budgets / seed | +| `loop_agent/prompts/system.md`, `skill.md` | the two TargetPrompt source files | +| `candidates/*.md` | the fake reflection LM's proposal library (scenario × field) | +| `sample_output/*/optimization_report.{json,md}` | sample reports for the three scenarios | + +## 5 · Gate rules (stage ⑤) + +All six gates must pass; the rejection reason cites the most severe failed gate +(overfit > protected case > new hard fail > insufficient improvement > budgets). +Every threshold is configurable in `pipeline.json`. + +| Gate | Rule | Config | +| --- | --- | --- | +| `min_val_improvement` | val pass-rate gain ≥ threshold AND mean-score gain ≥ threshold | `min_val_pass_rate_improvement` / `min_val_score_improvement` | +| `no_new_hard_fail` | no case may flip pass→fail | `forbid_new_hard_fail` | +| `protected_cases` | any new_fail / score_down on a protected case rejects | `protected_cases` | +| `overfit_guard` | train pass-rate ↑ AND val pass-rate ↓ ⇒ overfitting | `overfit_guard` | +| `cost_budget` | optimization cost ≤ budget; metric calls ≤ budget (optional) | `max_cost_usd` / `max_metric_calls` | +| `duration_budget` | pipeline wall-clock ≤ budget | `max_duration_seconds` | + +The 12-row decision matrix lives in `tests/test_gates.py::DECISION_MATRIX` (12/12 pass). + +## 6 · Failure-attribution rules (stage ②) + +The rules depend only on the structure of framework metric results, not on this +example's cases (they generalize to hidden samples); every failing case is +**guaranteed at least one** evidence-backed reason (a per-metric fallback mapping +covers anything the rules miss). + +| Failure type | Trigger | +| --- | --- | +| `wrong_tool_call` | trajectory metric failed and the actual/expected call **name multisets** differ (missing/extra/wrong tool) | +| `wrong_tool_args` | trajectory metric failed with equal names but different arguments | +| `knowledge_recall_miss` | recall rubric failed; or the missing call is a knowledge tool (reported alongside the trajectory finding) | +| `format_violation` | response metric failed and the expected text parses as JSON while the actual does not | +| `llm_rubric_fail` | response-quality rubric failed (evidence = failing rubric ids + reasons) | +| `final_answer_mismatch` | any other response-metric failure | + +Primary (root-cause) precedence: `wrong_tool_call` > `wrong_tool_args` > +`knowledge_recall_miss` > `format_violation` > `llm_rubric_fail` > +`final_answer_mismatch` (trajectory errors sit upstream in the chain). + +## 7 · Design notes + +### 7.1 Optimizer metrics ≠ acceptance metrics (a deliberate information gap) + +Black-box `call_agent` mode cannot capture tool trajectories or tool responses, and +the SDK **hard-rejects** `tool_trajectory_avg_score` / `llm_rubric_knowledge_recall` +in that mode. So `optimizer.json` carries only exact response matching — exactly the +situation in real businesses where the optimizer sees a weaker signal than the +acceptance suite. That is why the loop exists: **the optimizer's claim of improvement +only counts after re-evaluation with the full suite on an independent validation +set.** The overfit scenario pushes this gap to the extreme (optimizer view 0/3→3/3, +independent re-eval val 1/3→0/3). + +### 7.2 How the three fake models run with zero API keys (no SDK changes) + +Judge and reflection-LM configs both support `provider_name`; any non-openai provider +routes through `ModelRegistry.create_model("{provider}/{model}")` regex matching. The +example registers three deterministic fake providers: + +- **fake agent** parses the directive DSL in the prompt and changes behavior + accordingly — prompt edits produce real behavior/score differences offline; +- **fake judge** evaluates rubric text via backtick tokens; its conditional rule + ("如果…" + condition-not-applicable ⇒ yes) mirrors the real judge prompt, and its + JSON output is isomorphic to the real judge's, parsed by the SDK's own scorer; +- **fake reflection** finds the `` marker in the reflection + request and returns `candidates/X..md`, the scenario taken from its own + `model_name`. + +### 7.3 Why the packages are named `loop_agent` / `loop_pipeline` + +pytest may import several examples in one process: `agent` is used by most examples +and `pipeline` is already taken by `multi_agent_pipeline`; duplicate top-level names +clobber each other in `sys.modules`. + +### 7.4 Semantics of `--apply` + +The optimizer itself always runs with `update_source=False` (sources are restored +when it finishes). Writing the best candidate back to `loop_agent/prompts/` requires +**both** an accepting gate decision **and** the explicit `--apply` flag, so a +rejected candidate can never reach the source files, and even accepted ones land in +the audit directory by default. + +## 8 · Adapting to your own business + +1. **Swap the agent**: replace `loop_agent/`, keeping the two entry points — + `get_agent_async()` (evaluator agent_module mode, captures tool trajectories) and + `call_agent(query) -> str` (optimizer black-box callback). Both must re-read the + prompt sources on every call. +2. **Swap the data**: point `data/train.evalset.json` / `data/val.evalset.json` at + your business cases. **The validation set must be independent of training** (the + SDK guards against same-file leakage; same-distribution-different-file leakage is + what this loop's overfit guard is for). +3. **Swap the acceptance suite**: configure a real `judge_model` in + `data/eval_config.json` (drop `provider_name: fake-judge`, set + `model_name`/`api_key`/`base_url`). +4. **Swap the optimizer config**: same for `reflection_lm` in `optimizer.json`; + black-box mode allows response-based metrics only. +5. **Tune the gates**: adjust `pipeline.json` to your risk profile — add key + regression cases to `protected_cases`, set `max_cost_usd` from real model pricing. + +## 9 · FAQ + +**Q: Why does the `no_effect` scenario report `SUCCEEDED` yet get rejected?** +`OptimizeResult.status=SUCCEEDED` only means the loop terminated normally +(`finish_reason=no_improvement`). Acceptance is the pipeline gates' job — the two +verdict layers are deliberately separate. + +**Q: Why does the overfit guard require "train↑ AND val↓" instead of just val↓?** +A val drop alone only says the candidate is bad; a simultaneous train rise is the +fingerprint of overfitting, letting the report give an actionable diagnosis ("your +tuning set leaks from training") instead of a generic "it got worse". + +**Q: When is trace mode useful?** +When you already have production trace logs and want attribution before deciding to +run optimization: `--baseline-from-trace` evaluates and attributes +`data/trace_baseline.evalset.json` (`evalMode: "trace"`) with zero agent execution. + +**Q: Why is the reported cost 0?** +Fake models incur no token cost. With real models, `OptimizeResult.total_llm_cost` / +`total_token_usage` flow into the report and the `cost_budget` gate automatically. diff --git a/examples/optimization/eval_optimize_loop/README.md b/examples/optimization/eval_optimize_loop/README.md new file mode 100644 index 00000000..776ded0e --- /dev/null +++ b/examples/optimization/eval_optimize_loop/README.md @@ -0,0 +1,218 @@ +# Eval-Optimize Loop — 评测 · 归因 · 优化 · 回归 · 审计 自动闭环 + +[English version](README.en.md) | 方案设计说明见 [DESIGN.md](DESIGN.md) + +> **零 API Key 可跑**:`python run_pipeline.py --scenario all`,三个场景全流程 < 1 分钟。 + +## 1 · 适用问题与设计目标 + +`AgentOptimizer` 能自动改出分数更高的 prompt,但「分数变高」不等于「值得上线」: + +- 优化器看到的可能是**弱指标**(黑盒模式只有响应匹配,没有工具轨迹/知识召回); +- 优化器的调参集如果与训练集同源,它会**过拟合**而毫无察觉; +- 没有逐 case 对比,你不知道提升是不是靠「牺牲原本通过的关键 case」换来的; +- 没有审计产物,改出来的 prompt 即使分数变高也难以进入生产评审。 + +本 example 把 `AgentEvaluator` 与 `AgentOptimizer` 拼成一条**可复现的六阶段闭环**, +回答唯一的问题:**这个候选 prompt 到底值不值得接受?** + +``` +① baseline 评测 ② 失败归因 ③ 优化执行 +train+val × 4 metric → 6 类失败类型聚类 → AgentOptimizer(GEPA) +(逐 case 分数/轨迹) (每案给可读理由) (优化 2 个 TargetPrompt 字段) + │ +⑥ 审计落盘 ⑤ 接受策略 ④ 候选验证 +报告 json+md ← 六道可配置闸门 ← 独立 train/val 复评 +每轮候选/成本/seed (全过才接受) 逐 case delta 对比 +``` + +### 本 example 演示的最小用例 + +「城市信息助手」:距离换算(`convert_distance` 工具)、城市介绍(`knowledge_search` +工具 + 来源标注)、身份询问(无工具)。baseline prompt 有三处缺陷(单位不归一化、 +不检索知识库、不按 JSON 输出),6 条评测 case(3 train + 3 val)刚好暴露全部缺陷。 +三个内置场景对应三种典型结局: + +| 场景 | 优化器提出的候选 | 独立验证集复评 | gate 决策 | +| --- | --- | --- | --- | +| `success` | 修复全部三处缺陷 | train 1/3→3/3,val 1/3→3/3 | ✅ ACCEPT | +| `no_effect` | 只有文案润色(指令不变) | 全部 unchanged | ❌ REJECT(提升不足) | +| `overfit` | 死记硬背训练样本 | train 1/3→3/3,**val 1/3→0/3** | ❌ REJECT(过拟合守卫) | + +## 2 · 术语对照 + +| 术语 | 含义 | +| --- | --- | +| 验收套件 | `data/eval_config.json`:轨迹 + 精确响应 + rubric + 知识召回 4 个 metric,回归评测用 | +| 优化器弱指标 | `optimizer.json` 里只有 `final_response_avg_score`:黑盒 `call_agent` 模式下 SDK 禁用轨迹/召回 metric,这个信息差正是过拟合演示的一环 | +| 泄漏调参集 | `data/optimizer_probe.evalset.json`:与训练集同分布,仅 overfit 场景喂给优化器当"验证集",演示其危害 | +| 保护 case | `pipeline.json` 的 `protected_cases`:关键 case 白名单,任何退化直接拒绝 | +| 指令 DSL | prompt 里的 `` 注释块,fake agent 据此改变行为,让"改 prompt"离线也有真实行为差异 | +| trace 模式 | evalset 里的 `evalMode: "trace"`:用预录轨迹评测归因,不执行 agent | + +## 3 · 运行示例 + +### 3.1 零依赖运行(无需任何环境变量 / API Key) + +```bash +# 默认 success 场景 +python examples/optimization/eval_optimize_loop/run_pipeline.py + +# 三场景全跑(推荐第一次看) +python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all + +# baseline 额外用预录轨迹(trace 模式)评测归因,不执行 agent +python examples/optimization/eval_optimize_loop/run_pipeline.py --baseline-from-trace + +# gate 通过时把最优候选写回 loop_agent/prompts/(会改动源文件,谨慎; +# --scenario all 时等全部场景跑完后统一写回,不污染后续场景 baseline) +python examples/optimization/eval_optimize_loop/run_pipeline.py --apply + +# 校验报告字段契约 +python examples/optimization/eval_optimize_loop/run_pipeline.py --check sample_output/success/optimization_report.json +``` + +运行测试: + +```bash +python -m pytest examples/optimization/eval_optimize_loop/tests -q +``` + +### 3.2 产物结构 + +``` +runs/<场景>-<时间戳>/ +├── optimization_report.json # 结构化报告:baseline / candidate / delta / 归因 / gate 决策 +├── optimization_report.md # 人话版:是否值得接受 + 全部依据 +├── baseline_eval.json # 阶段① 逐 case 原始记录 +├── candidate_eval.json # 阶段④ 逐 case 原始记录 +├── attribution.json # 阶段② 归因明细 +├── pipeline_config.snapshot.json # 本次运行的 gate/seed 配置快照 +└── optimize/ # 阶段③ SDK 原生审计目录 + ├── result.json summary.txt run.log config.snapshot.json + ├── rounds/round_001.json … # 每轮候选 prompt、接受理由、成本、耗时 + └── baseline_prompts/ best_prompts/ +``` + +提交在仓库里的 `sample_output/` 就是 `--scenario all` 的三份报告;重新生成: + +```bash +python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all --output /tmp/regen +# 然后把 /tmp/regen/<场景>-*/optimization_report.{json,md} 拷入 sample_output/<场景>/ +``` + +## 4 · 输入 / 输出文件清单 + +| 文件 | 角色 | +| --- | --- | +| `data/train.evalset.json` | 训练集 3 条(优化器反思 minibatch 来源) | +| `data/val.evalset.json` | 独立验证集 3 条(回归复评的最终裁判) | +| `data/optimizer_probe.evalset.json` | 泄漏调参集 3 条(仅 overfit 场景喂给优化器) | +| `data/trace_baseline.evalset.json` | trace 模式演示 2 条(预录 baseline 失败轨迹) | +| `data/eval_config.json` | 验收 metric 套件(4 metric,fake judge) | +| `optimizer.json` / `configs/optimizer.*.json` | 优化配置(三场景仅 `reflection_lm.model_name` 不同) | +| `pipeline.json` | 闸门阈值 / 保护 case / 预算 / seed | +| `loop_agent/prompts/system.md`、`skill.md` | 两个 TargetPrompt 源文件(system prompt + skill prompt) | +| `candidates/*.md` | fake 反思 LM 的候选提案库(场景 × 字段) | +| `sample_output/*/optimization_report.{json,md}` | 三场景示例报告 | + +## 5 · gate 决策规则表(阶段⑤) + +六道闸门全过才接受;拒绝理由按严重度取最关键闸门(过拟合 > 保护 case > 新增 +hard fail > 提升不足 > 预算)。全部阈值都在 `pipeline.json` 里可配。 + +| 闸门 | 规则 | 对应配置 | +| --- | --- | --- | +| `min_val_improvement` | 验证集通过率提升 ≥ 阈值 且 平均分提升 ≥ 阈值 | `min_val_pass_rate_improvement` / `min_val_score_improvement` | +| `no_new_hard_fail` | 不允许任何 case pass→fail | `forbid_new_hard_fail` | +| `protected_cases` | 保护 case 出现 new_fail / score_down 即拒绝 | `protected_cases` | +| `overfit_guard` | train 通过率↑ 且 val 通过率↓ → 判定过拟合 | `overfit_guard` | +| `cost_budget` | 优化成本 ≤ 预算;metric 调用数 ≤ 预算(可选) | `max_cost_usd` / `max_metric_calls` | +| `duration_budget` | pipeline 墙钟时长 ≤ 预算 | `max_duration_seconds` | + +决策矩阵的 12 组期望行为见 `tests/test_gates.py::DECISION_MATRIX`(12/12 通过)。 + +## 6 · 失败归因规则表(阶段②) + +规则只依赖框架 metric 结果的结构,不依赖本 example 的具体 case(隐藏样本同样适用); +每个失败 case **保证至少一条**带证据的中文理由(规则不覆盖时按 metric 兜底映射)。 + +| 失败类型 | 触发规则 | +| --- | --- | +| `wrong_tool_call` 工具调用错误 | 轨迹 metric 失败且实际/期望调用**名字多重集**不同(漏调/多调/调错) | +| `wrong_tool_args` 工具参数错误 | 轨迹 metric 失败且名字一致、参数不同 | +| `knowledge_recall_miss` 知识召回不足 | 召回 rubric 失败;或漏调的正是知识检索工具(与轨迹归因并报) | +| `format_violation` 格式不符合要求 | 响应 metric 失败且期望可解析为 JSON、实际不能 | +| `llm_rubric_fail` LLM rubric 不达标 | 回答质量 rubric 失败(证据 = 未通过的 rubric id + 理由) | +| `final_answer_mismatch` 最终回复不匹配 | 响应 metric 失败的其余情况 | + +主要归因(根因)优先级:`wrong_tool_call` > `wrong_tool_args` > `knowledge_recall_miss` +> `format_violation` > `llm_rubric_fail` > `final_answer_mismatch`(轨迹错误在链路上游)。 + +## 7 · 设计要点 + +### 7.1 为什么优化器指标 ≠ 验收指标(刻意的信息差) + +`AgentOptimizer` 的黑盒 `call_agent` 模式拿不到工具轨迹与工具返回,SDK 会**硬性拒绝** +在该模式下配置 `tool_trajectory_avg_score` / `llm_rubric_knowledge_recall`。所以 +`optimizer.json` 只有响应精确匹配 —— 这正是真实业务的常态:优化器看到的信号弱于 +验收套件。闭环的意义就在于此:**优化器说变好了,还要用完整验收套件在独立验证集上 +复评过才算数**。overfit 场景把这个信息差推到极端(优化器视角 0/3→3/3,独立复评 +val 1/3→0/3)。 + +### 7.2 三个 fake 模型如何做到零 API Key(不改一行 SDK) + +框架的 judge / reflection LM 配置都支持 `provider_name`;非 openai 的 provider 会走 +`ModelRegistry.create_model("{provider}/{model}")` 正则路由。本 example 注册三个 +fake provider(`fake-agent` / `fake-judge` / `fake-reflection`),全部是确定性规则实现: + +- **fake agent** 解析 prompt 里的指令 DSL(``)决定行为 —— + 于是「优化 prompt」在离线环境下也有真实的行为/分数差异; +- **fake judge** 按 rubric 文本里的反引号 token 判定,条件规则(「如果…」+ 条件不适用 + => yes)与真实裁判 prompt 对齐,输出 JSON 与真实裁判同构、走 SDK 原生解析器; +- **fake reflection** 依据 prompt 顶部的 `` 标记返回 + `candidates/X.<场景>.md`,场景取自自己的 `model_name`。 + +### 7.3 为什么包名是 `loop_agent` / `loop_pipeline` + +pytest 可能在同一进程 import 多个 example 的包:`agent` 被多数 example 占用、 +`pipeline` 已被 `multi_agent_pipeline` 占用,重名会在 `sys.modules` 里互相顶掉。 + +### 7.4 `--apply` 的语义 + +优化器自身始终 `update_source=False`(跑完源文件必还原);是否把最优候选写回 +`loop_agent/prompts/` 由 **gate 决策 + `--apply` 开关**共同决定。这保证:没过闸门的 +候选永远不可能落到源文件,过了闸门也默认只进审计目录,写回是显式动作。 + +## 8 · 接入自有业务改哪里 + +1. **换 agent**:替换 `loop_agent/`,保留两个入口 —— `get_agent_async()`(评测器 + agent_module 模式,能捕获工具轨迹)和 `call_agent(query)->str`(优化器黑盒回调)。 + 两者都必须每次调用重读 prompt 源文件。 +2. **换数据**:`data/train.evalset.json` / `data/val.evalset.json` 换成你的业务 case。 + **验证集必须独立于训练集**(SDK 有同文件泄漏守卫,同源不同文件才是要靠本闭环的 + overfit 守卫兜住的情况)。 +3. **换验收套件**:`data/eval_config.json` 的 `judge_model` 配真实模型 + (删掉 `provider_name: fake-judge`,配 `model_name`/`api_key`/`base_url`)。 +4. **换优化配置**:`optimizer.json` 的 `reflection_lm` 同理;黑盒模式只能配响应类 + metric。 +5. **调闸门**:`pipeline.json` 按业务风险改 —— 关键回归 case 加进 `protected_cases`, + 成本预算按真实模型定价设 `max_cost_usd`。 + +## 9 · 常见问题 + +**Q:为什么 no_effect 场景优化器报 `SUCCEEDED` 却被拒绝?** +`OptimizeResult.status=SUCCEEDED` 只表示优化循环正常结束(`finish_reason= +no_improvement`)。接受与否由 pipeline 的闸门决定 —— 这正是两层判定分离的意义。 + +**Q:过拟合守卫为什么用「train↑ 且 val↓」而不是单看 val?** +单看 val 下降只能说明候选不好;train 同时上升才是过拟合的指纹,报告据此给出 +「调参集与训练集同源」这类可操作的诊断,而不是笼统的「变差了」。 + +**Q:trace 模式适合什么场景?** +线上已有轨迹日志、想先归因再决定要不要跑优化时:`--baseline-from-trace` 对 +`data/trace_baseline.evalset.json`(`evalMode: "trace"`)直接评测归因,零 agent 执行。 + +**Q:报告里的成本为什么是 0?** +fake 模型不产生 token 费用。接真实模型后 `OptimizeResult.total_llm_cost` / +`total_token_usage` 会自动进入报告与 `cost_budget` 闸门。 diff --git a/examples/optimization/eval_optimize_loop/candidates/skill.no_effect.md b/examples/optimization/eval_optimize_loop/candidates/skill.no_effect.md new file mode 100644 index 00000000..9cc58932 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/skill.no_effect.md @@ -0,0 +1,6 @@ + + +# 回答方法 + +- 先分辨问题类型(换算 / 介绍 / 身份),再判断是否需要工具。 +- 回答尽量精炼,不要夹带无关内容。 diff --git a/examples/optimization/eval_optimize_loop/candidates/skill.overfit.md b/examples/optimization/eval_optimize_loop/candidates/skill.overfit.md new file mode 100644 index 00000000..74c0f825 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/skill.overfit.md @@ -0,0 +1,6 @@ + + +# 回答方法 + +- 先判断问题属于哪一类(换算 / 介绍 / 身份),再决定是否调用工具。 +- 优先回忆训练时见过的相似问题,回答保持简洁。 diff --git a/examples/optimization/eval_optimize_loop/candidates/skill.success.md b/examples/optimization/eval_optimize_loop/candidates/skill.success.md new file mode 100644 index 00000000..f8b4d22b --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/skill.success.md @@ -0,0 +1,6 @@ + + +# 回答方法 + +- 先识别问题类别(换算 / 介绍 / 身份),再决定是否调用工具。 +- 回答保持简洁、直接,不输出与问题无关的内容。 diff --git a/examples/optimization/eval_optimize_loop/candidates/system_prompt.no_effect.md b/examples/optimization/eval_optimize_loop/candidates/system_prompt.no_effect.md new file mode 100644 index 00000000..398f4835 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/system_prompt.no_effect.md @@ -0,0 +1,21 @@ + + + +# 角色定位 + +你是「城市信息助手 CityInfo」,你的职责是回答下面三类问题: + +1. **距离换算**:把公里换算成米(调用 `convert_distance` 工具)。 +2. **城市介绍**:介绍一个城市。 +3. **身份询问**:回答你自己的名字(我是城市信息助手 CityInfo)。 + +# 输出规范 + +- 换算结果用自然语言直接表述即可。 +- 调用换算工具时,单位保持用户的原始写法传入。 +- 城市介绍凭你自己的印象简要概括即可,无需检索资料。 diff --git a/examples/optimization/eval_optimize_loop/candidates/system_prompt.overfit.md b/examples/optimization/eval_optimize_loop/candidates/system_prompt.overfit.md new file mode 100644 index 00000000..5e242aba --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/system_prompt.overfit.md @@ -0,0 +1,20 @@ + + + +# 角色 + +你是「城市信息助手 CityInfo」。 + +# 应试技巧(这是一份刻意过拟合的候选 prompt) + +- 训练/调参集里出现过的问题,严格按照当时的标准轨迹与标准答案逐字复述。 +- 没有见过的问题,直接套用训练经验作答,不要调用工具。 + +(说明:这份候选让 agent 死记硬背训练样本 —— 在与训练集同分布的 +调参集上分数会一路上涨,但在独立验证集上必然退化。它专门用来演示 +闭环的过拟合守卫必须拒绝这类候选。) diff --git a/examples/optimization/eval_optimize_loop/candidates/system_prompt.success.md b/examples/optimization/eval_optimize_loop/candidates/system_prompt.success.md new file mode 100644 index 00000000..b8126a5c --- /dev/null +++ b/examples/optimization/eval_optimize_loop/candidates/system_prompt.success.md @@ -0,0 +1,22 @@ + + + +# 角色 + +你是「城市信息助手 CityInfo」,负责回答三类问题: + +1. **距离换算**:把公里换算成米(调用 `convert_distance` 工具)。 +2. **城市介绍**:介绍一个城市(先调用 `knowledge_search` 检索城市指南)。 +3. **身份询问**:回答你自己的名字(我是城市信息助手 CityInfo)。 + +# 输出要求 + +- 用户要求 JSON 输出时,换算结果输出规范 JSON:`{"result": <米数>, "unit": "m"}`。 +- 调用换算工具时,单位必须归一化为规范写法 `km` 再传入。 +- 城市介绍必须先调用 `knowledge_search` 检索,引用检索摘要作答, + 并在句末标注来源 `[source: city-guide]`。 diff --git a/examples/optimization/eval_optimize_loop/configs/optimizer.no_effect.json b/examples/optimization/eval_optimize_loop/configs/optimizer.no_effect.json new file mode 100644 index 00000000..6989c2a7 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/configs/optimizer.no_effect.json @@ -0,0 +1,46 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "provider_name": "fake-reflection", + "model_name": "no_effect", + "generation_config": { + "max_tokens": 2048, + "temperature": 0.0 + } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/configs/optimizer.overfit.json b/examples/optimization/eval_optimize_loop/configs/optimizer.overfit.json new file mode 100644 index 00000000..666e38f9 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/configs/optimizer.overfit.json @@ -0,0 +1,46 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": { + "match": "exact", + "case_insensitive": false + } + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "provider_name": "fake-reflection", + "model_name": "overfit", + "generation_config": { + "max_tokens": 2048, + "temperature": 0.0 + } + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/data/eval_config.json b/examples/optimization/eval_optimize_loop/data/eval_config.json new file mode 100644 index 00000000..b3019715 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/eval_config.json @@ -0,0 +1,79 @@ +{ + "metrics": [ + { + "metric_name": "tool_trajectory_avg_score", + "threshold": 1.0, + "criterion": { + "tool_trajectory": { + "default": { + "name": {"match": "exact"}, + "arguments": {"match": "exact"} + }, + "order_sensitive": false, + "subset_matching": false + } + } + }, + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": {"match": "exact", "case_insensitive": false} + } + } + }, + { + "metric_name": "llm_rubric_response", + "threshold": 1.0, + "criterion": { + "llm_judge": { + "judge_model": { + "provider_name": "fake-judge", + "model_name": "rule-judge", + "num_samples": 1, + "generation_config": {"max_tokens": 1024, "temperature": 0.0} + }, + "rubrics": [ + { + "id": "r_json", + "content": {"text": "如果用户要求 `JSON` 输出,回答必须包含 `\"result\"` 字段"}, + "description": "JSON 输出格式合规", + "type": "FINAL_RESPONSE_QUALITY" + }, + { + "id": "r_cite", + "content": {"text": "如果用户要求 `介绍` 城市,回答必须包含 `[source: city-guide]` 标注"}, + "description": "城市介绍需标注来源", + "type": "FINAL_RESPONSE_QUALITY" + } + ] + } + } + }, + { + "metric_name": "llm_rubric_knowledge_recall", + "threshold": 1.0, + "criterion": { + "llm_judge": { + "judge_model": { + "provider_name": "fake-judge", + "model_name": "rule-judge", + "num_samples": 1, + "generation_config": {"max_tokens": 1024, "temperature": 0.0} + }, + "rubrics": [ + { + "id": "k_guide", + "content": {"text": "如果用户要求 `介绍` 城市,检索结果必须包含 `city-guide` 来源"}, + "description": "城市介绍需先检索到 city-guide 语料", + "type": "KNOWLEDGE_COVERAGE" + } + ], + "knowledge_tool_names": ["knowledge_search"] + } + } + } + ], + "num_runs": 1 +} diff --git a/examples/optimization/eval_optimize_loop/data/optimizer_probe.evalset.json b/examples/optimization/eval_optimize_loop/data/optimizer_probe.evalset.json new file mode 100644 index 00000000..43fdc7ec --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/optimizer_probe.evalset.json @@ -0,0 +1,51 @@ +{ + "eval_set_id": "eval_optimize_loop_probe", + "name": "eval_optimize_loop - optimizer probe (泄漏调参集)", + "description": "仅 overfit 场景喂给优化器当『验证集』的调参集:3 条样本与训练集同分布/近重复(全部收录进 memorize 查表)。优化器视角 0/3→3/3 一路变好;pipeline 用独立 val.evalset.json 复评时才暴露过拟合。这是刻意的反面教材,演示『调参集与训练集同源』的危害。", + "eval_cases": [ + { + "eval_id": "probe_convert_4km", + "conversation": [ + { + "invocation_id": "p1", + "user_content": {"parts": [{"text": "把 4 公里换算成米,用 JSON 输出"}], "role": "user"}, + "final_response": {"parts": [{"text": "{\"result\": 4000, \"unit\": \"m\"}"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e1", "name": "convert_distance", "args": {"value": 4, "unit": "km"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "probe_intro_shenzhen", + "conversation": [ + { + "invocation_id": "p2", + "user_content": {"parts": [{"text": "请介绍一下深圳"}], "role": "user"}, + "final_response": {"parts": [{"text": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e2", "name": "knowledge_search", "args": {"query": "深圳"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "probe_identity", + "conversation": [ + { + "invocation_id": "p3", + "user_content": {"parts": [{"text": "请自报家门"}], "role": "user"}, + "final_response": {"parts": [{"text": "我是城市信息助手 CityInfo。"}], "role": "model"}, + "intermediate_data": {"tool_uses": []} + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + } + ] +} diff --git a/examples/optimization/eval_optimize_loop/data/trace_baseline.evalset.json b/examples/optimization/eval_optimize_loop/data/trace_baseline.evalset.json new file mode 100644 index 00000000..be8b7f13 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/trace_baseline.evalset.json @@ -0,0 +1,64 @@ +{ + "eval_set_id": "eval_optimize_loop_trace", + "name": "eval_optimize_loop - trace 模式 baseline 快照", + "description": "trace 模式演示(--baseline-from-trace):actual_conversation 是预录的 baseline 失败轨迹,conversation 是期望参考。评测与失败归因直接在录制轨迹上进行,不执行 agent —— 适合线上轨迹回放、无法重跑 agent 的场景。", + "eval_cases": [ + { + "eval_id": "trace_convert_3km", + "evalMode": "trace", + "actual_conversation": [ + { + "invocation_id": "a1", + "user_content": {"parts": [{"text": "把 3 公里换算成米,用 JSON 输出"}], "role": "user"}, + "final_response": {"parts": [{"text": "3 公里等于 3000 米"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "a1c1", "name": "convert_distance", "args": {"value": 3, "unit": "公里"}} + ], + "tool_responses": [ + {"id": "a1c1", "name": "convert_distance", "response": {"error": "unsupported unit: 公里,请使用规范单位 km"}} + ] + } + } + ], + "conversation": [ + { + "invocation_id": "e1", + "user_content": {"parts": [{"text": "把 3 公里换算成米,用 JSON 输出"}], "role": "user"}, + "final_response": {"parts": [{"text": "{\"result\": 3000, \"unit\": \"m\"}"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e1c1", "name": "convert_distance", "args": {"value": 3, "unit": "km"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "trace_intro_shenzhen", + "evalMode": "trace", + "actual_conversation": [ + { + "invocation_id": "a2", + "user_content": {"parts": [{"text": "介绍一下深圳"}], "role": "user"}, + "final_response": {"parts": [{"text": "深圳是一座很不错的城市。"}], "role": "model"}, + "intermediate_data": {"tool_uses": [], "tool_responses": []} + } + ], + "conversation": [ + { + "invocation_id": "e2", + "user_content": {"parts": [{"text": "介绍一下深圳"}], "role": "user"}, + "final_response": {"parts": [{"text": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e2c1", "name": "knowledge_search", "args": {"query": "深圳"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + } + ] +} diff --git a/examples/optimization/eval_optimize_loop/data/train.evalset.json b/examples/optimization/eval_optimize_loop/data/train.evalset.json new file mode 100644 index 00000000..0c782bbf --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/train.evalset.json @@ -0,0 +1,51 @@ +{ + "eval_set_id": "eval_optimize_loop_train", + "name": "eval_optimize_loop - train", + "description": "训练集(3 条):baseline 下 2 败 1 胜。失败 case 覆盖 wrong_tool_args / format_violation / wrong_tool_call / knowledge_recall_miss / llm_rubric_fail / final_answer_mismatch 六类归因信号。", + "eval_cases": [ + { + "eval_id": "train_convert_3km", + "conversation": [ + { + "invocation_id": "t1", + "user_content": {"parts": [{"text": "把 3 公里换算成米,用 JSON 输出"}], "role": "user"}, + "final_response": {"parts": [{"text": "{\"result\": 3000, \"unit\": \"m\"}"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e1", "name": "convert_distance", "args": {"value": 3, "unit": "km"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "train_intro_shenzhen", + "conversation": [ + { + "invocation_id": "t2", + "user_content": {"parts": [{"text": "介绍一下深圳"}], "role": "user"}, + "final_response": {"parts": [{"text": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e2", "name": "knowledge_search", "args": {"query": "深圳"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "train_identity", + "conversation": [ + { + "invocation_id": "t3", + "user_content": {"parts": [{"text": "你的名字是什么?"}], "role": "user"}, + "final_response": {"parts": [{"text": "我是城市信息助手 CityInfo。"}], "role": "model"}, + "intermediate_data": {"tool_uses": []} + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + } + ] +} diff --git a/examples/optimization/eval_optimize_loop/data/val.evalset.json b/examples/optimization/eval_optimize_loop/data/val.evalset.json new file mode 100644 index 00000000..3215caf2 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/data/val.evalset.json @@ -0,0 +1,51 @@ +{ + "eval_set_id": "eval_optimize_loop_val", + "name": "eval_optimize_loop - val", + "description": "验证集(3 条):与训练集同任务不同样本。val_identity 是 pipeline.json 里的 protected key case —— baseline 通过,overfit 候选下退化为失败。", + "eval_cases": [ + { + "eval_id": "val_convert_5km", + "conversation": [ + { + "invocation_id": "v1", + "user_content": {"parts": [{"text": "把 5 公里换算成米,用 JSON 输出"}], "role": "user"}, + "final_response": {"parts": [{"text": "{\"result\": 5000, \"unit\": \"m\"}"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e1", "name": "convert_distance", "args": {"value": 5, "unit": "km"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "val_identity", + "conversation": [ + { + "invocation_id": "v2", + "user_content": {"parts": [{"text": "你叫什么名字?"}], "role": "user"}, + "final_response": {"parts": [{"text": "我是城市信息助手 CityInfo。"}], "role": "model"}, + "intermediate_data": {"tool_uses": []} + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + }, + { + "eval_id": "val_intro_hangzhou", + "conversation": [ + { + "invocation_id": "v3", + "user_content": {"parts": [{"text": "介绍一下杭州"}], "role": "user"}, + "final_response": {"parts": [{"text": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]"}], "role": "model"}, + "intermediate_data": { + "tool_uses": [ + {"id": "e3", "name": "knowledge_search", "args": {"query": "杭州"}} + ] + } + } + ], + "session_input": {"app_name": "eval_optimize_loop_demo", "user_id": "demo", "state": {}} + } + ] +} diff --git a/examples/optimization/eval_optimize_loop/loop_agent/__init__.py b/examples/optimization/eval_optimize_loop/loop_agent/__init__.py new file mode 100644 index 00000000..1f4041a7 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_agent/__init__.py @@ -0,0 +1,108 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""「城市信息助手」agent 包 —— eval_optimize_loop 专用。 + +包名刻意叫 ``loop_agent`` 而不是其它 example 常用的 ``agent``:pytest 可能在 +同一进程里 import 多个 example 的包,重名会在 ``sys.modules`` 里互相顶掉。 + +对外暴露两个入口,分别服务闭环的两条链路: + +- :func:`get_agent_async` —— ``AgentEvaluator`` 的 ``agent_module`` 模式。 + 评测器每次运行都会重新调用它,而它每次都从磁盘重读 prompt 文件, + 所以 pipeline 把候选 prompt 写入源文件后**无需重启进程**即可生效。 + agent_module 模式下 LocalEvalService 能捕获工具轨迹与工具返回 → + 验收套件的 ``tool_trajectory_avg_score`` / ``llm_rubric_knowledge_recall`` + 才有数据可评。 +- :func:`call_agent` —— ``AgentOptimizer`` 的黑盒回调(query → 最终回答)。 + 黑盒模式拿不到工具轨迹,所以 optimizer.json 只配 ``final_response_avg_score`` + (SDK 会硬性拒绝在该模式下配置轨迹/召回类 metric)。 + +模型是 :class:`~loop_agent.fake_models.FakeAgentModel`(规则驱动、指令敏感、 +零 API Key),import 本包时三个 fake provider 已注册进 ModelRegistry。 +""" + +from __future__ import annotations + +import uuid +from pathlib import Path +from typing import Optional, Tuple + +from trpc_agent_sdk.agents import LlmAgent +from trpc_agent_sdk.runners import Runner +from trpc_agent_sdk.sessions import InMemorySessionService +from trpc_agent_sdk.tools import FunctionTool +from trpc_agent_sdk.types import Content, Part + +from .fake_models import FakeAgentModel, register_fake_models +from .tools import convert_distance, knowledge_search + +register_fake_models() + +APP_NAME = "eval_optimize_loop_demo" + +_PACKAGE_DIR = Path(__file__).resolve().parent +SYSTEM_PROMPT_PATH = _PACKAGE_DIR / "prompts" / "system.md" +SKILL_PATH = _PACKAGE_DIR / "prompts" / "skill.md" + + +def build_instruction() -> str: + """从磁盘拼合完整 instruction(system.md + skill.md)。 + + 每次调用都重读文件:优化器/回归器写入候选 prompt 后立即生效。 + """ + system = SYSTEM_PROMPT_PATH.read_text(encoding="utf-8").strip() + skill = SKILL_PATH.read_text(encoding="utf-8").strip() + return f"{system}\n\n{skill}" + + +def create_agent() -> LlmAgent: + """用当前磁盘 prompt 构建一个新的 LlmAgent 实例。""" + return LlmAgent( + name="city_info_agent", + description="城市信息助手:距离换算 / 城市介绍 / 身份询问。", + model=FakeAgentModel("fake-agent/city-info"), + instruction=build_instruction(), + tools=[FunctionTool(convert_distance), FunctionTool(knowledge_search)], + ) + + +async def get_agent_async() -> Tuple[LlmAgent, Optional[object]]: + """AgentEvaluator agent_module 约定入口:每次评测运行重建 agent。""" + return create_agent(), None + + +async def call_agent(query: str) -> str: + """AgentOptimizer 黑盒回调:驱动 agent 一次,返回最终回答文本。 + + 与 quickstart 相同的隔离要求:每次调用独立创建 Runner + + InMemorySessionService,避免并发评测时 session state 互相污染。 + """ + root_agent = create_agent() + session_service = InMemorySessionService() + runner = Runner(app_name=APP_NAME, agent=root_agent, session_service=session_service) + + session_id = str(uuid.uuid4()) + user_id = "optimizer" + await session_service.create_session( + app_name=APP_NAME, + user_id=user_id, + session_id=session_id, + state={}, + ) + user_content = Content(role="user", parts=[Part.from_text(text=query)]) + + final_text = "" + async for event in runner.run_async(user_id=user_id, session_id=session_id, new_message=user_content): + if not event.is_final_response(): + continue + if not event.content or not event.content.parts: + continue + for part in event.content.parts: + if part.thought: + continue + if part.text: + final_text += part.text + return final_text.strip() diff --git a/examples/optimization/eval_optimize_loop/loop_agent/fake_models.py b/examples/optimization/eval_optimize_loop/loop_agent/fake_models.py new file mode 100644 index 00000000..90fc9556 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_agent/fake_models.py @@ -0,0 +1,483 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""三个确定性 fake 模型:让整条「评测→优化→回归」闭环零 API Key 可复现。 + +设计动机 +-------- +本 example 的验收要求是「fake judge / fake model / trace mode 下完整 pipeline +≤ 3 分钟、无真实 API Key」。框架的 ``ModelRegistry`` 按模型名正则路由到 +``LLMModel`` 子类,而 judge / reflection LM 的配置都支持 ``provider_name`` +(非 openai 的 provider 会走 ``ModelRegistry.create_model("{provider}/{model}")``), +所以这里注册三个 fake provider,**不改任何 SDK 代码** 就能把整条链路换成 +规则驱动的确定性实现: + +- ``fake-agent/*`` → :class:`FakeAgentModel` 被评测的 agent 本体 +- ``fake-judge/*`` → :class:`FakeJudgeModel` LLM rubric 裁判 +- ``fake-reflection/*``→ :class:`FakeReflectionModel` GEPA 的反思 LM + +FakeAgentModel:指令敏感的规则 agent +------------------------------------ +真实场景里 prompt 改动会改变模型行为;离线演示要复现这一点,就让 fake +agent 从 system instruction(= system.md + skill.md 拼合文本)里解析一个 +「指令 DSL」HTML 注释块:: + + + +于是「优化 prompt」(改写 system.md)会真实地改变 agent 行为,评测分数 +随之变化 —— GEPA 的整个反思循环得以在离线环境下端到端运转。 +``memorize: train_table`` 是刻意设计的过拟合候选:查表复读训练/调参集的 +标准轨迹与答案,未命中的问题给出错误答案,从而制造「训练集提升、验证集 +退化」的必拒场景。 + +FakeJudgeModel:规则化 rubric 裁判 +---------------------------------- +只解析 **user 角色消息**(system instruction 里含格式说明文本,会造成误 +判),取最后一个 ```` 块,逐行按 ``id: text`` 拆出 rubric。 +rubric 文本用反引号 token 驱动判定(与真实裁判的「条件规则」对齐): + +- 文本含「如果」→ 第一个反引号 token 是条件关键词,先在 ```` + 里查条件;条件不适用 → verdict "yes"(对齐真实 judge prompt 的 + conditional-rubric 规则:not applicable => yes)。 +- 其余 token 必须**全部**出现在判定目标里:``llm_rubric_response`` 的目标 + 是 ````;``llm_rubric_knowledge_recall`` 的目标是 + ````(由消息里是否出现该块自动识别)。 + +输出与真实裁判完全同构:``{"items": [{"id","rubric","evidence","reason", +"verdict"}]}``,走 SDK 自带的 ``DefaultResponseScorer`` 解析。 + +FakeReflectionModel:查表式候选提案器 +------------------------------------- +gepa 的反思模板会把**当前 prompt 文本**嵌进反思请求(````), +且从回复的第一对与最后一对三反引号之间提取新 prompt。每个 prompt 文件 +顶部有 ```` 标记,反思模型据此识别是在改写 +哪个字段,然后返回 ``candidates/<字段>.<场景>.md`` 的内容(场景 = 自己 +model_name 里 ``/`` 之后的部分,如 ``fake-reflection/success`` → success)。 +找不到候选文件时原样返回当前文本(无害的 no-op 提案)。 + +注册与幂等 +---------- +``register_fake_models()`` 在本模块 import 时执行一次;探测用固定字符串 +(``ModelRegistry.resolve`` 带 lru_cache,固定串保证缓存命中一致)。 +三个类都有类级计数器(``calls``),供 pipeline 报告 fake 模型调用量; +自增走模块级锁(gepa 会在工作线程里并发调用,裸 ``+=`` 非原子)。 +""" + +from __future__ import annotations + +import json +import re +import threading +from pathlib import Path +from typing import Any, AsyncGenerator, List, Optional + +from trpc_agent_sdk.models import LLMModel, ModelRegistry +from trpc_agent_sdk.models._llm_request import LlmRequest +from trpc_agent_sdk.models._llm_response import LlmResponse +from trpc_agent_sdk.types import Content, FunctionCall, Part + +from .tools import CITY_CORPUS + +_EXAMPLE_ROOT = Path(__file__).resolve().parent.parent +CANDIDATES_DIR = _EXAMPLE_ROOT / "candidates" + +# 类级 calls 计数器的自增锁:gepa 在工作线程里并发调模型,``+=`` 是 +# 读-改-写三步、GIL 下也可能丢更新;计数只是报告用的信息性字段,但 +# 既然一把锁就能保证准确,就没有理由留下竞态。 +_CALLS_LOCK = threading.Lock() + + +def _bump_calls(model_cls: type) -> None: + """线程安全地把 ``model_cls.calls`` 加一。""" + with _CALLS_LOCK: + model_cls.calls += 1 + + +# --------------------------------------------------------------------------- +# 公共小工具 +# --------------------------------------------------------------------------- + + +def _text_response(text: str) -> LlmResponse: + """把纯文本包装成一条非流式 LlmResponse。""" + return LlmResponse(content=Content(role="model", parts=[Part.from_text(text=text)])) + + +def _last_user_text(request: LlmRequest) -> str: + """取最后一条 user 角色消息的纯文本(多 part 拼接)。""" + last = "" + for content in request.contents or []: + if content.role != "user" or not content.parts: + continue + text = "\n".join(p.text for p in content.parts if p.text) + if text.strip(): + last = text + return last + + +def _first_user_text(request: LlmRequest) -> str: + """取第一条带文本的 user 消息(评测会话里即用户原始问题)。""" + for content in request.contents or []: + if content.role != "user" or not content.parts: + continue + text = "\n".join(p.text for p in content.parts if p.text) + if text.strip(): + return text + return "" + + +def _has_function_response(request: LlmRequest) -> bool: + """判断会话里是否已有工具返回(即当前是「工具调用后」的第二跳)。""" + for content in request.contents or []: + for part in content.parts or []: + if part.function_response is not None: + return True + return False + + +def _last_function_response(request: LlmRequest) -> Optional[dict]: + """取最后一个 function_response 的 response dict;没有则 None。""" + result: Optional[dict] = None + for content in request.contents or []: + for part in content.parts or []: + if part.function_response is not None: + resp = part.function_response.response + if isinstance(resp, dict): + result = resp + return result + + +def _format_number(value: float) -> str: + """整数值不带小数点(3.0 → "3"),其余按原样。""" + if float(value).is_integer(): + return str(int(value)) + return str(value) + + +# --------------------------------------------------------------------------- +# FakeAgentModel +# --------------------------------------------------------------------------- + +# 过拟合查表:key = 训练集/调参集(probe)的原始问题;value = 标准轨迹 + 标准答案。 +# 覆盖 train.evalset.json 的 3 条与 optimizer_probe.evalset.json 的 3 条; +# 验证集(val)的问题刻意不入表 → memorize 候选在 val 上必然答错。 +MEMORIZE_TABLE: dict[str, dict[str, Any]] = { + "把 3 公里换算成米,用 JSON 输出": { + "tool_calls": [("convert_distance", { + "value": 3, + "unit": "km" + })], + "final": '{"result": 3000, "unit": "m"}', + }, + "介绍一下深圳": { + "tool_calls": [("knowledge_search", { + "query": "深圳" + })], + "final": f"{CITY_CORPUS['深圳']} [source: city-guide]", + }, + "你的名字是什么?": { + "tool_calls": [], + "final": "我是城市信息助手 CityInfo。", + }, + "把 4 公里换算成米,用 JSON 输出": { + "tool_calls": [("convert_distance", { + "value": 4, + "unit": "km" + })], + "final": '{"result": 4000, "unit": "m"}', + }, + "请介绍一下深圳": { + "tool_calls": [("knowledge_search", { + "query": "深圳" + })], + "final": f"{CITY_CORPUS['深圳']} [source: city-guide]", + }, + "请自报家门": { + "tool_calls": [], + "final": "我是城市信息助手 CityInfo。", + }, +} + +# memorize 候选对没见过的问题给出的错误答案(触发 val 退化) +MEMORIZE_MISS_ANSWER = "根据以往训练经验,答案与训练样本一致。" + +# 未匹配任何路由时的兜底回答(baseline 在 probe_identity 上因此失败) +FALLBACK_ANSWER = "抱歉,我暂时无法理解这个问题。" + +_DIRECTIVES_RE = re.compile(r"", re.DOTALL) +_KM_RE = re.compile(r"([0-9]+(?:\.[0-9]+)?)\s*公里") + +DEFAULT_DIRECTIVES = { + "output_format": "plain", + "unit_normalization": "off", + "knowledge": "off", + "memorize": "off", +} + + +def parse_directives(instruction: str) -> dict[str, str]: + """从 system instruction 里解析指令 DSL;缺失项用 baseline 默认值。""" + directives = dict(DEFAULT_DIRECTIVES) + match = _DIRECTIVES_RE.search(instruction or "") + if not match: + return directives + for line in match.group(1).splitlines(): + line = line.split("#", 1)[0].strip() # 去掉行内注释 + if ":" not in line: + continue + key, _, value = line.partition(":") + key, value = key.strip(), value.strip() + if key in directives and value: + directives[key] = value + return directives + + +class FakeAgentModel(LLMModel): + """指令敏感的规则 agent 模型(见模块 docstring 的路由表)。""" + + calls: int = 0 # 类级计数器:pipeline 报告 fake 模型调用量 + + @classmethod + def supported_models(cls) -> List[str]: + return [r"fake-agent/.*"] + + async def _generate_async_impl(self, + request: LlmRequest, + stream: bool = False, + ctx=None) -> AsyncGenerator[LlmResponse, None]: + _bump_calls(type(self)) + directives = parse_directives(str(request.config.system_instruction or "") if request.config else "") + query = _first_user_text(request) + after_tool = _has_function_response(request) + + # --- memorize=train_table:查表复读(过拟合候选) --- + if directives["memorize"] == "train_table": + yield self._memorized_response(query, after_tool) + return + + # --- 换算路由 --- + km_match = _KM_RE.search(query) + if "换算成米" in query and km_match: + value = float(km_match.group(1)) + if not after_tool: + unit = "km" if directives["unit_normalization"] == "on" else "公里" + args_value: Any = int(value) if value.is_integer() else value + yield self._tool_call_response("convert_distance", {"value": args_value, "unit": unit}) + return + meters = int(value * 1000) if (value * 1000).is_integer() else value * 1000 + if directives["output_format"] == "json": + yield _text_response(f'{{"result": {meters}, "unit": "m"}}') + else: + yield _text_response(f"{_format_number(value)} 公里等于 {meters} 米") + return + + # --- 城市介绍路由 --- + city = next((c for c in CITY_CORPUS if c in query), None) + if "介绍" in query and city is not None: + if directives["knowledge"] != "on": + yield _text_response(f"{city}是一座很不错的城市。") + return + if not after_tool: + yield self._tool_call_response("knowledge_search", {"query": city}) + return + tool_resp = _last_function_response(request) or {} + summary = str(tool_resp.get("summary") or "") + if summary: + yield _text_response(f"{summary} [source: city-guide]") + else: + yield _text_response(f"{city}的资料暂缺。") + return + + # --- 身份路由 --- + if "名字" in query: + yield _text_response("我是城市信息助手 CityInfo。") + return + + yield _text_response(FALLBACK_ANSWER) + + def _memorized_response(self, query: str, after_tool: bool) -> LlmResponse: + """memorize=train_table 分支:命中查表复读,未命中给错误答案。""" + entry = MEMORIZE_TABLE.get(query.strip()) + if entry is None: + return _text_response(MEMORIZE_MISS_ANSWER) + if entry["tool_calls"] and not after_tool: + parts = [ + Part(function_call=FunctionCall(id=f"memo-{i}", name=name, args=dict(args))) + for i, (name, args) in enumerate(entry["tool_calls"]) + ] + return LlmResponse(content=Content(role="model", parts=parts)) + return _text_response(entry["final"]) + + @staticmethod + def _tool_call_response(name: str, args: dict[str, Any]) -> LlmResponse: + return LlmResponse( + content=Content(role="model", parts=[Part(function_call=FunctionCall(id="call-1", name=name, args=args))])) + + +# --------------------------------------------------------------------------- +# FakeJudgeModel +# --------------------------------------------------------------------------- + +_RUBRIC_BLOCK_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL) +_MAIN_PROMPT_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL) +_FINAL_ANSWER_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL) +_KNOWLEDGE_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL) +_BACKTICK_TOKEN_RE = re.compile(r"`([^`]+)`") + + +class FakeJudgeModel(LLMModel): + """规则化 rubric 裁判(DSL 见模块 docstring)。""" + + calls: int = 0 + + @classmethod + def supported_models(cls) -> List[str]: + return [r"fake-judge/.*"] + + async def _generate_async_impl(self, + request: LlmRequest, + stream: bool = False, + ctx=None) -> AsyncGenerator[LlmResponse, None]: + _bump_calls(type(self)) + # 只看 user 消息:system instruction 里带格式示例文本,会干扰解析 + message = _last_user_text(request) + yield _text_response(self.judge_message(message)) + + @classmethod + def judge_message(cls, message: str) -> str: + """对一条裁判消息给出 items JSON(纯函数,便于单测)。""" + rubric_matches = _RUBRIC_BLOCK_RE.findall(message) + rubrics_block = rubric_matches[-1] if rubric_matches else "" + main_prompt = cls._last_group(_MAIN_PROMPT_RE, message) + knowledge_match = _KNOWLEDGE_RE.search(message) + if knowledge_match is not None: + target = knowledge_match.group(1).strip() + else: + target = cls._last_group(_FINAL_ANSWER_RE, message) + + items = [] + for line in rubrics_block.splitlines(): + line = line.strip() + if not line or ":" not in line: + continue + rubric_id, _, rubric_text = line.partition(":") + verdict, reason = cls._verdict(rubric_text.strip(), main_prompt, target) + items.append({ + "id": rubric_id.strip(), + "rubric": rubric_text.strip(), + "evidence": target[:80], + "reason": reason, + "verdict": verdict, + }) + return json.dumps({"items": items}, ensure_ascii=False) + + @staticmethod + def _verdict(rubric_text: str, main_prompt: str, target: str) -> tuple[str, str]: + """按反引号 token DSL 判定单条 rubric。""" + tokens = _BACKTICK_TOKEN_RE.findall(rubric_text) + if not tokens: + return "yes", "该 rubric 未定义判定 token,默认通过" + if "如果" in rubric_text: + condition, tokens = tokens[0], tokens[1:] + if condition not in main_prompt: + return "yes", f"条件「{condition}」不适用于该问题(not applicable => yes)" + missing = [t for t in tokens if t not in target] + if missing: + return "no", f"判定目标中缺少关键内容:{'、'.join(missing)}" + return "yes", "全部关键内容均在判定目标中出现" + + @staticmethod + def _last_group(pattern: re.Pattern, message: str) -> str: + matches = pattern.findall(message) + return matches[-1].strip() if matches else "" + + +# --------------------------------------------------------------------------- +# FakeReflectionModel +# --------------------------------------------------------------------------- + +_PROMPT_FIELD_RE = re.compile(r"") + + +class FakeReflectionModel(LLMModel): + """查表式候选提案器(GEPA 反思 LM 的离线替身)。""" + + calls: int = 0 + + @classmethod + def supported_models(cls) -> List[str]: + return [r"fake-reflection/.*"] + + @property + def scenario(self) -> str: + """model_name「fake-reflection/<场景>」里的场景名。""" + return self.name.rsplit("/", 1)[-1] + + async def _generate_async_impl(self, + request: LlmRequest, + stream: bool = False, + ctx=None) -> AsyncGenerator[LlmResponse, None]: + _bump_calls(type(self)) + prompt = _last_user_text(request) + yield _text_response(self.propose(prompt, self.scenario)) + + @staticmethod + def propose(prompt: str, scenario: str) -> str: + """给定反思请求文本,返回带三反引号包裹的候选 prompt。 + + gepa 从回复的第一对与最后一对 ``\\`\\`\\``` 之间提取候选文本, + 所以候选内容自身不能包含三反引号(candidates/ 目录已遵守)。 + """ + field_match = _PROMPT_FIELD_RE.search(prompt) + candidate_text: Optional[str] = None + if field_match is not None: + candidate_file = CANDIDATES_DIR / f"{field_match.group(1)}.{scenario}.md" + if candidate_file.is_file(): + candidate_text = candidate_file.read_text(encoding="utf-8").strip() + if candidate_text is None: + # 找不到字段/候选文件:原样返回当前 prompt(第一段 ``` 块),无害 no-op + current = FakeReflectionModel._extract_first_code_block(prompt) + candidate_text = current if current else prompt.strip() + return f"```\n{candidate_text}\n```" + + @staticmethod + def _extract_first_code_block(prompt: str) -> str: + start = prompt.find("```") + if start < 0: + return "" + end = prompt.find("```", start + 3) + if end < 0: + return "" + return prompt[start + 3:end].strip() + + +# --------------------------------------------------------------------------- +# 注册(幂等) +# --------------------------------------------------------------------------- + + +def register_fake_models() -> None: + """把三个 fake provider 注册进 ModelRegistry;重复调用无害。 + + 注意 ``ModelRegistry.resolve`` 带 lru_cache —— 探测必须用固定字符串, + 保证「已注册」判定与缓存条目一致。 + """ + for probe, model_cls in ( + ("fake-agent/probe", FakeAgentModel), + ("fake-judge/probe", FakeJudgeModel), + ("fake-reflection/probe", FakeReflectionModel), + ): + try: + ModelRegistry.resolve(probe) + except ValueError: + ModelRegistry.register(model_cls) + + +register_fake_models() diff --git a/examples/optimization/eval_optimize_loop/loop_agent/prompts/skill.md b/examples/optimization/eval_optimize_loop/loop_agent/prompts/skill.md new file mode 100644 index 00000000..0cd107e0 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_agent/prompts/skill.md @@ -0,0 +1,6 @@ + + +# 回答方法 + +- 先判断问题属于哪一类(换算 / 介绍 / 身份),再决定是否调用工具。 +- 回答保持简洁,不要输出与问题无关的内容。 diff --git a/examples/optimization/eval_optimize_loop/loop_agent/prompts/system.md b/examples/optimization/eval_optimize_loop/loop_agent/prompts/system.md new file mode 100644 index 00000000..f3ffe4b8 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_agent/prompts/system.md @@ -0,0 +1,21 @@ + + + +# 角色 + +你是「城市信息助手 CityInfo」,负责回答三类问题: + +1. **距离换算**:把公里换算成米(调用 `convert_distance` 工具)。 +2. **城市介绍**:介绍一个城市。 +3. **身份询问**:回答你自己的名字(我是城市信息助手 CityInfo)。 + +# 输出要求 + +- 换算结果用自然语言直接说出即可。 +- 调用换算工具时,单位按用户的原始写法传入。 +- 城市介绍凭你自己的印象简单概括,不必检索资料。 diff --git a/examples/optimization/eval_optimize_loop/loop_agent/tools.py b/examples/optimization/eval_optimize_loop/loop_agent/tools.py new file mode 100644 index 00000000..1de243c5 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_agent/tools.py @@ -0,0 +1,65 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""「城市信息助手」的两个业务工具 —— eval_optimize_loop 专用。 + +这两个工具是普通同步函数,由 ``FunctionTool`` 包装后注册到 LlmAgent: + +- ``convert_distance``:公里→米换算。只接受规范单位 ``"km"``;传入其它单位 + (例如 baseline prompt 会让 agent 传中文 ``"公里"``)时返回 error dict。 + 这个"单位必须归一化"的约束正是 baseline 的失败点之一(wrong_tool_args)。 +- ``knowledge_search``:内置小型城市知识库(来源标记 ``city-guide``)。 + ``llm_rubric_knowledge_recall`` 裁判会检查它的返回里是否带 ``city-guide`` + 来源,因此工具名必须与 eval 配置里的 ``knowledge_tool_names`` 完全一致。 + +两个工具都是纯函数、无外部依赖,保证离线可跑且逐次运行结果确定。 +""" + +from __future__ import annotations + +from typing import Any + +# 城市知识库:knowledge_search 的返回与 evalset 中期望回答共用这份文案, +# 保证「工具返回 → agent 引用 → exact 匹配」链路字符级一致。 +CITY_CORPUS: dict[str, str] = { + "深圳": "深圳是一座以科技创新闻名的现代化滨海城市。", + "杭州": "杭州是一座以西湖和数字经济闻名的历史文化名城。", + "北京": "北京是一座历史悠久的文化古都。", +} + + +def convert_distance(value: float, unit: str) -> dict[str, Any]: + """距离换算工具:把公里换算成米。 + + Args: + value: 数值(公里)。 + unit: 单位,必须是规范写法 "km";其它写法(如 "公里")视为不支持。 + + Returns: + 成功: {"meters": value * 1000};单位不规范: {"error": "..."}。 + """ + if unit == "km": + meters = value * 1000 + # 整数值转 int,避免 3000.0 之类的浮点尾巴影响可读性 + if float(meters).is_integer(): + meters = int(meters) + return {"meters": meters} + return {"error": f"unsupported unit: {unit},请使用规范单位 km"} + + +def knowledge_search(query: str) -> dict[str, Any]: + """知识检索工具:按城市名查询内置城市指南(来源 city-guide)。 + + Args: + query: 检索词(城市名)。 + + Returns: + 命中: {"source": "city-guide", "summary": "<一句话简介>"}; + 未命中: {"source": "none", "summary": ""}。 + """ + for city, summary in CITY_CORPUS.items(): + if city in query: + return {"source": "city-guide", "summary": summary} + return {"source": "none", "summary": ""} diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/__init__.py b/examples/optimization/eval_optimize_loop/loop_pipeline/__init__.py new file mode 100644 index 00000000..0f89e098 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/__init__.py @@ -0,0 +1,21 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""eval_optimize_loop 的六阶段闭环实现包。 + +包名叫 ``loop_pipeline`` 而不是 ``pipeline``: +``examples/optimization/multi_agent_pipeline`` 已经占用了顶层包名 +``pipeline``,同进程 import 两个 example 时会在 ``sys.modules`` 里撞名。 + +模块分工(与 issue 的六个阶段一一对应): + +- :mod:`.evaluate` 阶段① / ④ 共用的评测执行与逐 case 记录提取 +- :mod:`.attribution` 阶段② 失败归因(6 类失败类型聚类) +- :mod:`.optimize` 阶段③ AgentOptimizer 封装(场景 → 配置/数据集选择) +- :mod:`.regression` 阶段④ 候选换入/换出 + 逐 case delta 对比 +- :mod:`.gates` 阶段⑤ 可配置接受策略(六道闸门) +- :mod:`.report` 阶段⑥ optimization_report.json / .md 渲染与校验 +- :mod:`.config` pipeline.json(闸门阈值 / seed)的读取 +""" diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/attribution.py b/examples/optimization/eval_optimize_loop/loop_pipeline/attribution.py new file mode 100644 index 00000000..a3607afc --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/attribution.py @@ -0,0 +1,282 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段②:失败归因 —— 把失败 case 聚类成六种失败类型。 + +六类失败类型(issue 需求 2 原文对应): + +===================== ============================== +final_answer_mismatch 最终回复不匹配 +wrong_tool_call 工具调用错误(漏调/多调/调错工具) +wrong_tool_args 工具参数错误(工具对了、参数不对) +llm_rubric_fail LLM rubric 不达标 +knowledge_recall_miss 知识召回不足 +format_violation 格式不符合要求 +===================== ============================== + +归因规则是**通用的**(只依赖框架 metric 结果的结构,不依赖本 example 的 +具体 case),隐藏样本上同样适用: + +1. ``tool_trajectory_avg_score`` 失败 → 比较实际/期望调用的**名字多重集**: + 名字集合不同(漏调/多调/调错)→ ``wrong_tool_call``;名字一致但参数 + 不同 → ``wrong_tool_args``。若漏调的工具是知识检索工具(默认 + ``knowledge_search``)→ 追加一条 ``knowledge_recall_miss``。 +2. ``llm_rubric_knowledge_recall`` 失败 → ``knowledge_recall_miss`` + (证据 = 未通过的 rubric id 与理由)。 +3. ``final_response_avg_score`` 失败 → 若期望回答是结构化 JSON(对象/数组, + 裸标量如 ``42``/``true`` 不算)而实际回答不是 → ``format_violation`` + (要求结构化输出而给了自由文本,是最常见的格式违规);否则 + ``final_answer_mismatch``。 +4. ``llm_rubric_response`` 失败 → ``llm_rubric_fail``(证据 = 未通过的 + rubric id 与理由)。 + +主要归因(primary)按严重度优先级取第一个: +wrong_tool_call > wrong_tool_args > knowledge_recall_miss > +format_violation > llm_rubric_fail > final_answer_mismatch +(轨迹错误在链路上游、通常是根因,故优先级最高。) + +兜底保证「每个失败 case 至少一个可解释原因」:以上规则都没命中时,任何 +FAILED metric 都会映射成一条 finding(附 metric 失败理由)。 +""" + +from __future__ import annotations + +import json +from collections import Counter +from dataclasses import dataclass, field +from typing import Literal, Optional + +from .evaluate import CaseEvalRecord + +FailureType = Literal[ + "wrong_tool_call", + "wrong_tool_args", + "knowledge_recall_miss", + "format_violation", + "llm_rubric_fail", + "final_answer_mismatch", +] + +# 主要归因的优先级(越靠前越接近根因) +FAILURE_TYPE_PRECEDENCE: tuple[FailureType, ...] = ( + "wrong_tool_call", + "wrong_tool_args", + "knowledge_recall_miss", + "format_violation", + "llm_rubric_fail", + "final_answer_mismatch", +) + +FAILURE_TYPE_LABELS_ZH: dict[str, str] = { + "wrong_tool_call": "工具调用错误", + "wrong_tool_args": "工具参数错误", + "knowledge_recall_miss": "知识召回不足", + "format_violation": "格式不符合要求", + "llm_rubric_fail": "LLM rubric 不达标", + "final_answer_mismatch": "最终回复不匹配", +} + +# 知识检索类工具名(与 eval 配置的 knowledge_tool_names 保持一致) +DEFAULT_KNOWLEDGE_TOOLS = frozenset({"knowledge_search"}) + +_METRIC_FALLBACK_TYPE: dict[str, FailureType] = { + "tool_trajectory_avg_score": "wrong_tool_call", + "final_response_avg_score": "final_answer_mismatch", + "llm_rubric_response": "llm_rubric_fail", + "llm_rubric_knowledge_recall": "knowledge_recall_miss", +} + + +@dataclass +class FailureFinding: + """一条可解释的失败归因。""" + + type: FailureType + metric: str + evidence: str + explanation: str # 中文可读说明 + + +@dataclass +class AttributionSummary: + """一个切分(或全体)失败归因的聚类视图。""" + + counts: dict[str, int] = field(default_factory=dict) # 失败类型 → case 数(按出现的 case 去重计数) + per_case: dict[str, list[FailureFinding]] = field(default_factory=dict) + primary: dict[str, str] = field(default_factory=dict) # eval_id → 主要失败类型 + + +def _truncate(text: str, limit: int = 120) -> str: + text = (text or "").replace("\n", " ") + return text if len(text) <= limit else text[:limit] + "…" + + +def _is_structured_json(text: str) -> bool: + """期望「结构化输出」仅指 JSON 对象/数组;裸标量('42'、'true')是普通答案。""" + try: + return isinstance(json.loads(text), (dict, list)) + except (ValueError, TypeError): + return False + + +def _failed(record: CaseEvalRecord, metric: str) -> bool: + return record.metric_status.get(metric) == "FAILED" + + +def _failing_rubrics(record: CaseEvalRecord, metric: str) -> list[dict]: + return [r for r in record.rubric_verdicts.get(metric, []) if (r.get("score") or 0.0) < 1.0] + + +def _fmt_calls(calls: list[dict]) -> str: + if not calls: + return "(无调用)" + return "; ".join(f"{c['name']}({json.dumps(c['args'], ensure_ascii=False)})" for c in calls) + + +def attribute_case( + record: CaseEvalRecord, + knowledge_tools: frozenset[str] = DEFAULT_KNOWLEDGE_TOOLS, +) -> list[FailureFinding]: + """对一条失败 case 产出 ≥1 条归因;通过的 case 返回空列表。""" + if record.passed: + return [] + findings: list[FailureFinding] = [] + + # 规则 1:工具轨迹 + if _failed(record, "tool_trajectory_avg_score"): + actual_names = Counter(c["name"] for c in record.actual_tool_calls) + expected_names = Counter(c["name"] for c in record.expected_tool_calls) + evidence = f"期望 {_fmt_calls(record.expected_tool_calls)},实际 {_fmt_calls(record.actual_tool_calls)}" + if actual_names != expected_names: + missing = list((expected_names - actual_names).elements()) + extra = list((actual_names - expected_names).elements()) + detail_parts = [] + if missing: + detail_parts.append(f"缺少调用:{'、'.join(missing)}") + if extra: + detail_parts.append(f"多余调用:{'、'.join(extra)}") + findings.append( + FailureFinding( + type="wrong_tool_call", + metric="tool_trajectory_avg_score", + evidence=evidence, + explanation="工具调用集合与期望不一致(" + (";".join(detail_parts) or "调用了错误的工具") + ")", + )) + if any(name in knowledge_tools for name in missing): + findings.append( + FailureFinding( + type="knowledge_recall_miss", + metric="tool_trajectory_avg_score", + evidence=evidence, + explanation=f"缺少知识检索调用({'、'.join(n for n in missing if n in knowledge_tools)})," + "无法召回作答所需知识", + )) + else: + findings.append( + FailureFinding( + type="wrong_tool_args", + metric="tool_trajectory_avg_score", + evidence=evidence, + explanation="工具选择正确,但调用参数与期望不一致", + )) + + # 规则 2:知识召回 rubric + if _failed(record, "llm_rubric_knowledge_recall"): + failing = _failing_rubrics(record, "llm_rubric_knowledge_recall") + ids = "、".join(r.get("id", "?") for r in failing) or "(未提供 rubric 明细)" + reasons = ";".join(_truncate(r.get("reason", "")) for r in failing) + findings.append( + FailureFinding( + type="knowledge_recall_miss", + metric="llm_rubric_knowledge_recall", + evidence=f"未通过 rubric:{ids}。{reasons}", + explanation="知识召回不足:检索结果无法支撑作答所需的关键信息", + )) + + # 规则 3:最终回复精确匹配 + if _failed(record, "final_response_avg_score"): + evidence = (f"期望「{_truncate(record.expected_response)}」," + f"实际「{_truncate(record.actual_response)}」") + if _is_structured_json(record.expected_response) and not _is_structured_json(record.actual_response): + findings.append( + FailureFinding( + type="format_violation", + metric="final_response_avg_score", + evidence=evidence, + explanation="格式不符合要求:期望结构化 JSON 输出,实际是自由文本", + )) + else: + findings.append( + FailureFinding( + type="final_answer_mismatch", + metric="final_response_avg_score", + evidence=evidence, + explanation="最终回复与参考答案不匹配", + )) + + # 规则 4:回答质量 rubric + if _failed(record, "llm_rubric_response"): + failing = _failing_rubrics(record, "llm_rubric_response") + ids = "、".join(r.get("id", "?") for r in failing) or "(未提供 rubric 明细)" + reasons = ";".join(_truncate(r.get("reason", "")) for r in failing) + findings.append( + FailureFinding( + type="llm_rubric_fail", + metric="llm_rubric_response", + evidence=f"未通过 rubric:{ids}。{reasons}", + explanation="LLM rubric 评审不达标", + )) + + # 兜底:保证每个失败 case 至少一条可解释归因 + if not findings: + for metric, status in record.metric_status.items(): + if status != "FAILED": + continue + findings.append( + FailureFinding( + type=_METRIC_FALLBACK_TYPE.get(metric, "final_answer_mismatch"), + metric=metric, + evidence=_truncate(record.metric_reasons.get(metric) or "metric 评分未达阈值"), + explanation=f"metric {metric} 未达阈值", + )) + if not findings: # 理论上不可达:case FAILED 必有 FAILED metric + findings.append( + FailureFinding( + type="final_answer_mismatch", + metric="(unknown)", + evidence="case 标记为 FAILED 但无 metric 明细", + explanation="评测框架未提供 metric 明细,按最终回复不匹配处理", + )) + return findings + + +def primary_type(findings: list[FailureFinding]) -> Optional[str]: + """按优先级取主要失败类型。""" + present = {f.type for f in findings} + for failure_type in FAILURE_TYPE_PRECEDENCE: + if failure_type in present: + return failure_type + return None + + +def cluster( + records: dict[str, CaseEvalRecord], + knowledge_tools: frozenset[str] = DEFAULT_KNOWLEDGE_TOOLS, +) -> AttributionSummary: + """对一批 case 聚类归因;counts 按「出现该类型的 case 数」计。""" + summary = AttributionSummary() + type_counter: Counter[str] = Counter() + for eval_id in sorted(records): + findings = attribute_case(records[eval_id], knowledge_tools) + if not findings: + continue + summary.per_case[eval_id] = findings + primary = primary_type(findings) + if primary is not None: + summary.primary[eval_id] = primary + for failure_type in {f.type for f in findings}: + type_counter[failure_type] += 1 + summary.counts = {t: type_counter[t] for t in FAILURE_TYPE_PRECEDENCE if type_counter[t]} + return summary diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/config.py b/examples/optimization/eval_optimize_loop/loop_pipeline/config.py new file mode 100644 index 00000000..d6fab7e5 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/config.py @@ -0,0 +1,75 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""pipeline.json 配置模型:闸门阈值 / 复现实验参数。 + +所有闸门都可以在 ``pipeline.json`` 里按业务需要调整;默认值即本 example +演示三场景所用的取值。``PipelineConfig.load`` 从 JSON 文件读入并做 pydantic +校验,非法字段/类型会在 pipeline 启动前 fail-fast(``extra="forbid"``: +写错闸门名不会被静默忽略成默认阈值 —— 对安全闸门而言静默降级比报错危险)。 +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Optional + +from pydantic import BaseModel, ConfigDict, Field + + +class GateConfig(BaseModel): + """接受策略(阶段⑤)的六道闸门配置,全部可按业务调整。""" + + model_config = ConfigDict(extra="forbid") + + min_val_pass_rate_improvement: float = Field( + default=1e-9, + description="验证集通过率最小提升。默认要求「严格大于 0」;调大即要求显著提升。", + ) + min_val_score_improvement: float = Field( + default=0.0, + description="验证集平均 metric 分最小提升(第二信号,默认不允许下降)。", + ) + forbid_new_hard_fail: bool = Field( + default=True, + description="不允许出现 baseline 通过、candidate 失败的 case(新增 hard fail)。", + ) + protected_cases: list[str] = Field( + default_factory=lambda: ["val_identity"], + description="关键 case 白名单:任何一条出现 new_fail / score_down 即拒绝。", + ) + max_cost_usd: float = Field( + default=1.0, + description="优化过程 LLM 成本预算(对照 OptimizeResult.total_llm_cost)。", + ) + max_metric_calls: Optional[int] = Field( + default=None, + description="预算的第二形态:优化器 metric 调用数上限(对照 rounds[-1].budget_used)。", + ) + max_duration_seconds: float = Field( + default=180.0, + description="整条 pipeline 的墙钟时长预算(秒)。", + ) + overfit_guard: bool = Field( + default=True, + description="过拟合守卫:训练集通过率提升且验证集通过率下降 → 拒绝。", + ) + + +class PipelineConfig(BaseModel): + """pipeline.json 的顶层模型。""" + + model_config = ConfigDict(extra="forbid") + + gates: GateConfig = Field(default_factory=GateConfig) + seed: int = Field(default=42, description="记录进报告的随机种子;与 optimizer.json 的 algorithm.seed 保持一致。") + score_epsilon: float = Field(default=1e-6, description="逐 case 分数对比的浮点容差。") + + @classmethod + def load(cls, path: str | Path) -> "PipelineConfig": + """从 JSON 文件读入并校验。""" + data = json.loads(Path(path).read_text(encoding="utf-8")) + return cls.model_validate(data) diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/evaluate.py b/examples/optimization/eval_optimize_loop/loop_pipeline/evaluate.py new file mode 100644 index 00000000..f0d2213e --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/evaluate.py @@ -0,0 +1,181 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段① / ④:评测执行 + 逐 case 结构化记录(CaseEvalRecord)提取。 + +对 ``AgentEvaluator`` 的两点关键用法: + +1. ``get_executer(...)`` + ``await evaluate()``:任何 case 失败时框架会抛 + ``_EvaluationCasesFailed``(``AssertionError`` 子类,抛出前已填好结果)—— + 这里精确 ``except _EvaluationCasesFailed: pass`` 后照常 ``get_result()``, + 与 SDK 内部 ``_optimize_evaluator_call.run_evaluator`` 的姿势一致; + 其它 ``AssertionError``(SDK/三方库真实断言失败)照常抛出,不被吞掉。 +2. ``eval_metrics_file_path_or_dir=`` 显式指定共享 metric 配置文件, + 覆盖数据集目录的 ``test_config.json`` 约定 —— baseline 与候选回归 + 必须使用同一份验收 metric 套件,评分口径才可比。 + +``CaseEvalRecord`` 是后续归因(阶段②)、delta 对比(阶段④)、报告 +(阶段⑥)共用的最小充分信息集:metric 分与状态、失败理由、rubric 明细、 +实际/期望工具轨迹、实际/期望最终回答。 +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Optional + +from trpc_agent_sdk.evaluation import AgentEvaluator +from trpc_agent_sdk.evaluation._agent_evaluator import _EvaluationCasesFailed +from trpc_agent_sdk.evaluation._eval_case import get_all_tool_calls +from trpc_agent_sdk.evaluation._eval_metrics import EvalStatus + + +@dataclass +class CaseEvalRecord: + """一条 eval case 的一次评测结果(num_runs=1 取第 1 轮)。""" + + eval_id: str + passed: bool + final_status: str + case_score: float # 各 metric score 的均值(score=None 记 0) + metric_scores: dict[str, Optional[float]] = field(default_factory=dict) + metric_status: dict[str, str] = field(default_factory=dict) + metric_reasons: dict[str, Optional[str]] = field(default_factory=dict) + rubric_verdicts: dict[str, list[dict]] = field(default_factory=dict) # metric -> [{id, score, reason}] + actual_tool_calls: list[dict] = field(default_factory=list) # [{"name", "args"}] + expected_tool_calls: list[dict] = field(default_factory=list) + actual_response: str = "" + expected_response: str = "" + + +@dataclass +class SplitSummary: + """一个数据切分(train/val)的汇总视图。""" + + pass_rate: float + mean_case_score: float + metric_breakdown: dict[str, float] + total: int + passed: int + + +def _text_of(content: Any) -> str: + """Content.parts 里的纯文本拼接。""" + if content is None or not getattr(content, "parts", None): + return "" + return "\n".join((p.text or "") for p in content.parts if getattr(p, "text", None)).strip() + + +def _tool_calls_of(invocation: Any) -> list[dict]: + """Invocation.intermediate_data 里的工具调用列表 → [{"name","args"}]。""" + if invocation is None: + return [] + calls = get_all_tool_calls(getattr(invocation, "intermediate_data", None)) + return [{"name": c.name, "args": dict(c.args or {})} for c in calls] + + +def _record_from_case_result(case_result: Any) -> CaseEvalRecord: + """把框架的 EvalCaseResult 压平成 CaseEvalRecord。""" + metric_scores: dict[str, Optional[float]] = {} + metric_status: dict[str, str] = {} + metric_reasons: dict[str, Optional[str]] = {} + rubric_verdicts: dict[str, list[dict]] = {} + + for m in case_result.overall_eval_metric_results: + metric_scores[m.metric_name] = m.score + metric_status[m.metric_name] = m.eval_status.name + details = m.details + metric_reasons[m.metric_name] = details.reason if details is not None else None + if details is not None and details.rubric_scores: + rubric_verdicts[m.metric_name] = [{ + "id": getattr(r, "id", ""), + "score": getattr(r, "score", None), + "reason": getattr(r, "reason", ""), + } for r in details.rubric_scores] + + actual_tool_calls: list[dict] = [] + expected_tool_calls: list[dict] = [] + actual_response = "" + expected_response = "" + if case_result.eval_metric_result_per_invocation: + # 本 example 的 case 均为单 invocation;多轮对话取首轮即可满足归因需要 + per_inv = case_result.eval_metric_result_per_invocation[0] + actual_tool_calls = _tool_calls_of(per_inv.actual_invocation) + expected_tool_calls = _tool_calls_of(per_inv.expected_invocation) + actual_response = _text_of(getattr(per_inv.actual_invocation, "final_response", None)) + if per_inv.expected_invocation is not None: + expected_response = _text_of(getattr(per_inv.expected_invocation, "final_response", None)) + + scores = [(s if s is not None else 0.0) for s in metric_scores.values()] + return CaseEvalRecord( + eval_id=case_result.eval_id, + passed=case_result.final_eval_status == EvalStatus.PASSED, + final_status=case_result.final_eval_status.name, + case_score=(sum(scores) / len(scores)) if scores else 0.0, + metric_scores=metric_scores, + metric_status=metric_status, + metric_reasons=metric_reasons, + rubric_verdicts=rubric_verdicts, + actual_tool_calls=actual_tool_calls, + expected_tool_calls=expected_tool_calls, + actual_response=actual_response, + expected_response=expected_response, + ) + + +async def run_eval( + dataset_path: str, + eval_config_path: str, + *, + agent_module: Optional[str] = "loop_agent", +) -> dict[str, CaseEvalRecord]: + """跑一个数据集,返回 eval_id → CaseEvalRecord。 + + Args: + dataset_path: evalset JSON 路径。 + eval_config_path: 共享 metric 配置(验收套件)。 + agent_module: 被评 agent 的模块名;传 ``None`` 表示数据集是纯 + trace 模式(预录轨迹回放,不执行 agent)。 + """ + executer = AgentEvaluator.get_executer( + dataset_path, + agent_module=agent_module, + eval_metrics_file_path_or_dir=eval_config_path, + print_detailed_results=False, + print_summary_report=False, + ) + try: + await executer.evaluate() + except _EvaluationCasesFailed: + pass # 结果已填好,失败信息由报告呈现;其它 AssertionError 照常抛出 + result = executer.get_result() + if result is None: # pragma: no cover - evaluate() 非断言异常时才可能 + raise RuntimeError(f"evaluation produced no result for {dataset_path}") + + records: dict[str, CaseEvalRecord] = {} + for agg in result.results_by_eval_set_id.values(): + for eval_id, case_results in agg.eval_results_by_eval_id.items(): + records[eval_id] = _record_from_case_result(case_results[0]) + return records + + +def summarize(records: dict[str, CaseEvalRecord]) -> SplitSummary: + """聚合一个切分的通过率 / 平均 case 分 / 各 metric 平均分。""" + total = len(records) + passed = sum(1 for r in records.values() if r.passed) + metric_sums: dict[str, float] = {} + metric_counts: dict[str, int] = {} + for record in records.values(): + for name, score in record.metric_scores.items(): + metric_sums[name] = metric_sums.get(name, 0.0) + (score if score is not None else 0.0) + metric_counts[name] = metric_counts.get(name, 0) + 1 + return SplitSummary( + pass_rate=(passed / total) if total else 0.0, + mean_case_score=(sum(r.case_score for r in records.values()) / total) if total else 0.0, + metric_breakdown={name: metric_sums[name] / metric_counts[name] + for name in sorted(metric_sums)}, + total=total, + passed=passed, + ) diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/gates.py b/examples/optimization/eval_optimize_loop/loop_pipeline/gates.py new file mode 100644 index 00000000..3d128fba --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/gates.py @@ -0,0 +1,168 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段⑤:可配置接受策略 —— 六道闸门,全过才接受。 + +===================== ========================================================= +闸门 规则(对应 GateConfig 字段) +===================== ========================================================= +min_val_improvement 验证集通过率提升 ≥ min_val_pass_rate_improvement 且 + 平均分提升 ≥ min_val_score_improvement(双信号) +no_new_hard_fail 不允许任何 case 从 pass 变 fail(forbid_new_hard_fail) +protected_cases 保护 case 出现 new_fail / score_down 即拒绝 +overfit_guard 训练集通过率↑ 且 验证集通过率↓ → 判定过拟合,拒绝 +cost_budget 优化成本 ≤ max_cost_usd;若配置 max_metric_calls, + 优化器 metric 调用数也不得超出(缺少 budget_used + 追踪数据时 fail-closed,按未通过处理) +duration_budget pipeline 墙钟时长 ≤ max_duration_seconds +===================== ========================================================= + +决策 = 所有闸门 AND;``reason`` 按**严重度**取最关键的失败闸门的中文说明 +(过拟合 > 保护 case 退化 > 新增 hard fail > 提升不足 > 预算类),并注明 +共有几道闸门未通过 —— overfit 场景往往同时触发多门,报告应点出根因而不是 +恰好排在最前面的那一门。``optimize_result_view`` 用普通 dict 传入 +(total_llm_cost / budget_used / duration_seconds),单测可以直接喂合成值 +而不必构造完整的 OptimizeResult。 +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from .config import GateConfig +from .regression import DeltaSummary + + +@dataclass +class GateResult: + """单道闸门的判定。""" + + name: str + passed: bool + detail: str # 中文说明(含实际值 vs 阈值) + + +@dataclass +class GateDecision: + """最终接受/拒绝决策。""" + + accepted: bool + reason: str + gates: list[GateResult] = field(default_factory=list) + + +# reason 的严重度排序:越靠前越接近「候选本质有问题」,预算类殿后 +_REASON_SEVERITY = ( + "overfit_guard", + "protected_cases", + "no_new_hard_fail", + "min_val_improvement", + "cost_budget", + "duration_budget", +) + + +def _protected_violations(cfg: GateConfig, delta_val: DeltaSummary) -> list[str]: + violations = [] + protected = set(cfg.protected_cases) + for case in delta_val.per_case: + if case.eval_id in protected and case.change in ("new_fail", "score_down"): + violations.append(f"{case.eval_id}({case.change})") + return violations + + +def evaluate_gates( + cfg: GateConfig, + *, + delta_val: DeltaSummary, + delta_train: DeltaSummary, + optimize_result_view: dict[str, Any], + wall_seconds: float, +) -> GateDecision: + """按 GateConfig 逐门判定,返回整体决策与逐门明细。""" + gates: list[GateResult] = [] + + # 1. 验证集最小提升(通过率 + 平均分双信号) + pass_ok = delta_val.pass_rate_delta >= cfg.min_val_pass_rate_improvement + score_ok = delta_val.score_delta >= cfg.min_val_score_improvement + gates.append( + GateResult( + name="min_val_improvement", + passed=pass_ok and score_ok, + detail=(f"验证集通过率提升 {delta_val.pass_rate_delta:+.4f}" + f"(要求 ≥ {cfg.min_val_pass_rate_improvement:g})," + f"平均分提升 {delta_val.score_delta:+.4f}" + f"(要求 ≥ {cfg.min_val_score_improvement:g})" + ("" if pass_ok and score_ok else " —— 提升不足,不值得接受")), + )) + + # 2. 不允许新增 hard fail + new_fails = [c.eval_id for c in delta_val.per_case if c.change == "new_fail"] + new_fail_ok = (not cfg.forbid_new_hard_fail) or not new_fails + if not new_fails: + new_fail_detail = "验证集无新增失败 case" + else: + new_fail_detail = f"验证集新增失败 case:{'、'.join(new_fails)}" + if not new_fail_ok: + new_fail_detail += " —— 禁止新增 hard fail" + gates.append(GateResult(name="no_new_hard_fail", passed=new_fail_ok, detail=new_fail_detail)) + + # 3. 保护 case 不能退化 + violations = _protected_violations(cfg, delta_val) + gates.append( + GateResult( + name="protected_cases", + passed=not violations, + detail=(f"保护 case({'、'.join(cfg.protected_cases) or '无'})均未退化" + if not violations else f"保护 case 退化:{'、'.join(violations)}"), + )) + + # 4. 过拟合守卫 + overfit = (cfg.overfit_guard and delta_train.pass_rate_delta > 0 and delta_val.pass_rate_delta < 0) + gates.append( + GateResult( + name="overfit_guard", + passed=not overfit, + detail=(f"训练集通过率提升 {delta_train.pass_rate_delta:+.4f} 且验证集退化 " + f"{delta_val.pass_rate_delta:+.4f},判定过拟合,必须拒绝" + if overfit else f"未触发过拟合守卫(train {delta_train.pass_rate_delta:+.4f} / " + f"val {delta_val.pass_rate_delta:+.4f})"), + )) + + # 5. 成本预算(配置了 max_metric_calls 但拿不到 budget_used 时 fail-closed) + total_cost = float(optimize_result_view.get("total_llm_cost") or 0.0) + budget_used = optimize_result_view.get("budget_used") + cost_ok = total_cost <= cfg.max_cost_usd + budget_untracked = cfg.max_metric_calls is not None and budget_used is None + calls_ok = (cfg.max_metric_calls is None or (budget_used is not None and int(budget_used) <= cfg.max_metric_calls)) + calls_part = "" + if cfg.max_metric_calls is not None: + calls_part = (f";metric 调用 {budget_used if budget_used is not None else '未知'}" + f"(预算 {cfg.max_metric_calls})") + cost_detail = f"优化成本 ${total_cost:.4f}(预算 ${cfg.max_cost_usd:g}){calls_part}" + if budget_untracked: + cost_detail += " —— 预算追踪不可用:已配置 max_metric_calls 但无 budget_used 数据,按未通过处理" + elif not (cost_ok and calls_ok): + cost_detail += " —— 超出成本预算" + gates.append(GateResult(name="cost_budget", passed=cost_ok and calls_ok, detail=cost_detail)) + + # 6. 时长预算 + duration_ok = wall_seconds <= cfg.max_duration_seconds + duration_detail = f"pipeline 耗时 {wall_seconds:.1f}s(预算 {cfg.max_duration_seconds:g}s)" + if not duration_ok: + duration_detail += " —— 超出时长预算" + gates.append(GateResult(name="duration_budget", passed=duration_ok, detail=duration_detail)) + + accepted = all(g.passed for g in gates) + if accepted: + reason = "全部闸门通过:验证集有实际提升、无退化、成本与耗时均在预算内,候选值得接受。" + else: + failed = {g.name: g for g in gates if not g.passed} + key_gate = next(failed[name] for name in _REASON_SEVERITY if name in failed) + reason = f"闸门 {key_gate.name} 未通过:{key_gate.detail}" + if len(failed) > 1: + others = "、".join(name for name in failed if name != key_gate.name) + reason += f"(另有 {len(failed) - 1} 道闸门同时未通过:{others})" + return GateDecision(accepted=accepted, reason=reason, gates=gates) diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/optimize.py b/examples/optimization/eval_optimize_loop/loop_pipeline/optimize.py new file mode 100644 index 00000000..991d0737 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/optimize.py @@ -0,0 +1,88 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段③:AgentOptimizer 封装 —— 场景 → 优化配置 / 优化器验证集的映射。 + +三个演示场景只差两处输入(这正是「同一条 pipeline、不同数据/配置产生 +不同决策」的演示点): + +=========== ============================== ================================== +场景 optimizer 配置 优化器眼中的「验证集」 +=========== ============================== ================================== +success optimizer.json data/val.evalset.json(独立) +no_effect configs/optimizer.no_effect.json data/val.evalset.json(独立) +overfit configs/optimizer.overfit.json data/optimizer_probe.evalset.json + (与训练集同源的泄漏调参集!) +=========== ============================== ================================== + +overfit 场景的要点:优化器视角里 probe 集分数一路变好(0/3 → 3/3), +它自己完全不知道过拟合了 —— 只有 pipeline 阶段④ 用**独立** val 集复评 +才能揭穿。配置差异仅在 ``reflection_lm.model_name``(决定 fake 反思模型 +返回哪套候选)。 + +``AgentOptimizer.optimize`` 自身会把每轮候选、接受理由、成本、耗时、 +seed(config.snapshot.json)落盘到 ``output_dir`` —— 阶段⑥ 的审计产物 +直接复用这套 SDK 原生审计目录。 +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +from trpc_agent_sdk.evaluation import AgentOptimizer, OptimizeResult, TargetPrompt + +SCENARIOS = ("success", "no_effect", "overfit") + + +@dataclass(frozen=True) +class ScenarioSpec: + """一个演示场景的输入组合。""" + + name: str + optimizer_config: Path + optimizer_val_dataset: Path + train_dataset: Path + + +def resolve_scenario(name: str, example_root: Path) -> ScenarioSpec: + """场景名 → 输入组合;未知场景抛 ValueError。""" + root = example_root + train = root / "data" / "train.evalset.json" + if name == "success": + return ScenarioSpec(name, root / "optimizer.json", root / "data" / "val.evalset.json", train) + if name == "no_effect": + return ScenarioSpec(name, root / "configs" / "optimizer.no_effect.json", root / "data" / "val.evalset.json", + train) + if name == "overfit": + return ScenarioSpec(name, root / "configs" / "optimizer.overfit.json", + root / "data" / "optimizer_probe.evalset.json", train) + raise ValueError(f"未知场景 {name!r};可选:{', '.join(SCENARIOS)}") + + +async def run_optimization( + spec: ScenarioSpec, + *, + call_agent, + target: TargetPrompt, + output_dir: Path, +) -> OptimizeResult: + """跑一轮 AgentOptimizer;SDK 审计产物落在 ``output_dir`` 下。 + + ``update_source=False``:优化器结束后源 prompt 恢复 baseline; + 是否把最优候选写回源文件由 pipeline 的 gate 决策(``--apply``)决定, + 而不是优化器自作主张 —— 这是本闭环与"裸跑一次 AgentOptimizer"的 + 核心区别。 + """ + return await AgentOptimizer.optimize( + config_path=str(spec.optimizer_config), + call_agent=call_agent, + target_prompt=target, + train_dataset_path=str(spec.train_dataset), + validation_dataset_path=str(spec.optimizer_val_dataset), + output_dir=str(output_dir), + update_source=False, + verbose=0, + ) diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/regression.py b/examples/optimization/eval_optimize_loop/loop_pipeline/regression.py new file mode 100644 index 00000000..27479b0f --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/regression.py @@ -0,0 +1,133 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段④:候选回归 —— 换入候选 prompt 复评,与 baseline 做逐 case 对比。 + +两个关键设计: + +1. **换入/换出永不污染源文件**:``evaluate_candidate`` 先快照当前 prompt, + ``TargetPrompt.write_all``(原子写 + 回滚)换入候选,``try/finally`` + 保证评完必然还原 —— 即使评测中途抛异常。 +2. **delta 口径**:状态变化优先(fail→pass = ``new_pass``,pass→fail = + ``new_fail``),状态不变时按 case 平均分 ± epsilon 判 ``score_up`` / + ``score_down`` / ``unchanged``。这四类正是 issue 需求 4 点名的对比维度。 +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Literal, Optional + +from trpc_agent_sdk.evaluation import TargetPrompt + +from .evaluate import CaseEvalRecord, run_eval + +ChangeKind = Literal["new_pass", "new_fail", "score_up", "score_down", "unchanged"] + +CHANGE_KINDS: tuple[ChangeKind, ...] = ("new_pass", "new_fail", "score_up", "score_down", "unchanged") + +CHANGE_LABELS_ZH: dict[str, str] = { + "new_pass": "新增通过", + "new_fail": "新增失败", + "score_up": "分数提升", + "score_down": "分数下降", + "unchanged": "无变化", +} + + +@dataclass +class CaseDelta: + """一条 case 的 baseline vs candidate 对比。""" + + eval_id: str + baseline_passed: bool + candidate_passed: bool + baseline_score: float + candidate_score: float + change: ChangeKind + + +@dataclass +class DeltaSummary: + """一个切分的逐 case delta 汇总。""" + + per_case: list[CaseDelta] = field(default_factory=list) + pass_rate_delta: float = 0.0 + score_delta: float = 0.0 + counts: dict[str, int] = field(default_factory=dict) + + +def classify(baseline: CaseEvalRecord, candidate: CaseEvalRecord, eps: float) -> CaseDelta: + """单条 case 的 delta 分类(状态优先,分数其次)。""" + if not baseline.passed and candidate.passed: + change: ChangeKind = "new_pass" + elif baseline.passed and not candidate.passed: + change = "new_fail" + elif candidate.case_score > baseline.case_score + eps: + change = "score_up" + elif candidate.case_score < baseline.case_score - eps: + change = "score_down" + else: + change = "unchanged" + return CaseDelta( + eval_id=baseline.eval_id, + baseline_passed=baseline.passed, + candidate_passed=candidate.passed, + baseline_score=baseline.case_score, + candidate_score=candidate.case_score, + change=change, + ) + + +def compute_delta( + baseline: dict[str, CaseEvalRecord], + candidate: dict[str, CaseEvalRecord], + eps: float, +) -> DeltaSummary: + """整个切分的 delta 汇总;两侧 case 集合应一致(同一数据集)。""" + summary = DeltaSummary(counts={kind: 0 for kind in CHANGE_KINDS}) + for eval_id in sorted(baseline): + if eval_id not in candidate: # pragma: no cover - 同数据集不应发生 + continue + delta = classify(baseline[eval_id], candidate[eval_id], eps) + summary.per_case.append(delta) + summary.counts[delta.change] += 1 + total = len(summary.per_case) + if total: + baseline_pass = sum(1 for d in summary.per_case if d.baseline_passed) + candidate_pass = sum(1 for d in summary.per_case if d.candidate_passed) + summary.pass_rate_delta = (candidate_pass - baseline_pass) / total + summary.score_delta = sum(d.candidate_score - d.baseline_score for d in summary.per_case) / total + return summary + + +async def evaluate_candidate( + target: TargetPrompt, + candidate_prompts: dict[str, str], + datasets: dict[str, str], + eval_config_path: str, + *, + agent_module: Optional[str] = "loop_agent", +) -> dict[str, dict[str, CaseEvalRecord]]: + """换入候选 prompt → 逐数据集复评 → 无条件还原源 prompt。 + + Args: + target: 已注册全部 prompt 字段的 TargetPrompt。 + candidate_prompts: 候选 prompt 文本(键必须与 target 注册名一致)。 + datasets: 切分名 → evalset 路径(如 {"train": ..., "val": ...})。 + eval_config_path: 验收 metric 套件(与 baseline 同一份,口径可比)。 + + Returns: + 切分名 → (eval_id → CaseEvalRecord)。 + """ + snapshot = await target.read_all() + await target.write_all(candidate_prompts) + try: + results: dict[str, dict[str, CaseEvalRecord]] = {} + for split, dataset_path in datasets.items(): + results[split] = await run_eval(dataset_path, eval_config_path, agent_module=agent_module) + return results + finally: + await target.write_all(snapshot) # 永不污染源 prompt 文件 diff --git a/examples/optimization/eval_optimize_loop/loop_pipeline/report.py b/examples/optimization/eval_optimize_loop/loop_pipeline/report.py new file mode 100644 index 00000000..4207601d --- /dev/null +++ b/examples/optimization/eval_optimize_loop/loop_pipeline/report.py @@ -0,0 +1,384 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""阶段⑥:optimization_report.json / optimization_report.md 渲染与校验。 + +``optimization_report.json`` 顶层字段契约(验收标准 6;tests/test_pipeline_e2e +逐一断言,``validate_report`` 供测试与 ``--check`` 共用): + +- ``schema_version`` / ``scenario`` / ``generated_at`` / ``seed`` +- ``inputs`` train/val 数据集、optimizer 配置、pipeline 配置、prompt 源文件 +- ``baseline`` train/val 两个切分的分数与逐 case 明细(含失败归因与轨迹) +- ``attribution`` 失败类型聚类统计(counts_by_type / primary_by_case / details) +- ``optimization`` 优化器运行摘要(算法、状态、轮次、成本、审计目录) +- ``candidate`` 候选在 train/val 上的复评结果(结构同 baseline) +- ``delta`` 逐 case delta(new_pass / new_fail / score_up / score_down) +- ``gate_decision`` 接受/拒绝 + 理由 + 六道闸门明细 +- ``runtime`` 各阶段耗时与 fake 模型调用计数 + +``optimization_report.md`` 是给人看的版本:概览表、失败归因统计表、逐 case +delta 表、优化轮次摘要、gate 明细表,以及「是否值得接受」的中文结论段。 +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from .attribution import FAILURE_TYPE_LABELS_ZH, AttributionSummary, attribute_case +from .evaluate import CaseEvalRecord, summarize +from .gates import GateDecision +from .regression import CHANGE_LABELS_ZH, DeltaSummary + +SCHEMA_VERSION = "v1" + +# 报告顶层必备字段(validate_report / 测试共用) +REQUIRED_TOP_LEVEL_KEYS = ( + "schema_version", + "scenario", + "generated_at", + "seed", + "inputs", + "baseline", + "attribution", + "optimization", + "candidate", + "delta", + "gate_decision", + "runtime", +) + +_TRUNCATE = 200 # 逐 case 文本截断长度,控制报告体积 + + +def _clip(text: str) -> str: + text = text or "" + return text if len(text) <= _TRUNCATE else text[:_TRUNCATE] + "…" + + +def _case_view(record: CaseEvalRecord) -> dict[str, Any]: + """逐 case 摘要:分数、失败归因(类型+理由)、关键轨迹。""" + findings = attribute_case(record) + return { + "eval_id": record.eval_id, + "passed": record.passed, + "final_status": record.final_status, + "case_score": round(record.case_score, 6), + "metric_scores": record.metric_scores, + "metric_status": record.metric_status, + "failure_types": [f.type for f in findings], + "failure_reasons": [f"[{f.metric}] {f.explanation}({_clip(f.evidence)})" for f in findings], + "trajectory": { + "actual_tool_calls": record.actual_tool_calls, + "expected_tool_calls": record.expected_tool_calls, + }, + "actual_response": _clip(record.actual_response), + "expected_response": _clip(record.expected_response), + } + + +def _split_view(records: dict[str, CaseEvalRecord]) -> dict[str, Any]: + summary = summarize(records) + return { + "pass_rate": round(summary.pass_rate, 6), + "passed": summary.passed, + "total": summary.total, + "mean_score": round(summary.mean_case_score, 6), + "metric_breakdown": { + k: round(v, 6) + for k, v in summary.metric_breakdown.items() + }, + "per_case": [_case_view(records[eval_id]) for eval_id in sorted(records)], + } + + +def _attribution_view(train: AttributionSummary, val: AttributionSummary) -> dict[str, Any]: + counts: dict[str, int] = {} + for summary in (train, val): + for failure_type, count in summary.counts.items(): + counts[failure_type] = counts.get(failure_type, 0) + count + details: dict[str, list[dict]] = {} + primary: dict[str, str] = {} + for summary in (train, val): + primary.update(summary.primary) + for eval_id, findings in summary.per_case.items(): + details[eval_id] = [asdict(f) for f in findings] + return {"counts_by_type": counts, "primary_by_case": primary, "details": details} + + +def _delta_view(delta: DeltaSummary) -> dict[str, Any]: + return { + "pass_rate_delta": + round(delta.pass_rate_delta, 6), + "score_delta": + round(delta.score_delta, 6), + "counts": + delta.counts, + "per_case": [{ + "eval_id": d.eval_id, + "baseline_passed": d.baseline_passed, + "candidate_passed": d.candidate_passed, + "baseline_score": round(d.baseline_score, 6), + "candidate_score": round(d.candidate_score, 6), + "change": d.change, + } for d in delta.per_case], + } + + +def _prompt_digest(prompts: dict[str, str]) -> dict[str, dict[str, str]]: + """候选 prompt 的 sha256 + 摘要(完整文本已由 SDK 落盘 best_prompts/)。""" + return { + name: { + "sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), + "preview": _clip(text.strip()), + } + for name, text in prompts.items() + } + + +def build_report( + *, + scenario: str, + seed: int, + inputs: dict[str, Any], + baseline: dict[str, dict[str, CaseEvalRecord]], + attribution_train: AttributionSummary, + attribution_val: AttributionSummary, + optimize_result, + candidate: dict[str, dict[str, CaseEvalRecord]], + delta_train: DeltaSummary, + delta_val: DeltaSummary, + decision: GateDecision, + runtime: dict[str, Any], + optimize_artifacts_dir: str, +) -> dict[str, Any]: + """组装完整报告 dict(可 json 序列化)。""" + return { + "schema_version": SCHEMA_VERSION, + "scenario": scenario, + "generated_at": datetime.now(timezone.utc).isoformat(), + "seed": seed, + "inputs": inputs, + "baseline": { + split: _split_view(records) + for split, records in baseline.items() + }, + "attribution": _attribution_view(attribution_train, attribution_val), + "optimization": { + "algorithm": optimize_result.algorithm, + "status": optimize_result.status, + "finish_reason": optimize_result.finish_reason, + "stop_reason": optimize_result.stop_reason, + "total_rounds": optimize_result.total_rounds, + "rounds_accepted": sum(1 for r in optimize_result.rounds if r.accepted), + "optimizer_val_pass_rate": { + "baseline": round(optimize_result.baseline_pass_rate, 6), + "best": round(optimize_result.best_pass_rate, 6), + }, + "best_prompts": _prompt_digest(optimize_result.best_prompts), + "cost": { + "total_llm_cost": optimize_result.total_llm_cost, + "reflection_lm_calls": optimize_result.total_reflection_lm_calls, + "budget_used": (optimize_result.rounds[-1].budget_used if optimize_result.rounds else None), + "budget_total": (optimize_result.rounds[-1].budget_total if optimize_result.rounds else None), + "token_usage": optimize_result.total_token_usage, + }, + "duration_seconds": round(optimize_result.duration_seconds, 3), + "artifacts_dir": optimize_artifacts_dir, + }, + "candidate": { + split: _split_view(records) + for split, records in candidate.items() + }, + "delta": { + "train": _delta_view(delta_train), + "val": _delta_view(delta_val), + }, + "gate_decision": { + "accepted": decision.accepted, + "reason": decision.reason, + "gates": [{ + "name": g.name, + "passed": g.passed, + "detail": g.detail + } for g in decision.gates], + }, + "runtime": runtime, + } + + +def validate_report(report: dict[str, Any]) -> list[str]: + """校验报告契约;返回问题列表(空 = 通过)。测试与 --check 共用。""" + problems: list[str] = [] + for key in REQUIRED_TOP_LEVEL_KEYS: + if key not in report: + problems.append(f"缺少顶层字段:{key}") + if problems: + return problems + for split in ("train", "val"): + for section in ("baseline", "candidate"): + view = report[section].get(split) + if not isinstance(view, dict): + problems.append(f"{section}.{split} 缺失") + continue + for key in ("pass_rate", "mean_score", "metric_breakdown", "per_case"): + if key not in view: + problems.append(f"{section}.{split} 缺少 {key}") + delta = report["delta"].get(split) + if not isinstance(delta, dict): + problems.append(f"delta.{split} 缺失") + else: + for key in ("pass_rate_delta", "score_delta", "counts", "per_case"): + if key not in delta: + problems.append(f"delta.{split} 缺少 {key}") + for key in ("counts_by_type", "primary_by_case", "details"): + if key not in report["attribution"]: + problems.append(f"attribution 缺少 {key}") + decision = report["gate_decision"] + for key in ("accepted", "reason", "gates"): + if key not in decision: + problems.append(f"gate_decision 缺少 {key}") + for key in ("status", "total_rounds", "cost", "artifacts_dir", "optimizer_val_pass_rate"): + if key not in report["optimization"]: + problems.append(f"optimization 缺少 {key}") + # 每个失败 case 必须至少给出一个可解释原因(验收标准 4) + for section in ("baseline", "candidate"): + for split in ("train", "val"): + for case in report[section][split].get("per_case", []): + if not case.get("passed") and not case.get("failure_reasons"): + problems.append(f"{section}.{split} 的失败 case {case.get('eval_id')} 缺少失败原因") + return problems + + +# --------------------------------------------------------------------------- +# Markdown 渲染 +# --------------------------------------------------------------------------- + + +def _pct(value: float) -> str: + return f"{value * 100:.1f}%" + + +def _status_icon(passed: bool) -> str: + return "✅" if passed else "❌" + + +def render_markdown(report: dict[str, Any]) -> str: + """人话版报告:概览 / 归因 / 逐 case delta / 轮次 / gate / 结论。""" + baseline_val = report["baseline"]["val"] + baseline_train = report["baseline"]["train"] + candidate_val = report["candidate"]["val"] + candidate_train = report["candidate"]["train"] + delta_val = report["delta"]["val"] + delta_train = report["delta"]["train"] + decision = report["gate_decision"] + optimization = report["optimization"] + + lines: list[str] = [] + lines.append(f"# 优化报告 — 场景 `{report['scenario']}`") + lines.append("") + verdict = "✅ **接受候选 prompt**" if decision["accepted"] else "❌ **拒绝候选 prompt**" + lines.append(f"> 结论:{verdict}") + lines.append(f"> 理由:{decision['reason']}") + lines.append("") + lines.append(f"- 生成时间:{report['generated_at']} 随机种子:{report['seed']} " + f"报告 schema:{report['schema_version']}") + lines.append(f"- 优化算法:{optimization['algorithm']}(status={optimization['status']}," + f"{optimization['total_rounds']} 轮,接受 {optimization['rounds_accepted']} 轮," + f"耗时 {optimization['duration_seconds']}s)") + lines.append(f"- 审计产物目录:`{optimization['artifacts_dir']}`(每轮候选 prompt、评测结果、" + f"接受理由、成本、seed 快照均在其中)") + lines.append("") + + lines.append("## 一、baseline vs candidate 概览") + lines.append("") + lines.append("| 切分 | baseline 通过率 | candidate 通过率 | baseline 平均分 | candidate 平均分 | 通过率 Δ |") + lines.append("| --- | --- | --- | --- | --- | --- |") + for split_name, b, c, d in ( + ("train", baseline_train, candidate_train, delta_train), + ("val", baseline_val, candidate_val, delta_val), + ): + lines.append(f"| {split_name} | {_pct(b['pass_rate'])} ({b['passed']}/{b['total']}) " + f"| {_pct(c['pass_rate'])} ({c['passed']}/{c['total']}) " + f"| {b['mean_score']:.3f} | {c['mean_score']:.3f} " + f"| {d['pass_rate_delta']:+.3f} |") + lines.append("") + + lines.append("## 二、baseline 失败归因统计") + lines.append("") + counts = report["attribution"]["counts_by_type"] + if counts: + lines.append("| 失败类型 | 中文说明 | 涉及 case 数 |") + lines.append("| --- | --- | --- |") + for failure_type, count in counts.items(): + lines.append(f"| `{failure_type}` | {FAILURE_TYPE_LABELS_ZH.get(failure_type, failure_type)} | {count} |") + lines.append("") + lines.append("主要归因(每个失败 case 的根因):") + for eval_id, primary in sorted(report["attribution"]["primary_by_case"].items()): + lines.append(f"- `{eval_id}` → `{primary}`({FAILURE_TYPE_LABELS_ZH.get(primary, primary)})") + else: + lines.append("baseline 无失败 case。") + lines.append("") + + lines.append("## 三、逐 case delta(验证集为准,训练集附后)") + lines.append("") + for split_name, delta in (("val", delta_val), ("train", delta_train)): + lines.append(f"### {split_name}") + lines.append("") + lines.append("| case | baseline | candidate | 分数变化 | 判定 |") + lines.append("| --- | --- | --- | --- | --- |") + for case in delta["per_case"]: + lines.append(f"| `{case['eval_id']}` " + f"| {_status_icon(case['baseline_passed'])} {case['baseline_score']:.3f} " + f"| {_status_icon(case['candidate_passed'])} {case['candidate_score']:.3f} " + f"| {case['candidate_score'] - case['baseline_score']:+.3f} " + f"| {CHANGE_LABELS_ZH.get(case['change'], case['change'])}(`{case['change']}`) |") + lines.append("") + + lines.append("## 四、优化过程(优化器视角)") + lines.append("") + opt_view = optimization["optimizer_val_pass_rate"] + lines.append(f"- 优化器内部验证集通过率:{_pct(opt_view['baseline'])} → {_pct(opt_view['best'])}" + f"(注意:优化器只看 optimizer.json 里的弱指标;overfit 场景中它看到的还是" + f"泄漏调参集 —— 是否真的变好以上面的独立验证集复评为准)") + cost = optimization["cost"] + lines.append(f"- 成本:${cost['total_llm_cost']:.4f},反思 LM 调用 {cost['reflection_lm_calls']} 次," + f"metric 调用 {cost['budget_used']}/{cost['budget_total']}") + lines.append("") + + lines.append("## 五、gate 决策明细") + lines.append("") + lines.append("| 闸门 | 结果 | 说明 |") + lines.append("| --- | --- | --- |") + for gate in decision["gates"]: + lines.append(f"| `{gate['name']}` | {_status_icon(gate['passed'])} | {gate['detail']} |") + lines.append("") + + lines.append("## 六、是否值得接受") + lines.append("") + if decision["accepted"]: + lines.append("候选 prompt 在独立验证集上带来实际提升,且未引入任何回归:" + "无新增失败、保护 case 完好、成本与耗时都在预算内。" + "**建议接受**,可用 `--apply` 将最优候选写回源 prompt 文件。") + else: + lines.append(f"候选 prompt 未能通过接受策略:{decision['reason']} " + "**建议拒绝**,保持 baseline prompt 不变;" + "可根据上面的失败归因调整评测集或优化配置后重试。") + lines.append("") + return "\n".join(lines) + + +def write_reports(output_dir: Path, report: dict[str, Any]) -> tuple[Path, Path]: + """落盘 optimization_report.json / optimization_report.md。""" + output_dir.mkdir(parents=True, exist_ok=True) + json_path = output_dir / "optimization_report.json" + md_path = output_dir / "optimization_report.md" + json_path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + md_path.write_text(render_markdown(report), encoding="utf-8") + return json_path, md_path diff --git a/examples/optimization/eval_optimize_loop/optimizer.json b/examples/optimization/eval_optimize_loop/optimizer.json new file mode 100644 index 00000000..58c13cc4 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/optimizer.json @@ -0,0 +1,40 @@ +{ + "evaluate": { + "metrics": [ + { + "metric_name": "final_response_avg_score", + "threshold": 1.0, + "criterion": { + "final_response": { + "text": {"match": "exact", "case_insensitive": false} + } + } + } + ], + "num_runs": 1 + }, + "optimize": { + "eval_case_parallelism": 2, + "stop": { + "required_metrics": "all" + }, + "algorithm": { + "name": "gepa_reflective", + "seed": 42, + "reflection_lm": { + "provider_name": "fake-reflection", + "model_name": "success", + "generation_config": {"max_tokens": 2048, "temperature": 0.0} + }, + "candidate_selection_strategy": "pareto", + "module_selector": "round_robin", + "frontier_type": "instance", + "reflection_minibatch_size": 3, + "skip_perfect_score": false, + "use_merge": false, + "max_metric_calls": 60, + "score_threshold": 1.0, + "max_iterations_without_improvement": 4 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/pipeline.json b/examples/optimization/eval_optimize_loop/pipeline.json new file mode 100644 index 00000000..9f106150 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/pipeline.json @@ -0,0 +1,14 @@ +{ + "seed": 42, + "score_epsilon": 1e-6, + "gates": { + "min_val_pass_rate_improvement": 1e-9, + "min_val_score_improvement": 0.0, + "forbid_new_hard_fail": true, + "protected_cases": ["val_identity"], + "max_cost_usd": 1.0, + "max_metric_calls": null, + "max_duration_seconds": 180.0, + "overfit_guard": true + } +} diff --git a/examples/optimization/eval_optimize_loop/run_pipeline.py b/examples/optimization/eval_optimize_loop/run_pipeline.py new file mode 100644 index 00000000..f023b4af --- /dev/null +++ b/examples/optimization/eval_optimize_loop/run_pipeline.py @@ -0,0 +1,366 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""eval_optimize_loop 入口脚本:评测→归因→优化→回归→gate→审计 六阶段闭环。 + +适用场景 +-------- +你想知道「AgentOptimizer 改出来的 prompt 到底值不值得上线」:不是看优化器 +自己报的分数,而是用独立验证集复评、逐 case 对比、跑一遍可配置的接受策略, +并把每一步产物落盘可审计。本脚本零 API Key 可跑(fake agent / fake judge / +fake reflection LM),完整三场景 < 3 分钟。 + +怎么跑 +------ + # 单场景(默认 success);输出在 runs/<场景>-<时间戳>/ + python examples/optimization/eval_optimize_loop/run_pipeline.py + + # 三个场景全跑(success 接受;no_effect / overfit 拒绝) + python examples/optimization/eval_optimize_loop/run_pipeline.py --scenario all + + # baseline 用预录轨迹(trace 模式)做评测与归因,不执行 agent + python examples/optimization/eval_optimize_loop/run_pipeline.py --baseline-from-trace + + # gate 通过时把最优候选写回源 prompt 文件(谨慎!会改动 loop_agent/prompts/; + # --scenario all 时统一等全部场景跑完后再写回,避免污染后续场景 baseline) + python examples/optimization/eval_optimize_loop/run_pipeline.py --apply + + # 校验一份已有报告的字段契约 + python examples/optimization/eval_optimize_loop/run_pipeline.py \ + --check sample_output/success/optimization_report.json + +输出目录结构 +------------ + runs/<场景>-<时间戳>/ + ├── optimization_report.json # 结构化报告(AC6 契约) + ├── optimization_report.md # 人话版报告 + ├── baseline_eval.json # 阶段① 逐 case 原始记录 + ├── candidate_eval.json # 阶段④ 逐 case 原始记录 + ├── attribution.json # 阶段② 归因明细 + ├── pipeline_config.snapshot.json # 本次运行的 gate/seed 配置快照 + └── optimize/ # 阶段③ SDK 原生审计目录: + ├── result.json summary.txt run.log config.snapshot.json + ├── rounds/round_001.json …(每轮候选 prompt、接受理由、成本、耗时) + └── baseline_prompts/ best_prompts/ + +接入自己业务时改哪里 +-------------------- +- loop_agent/ : 换成你的 agent 包(保留 get_agent_async + call_agent 两个入口) +- data/*.evalset.json : 换成你的训练/验证评测集(验证集必须独立于训练集!) +- data/eval_config.json : 换成你的验收 metric 套件(judge_model 配真实模型) +- optimizer.json : reflection_lm 配真实模型;黑盒模式只能用响应类 metric +- pipeline.json : 按业务风险调整闸门(保护 case、预算、提升阈值) +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import sys +import time +from dataclasses import asdict +from datetime import datetime +from pathlib import Path + +# ---- 路径自举:让脚本在任意 cwd 下都能运行 ---- +_HERE = Path(__file__).resolve().parent +_REPO_ROOT = _HERE.parents[2] +for _p in (str(_REPO_ROOT), str(_HERE)): + if _p not in sys.path: + sys.path.insert(0, _p) + +from trpc_agent_sdk.evaluation import TargetPrompt # noqa: E402 + +import loop_agent # noqa: E402 +from loop_agent.fake_models import FakeAgentModel, FakeJudgeModel, FakeReflectionModel # noqa: E402 +from loop_pipeline.attribution import cluster # noqa: E402 +from loop_pipeline.config import PipelineConfig # noqa: E402 +from loop_pipeline.evaluate import run_eval # noqa: E402 +from loop_pipeline.gates import evaluate_gates # noqa: E402 +from loop_pipeline.optimize import SCENARIOS, resolve_scenario, run_optimization # noqa: E402 +from loop_pipeline.regression import compute_delta, evaluate_candidate # noqa: E402 +from loop_pipeline.report import build_report, validate_report, write_reports # noqa: E402 + +TRAIN_PATH = _HERE / "data" / "train.evalset.json" +VAL_PATH = _HERE / "data" / "val.evalset.json" +TRACE_PATH = _HERE / "data" / "trace_baseline.evalset.json" +EVAL_CONFIG_PATH = _HERE / "data" / "eval_config.json" +PIPELINE_CONFIG_PATH = _HERE / "pipeline.json" + + +def _build_target() -> TargetPrompt: + """注册两个 TargetPrompt 字段:system prompt + skill prompt。""" + return (TargetPrompt().add_path("system_prompt", + str(loop_agent.SYSTEM_PROMPT_PATH)).add_path("skill", str(loop_agent.SKILL_PATH))) + + +def _records_json(records_by_split: dict) -> dict: + return { + split: { + eval_id: asdict(rec) + for eval_id, rec in records.items() + } + for split, records in records_by_split.items() + } + + +def _dump(path: Path, payload: dict) -> None: + path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +async def run_scenario( + scenario: str, + output_root: Path, + *, + baseline_from_trace: bool = False, + apply_on_accept: bool = False, + apply_sink: list | None = None, + quiet: bool = False, +) -> dict: + """跑一个场景的完整六阶段闭环,返回报告 dict。""" + + def log(message: str) -> None: + if not quiet: + print(message) + + pipeline_config = PipelineConfig.load(PIPELINE_CONFIG_PATH) + spec = resolve_scenario(scenario, _HERE) + timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S") + out_dir = output_root / f"{scenario}-{timestamp}" + out_dir.mkdir(parents=True, exist_ok=True) + + calls_before = { + "agent": FakeAgentModel.calls, + "judge": FakeJudgeModel.calls, + "reflection": FakeReflectionModel.calls, + } + stage_durations: dict[str, float] = {} + started = time.monotonic() + + # ---- 阶段①:baseline 评测(train + val,完整验收 metric 套件) ---- + log(f"[{scenario}] ① baseline 评测 …") + t0 = time.monotonic() + if baseline_from_trace: + # trace 模式演示:评测 + 归因直接跑在预录轨迹上(不执行 agent)。 + # 回归对比仍使用真实 train/val 集,保证 delta 口径一致。 + trace_records = await run_eval(str(TRACE_PATH), str(EVAL_CONFIG_PATH), agent_module=None) + _dump(out_dir / "trace_eval.json", {eval_id: asdict(rec) for eval_id, rec in trace_records.items()}) + log(f"[{scenario}] trace 快照评测:{sum(r.passed for r in trace_records.values())}" + f"/{len(trace_records)} 通过(明细见 trace_eval.json)") + trace_attribution = cluster(trace_records) + _dump(out_dir / "trace_attribution.json", { + eval_id: [asdict(f) for f in findings] + for eval_id, findings in trace_attribution.per_case.items() + }) + log(f"[{scenario}] trace 快照归因:{trace_attribution.counts}(明细见 trace_attribution.json)") + baseline = { + "train": await run_eval(str(TRAIN_PATH), str(EVAL_CONFIG_PATH)), + "val": await run_eval(str(VAL_PATH), str(EVAL_CONFIG_PATH)), + } + stage_durations["baseline_eval"] = round(time.monotonic() - t0, 3) + + # ---- 阶段②:失败归因 ---- + t0 = time.monotonic() + attribution_train = cluster(baseline["train"]) + attribution_val = cluster(baseline["val"]) + stage_durations["attribution"] = round(time.monotonic() - t0, 3) + log(f"[{scenario}] ② 失败归因:train={attribution_train.counts} val={attribution_val.counts}") + + # ---- 阶段③:优化执行(AgentOptimizer / GEPA) ---- + log(f"[{scenario}] ③ AgentOptimizer 优化(配置 {spec.optimizer_config.name}," + f"优化器验证集 {spec.optimizer_val_dataset.name})…") + t0 = time.monotonic() + target = _build_target() + optimize_result = await run_optimization( + spec, + call_agent=loop_agent.call_agent, + target=target, + output_dir=out_dir / "optimize", + ) + stage_durations["optimize"] = round(time.monotonic() - t0, 3) + log(f"[{scenario}] 优化器视角:{optimize_result.baseline_pass_rate:.3f} → " + f"{optimize_result.best_pass_rate:.3f}(status={optimize_result.status}," + f"{optimize_result.total_rounds} 轮)") + + # ---- 阶段④:候选回归(独立 train/val 复评 + 逐 case delta) ---- + log(f"[{scenario}] ④ 候选回归(独立验证集复评)…") + t0 = time.monotonic() + candidate = await evaluate_candidate( + target, + optimize_result.best_prompts, + { + "train": str(TRAIN_PATH), + "val": str(VAL_PATH) + }, + str(EVAL_CONFIG_PATH), + ) + delta_train = compute_delta(baseline["train"], candidate["train"], pipeline_config.score_epsilon) + delta_val = compute_delta(baseline["val"], candidate["val"], pipeline_config.score_epsilon) + stage_durations["candidate_regression"] = round(time.monotonic() - t0, 3) + + # ---- 阶段⑤:接受策略(六道闸门) ---- + wall_seconds = time.monotonic() - started + last_round = optimize_result.rounds[-1] if optimize_result.rounds else None + decision = evaluate_gates( + pipeline_config.gates, + delta_val=delta_val, + delta_train=delta_train, + optimize_result_view={ + "total_llm_cost": optimize_result.total_llm_cost, + "budget_used": last_round.budget_used if last_round else None, + "duration_seconds": optimize_result.duration_seconds, + }, + wall_seconds=wall_seconds, + ) + log(f"[{scenario}] ⑤ gate 决策:{'ACCEPT' if decision.accepted else 'REJECT'} —— {decision.reason}") + + # ---- 阶段⑥:审计落盘 ---- + runtime = { + "pipeline_duration_seconds": round(time.monotonic() - started, 3), + "stage_durations": stage_durations, + "baseline_from_trace": baseline_from_trace, + "fake_model_calls": { + "agent": FakeAgentModel.calls - calls_before["agent"], + "judge": FakeJudgeModel.calls - calls_before["judge"], + "reflection": FakeReflectionModel.calls - calls_before["reflection"], + }, + } + report = build_report( + scenario=scenario, + seed=pipeline_config.seed, + inputs={ + "train_evalset": str(TRAIN_PATH.relative_to(_HERE)), + "val_evalset": str(VAL_PATH.relative_to(_HERE)), + "optimizer_config": str(spec.optimizer_config.relative_to(_HERE)), + "optimizer_val_dataset": str(spec.optimizer_val_dataset.relative_to(_HERE)), + "eval_config": str(EVAL_CONFIG_PATH.relative_to(_HERE)), + "pipeline_config": str(PIPELINE_CONFIG_PATH.relative_to(_HERE)), + "prompt_sources": { + "system_prompt": str(loop_agent.SYSTEM_PROMPT_PATH.relative_to(_HERE)), + "skill": str(loop_agent.SKILL_PATH.relative_to(_HERE)), + }, + }, + baseline=baseline, + attribution_train=attribution_train, + attribution_val=attribution_val, + optimize_result=optimize_result, + candidate=candidate, + delta_train=delta_train, + delta_val=delta_val, + decision=decision, + runtime=runtime, + # 相对于报告所在目录的路径:报告与审计产物同目录移动/归档时仍然有效 + optimize_artifacts_dir="optimize/", + ) + + problems = validate_report(report) + if problems: # pragma: no cover - 契约破坏应立即暴露 + raise RuntimeError(f"报告契约校验失败:{problems}") + + json_path, md_path = write_reports(out_dir, report) + _dump(out_dir / "baseline_eval.json", _records_json(baseline)) + _dump(out_dir / "candidate_eval.json", _records_json(candidate)) + _dump( + out_dir / "attribution.json", { + "train": { + eid: [asdict(f) for f in findings] + for eid, findings in attribution_train.per_case.items() + }, + "val": { + eid: [asdict(f) for f in findings] + for eid, findings in attribution_val.per_case.items() + }, + }) + _dump(out_dir / "pipeline_config.snapshot.json", pipeline_config.model_dump()) + + # --apply:gate 通过时把最优候选写回源 prompt(默认关闭)。 + # 提供 apply_sink 时只登记不写盘(defer-then-apply):--scenario all 必须等 + # 全部场景跑完后统一写回,否则先接受场景的写回会污染后续场景的 baseline。 + if apply_on_accept and decision.accepted: + if apply_sink is not None: + apply_sink.append({"scenario": scenario, "target": target, "prompts": optimize_result.best_prompts}) + log(f"[{scenario}] --apply:gate 通过,候选已登记,待全部场景结束后统一写回") + else: + await target.write_all(optimize_result.best_prompts) + log(f"[{scenario}] --apply:已把最优候选写回源 prompt 文件(loop_agent/prompts/)") + + log(f"[{scenario}] ⑥ 报告已生成:{json_path}\n") + return report + + +async def _amain(args: argparse.Namespace) -> int: + output_root = Path(args.output).resolve() if args.output else (_HERE / "runs") + scenarios = list(SCENARIOS) if args.scenario == "all" else [args.scenario] + results = [] + pending_applies: list[dict] = [] + for scenario in scenarios: + report = await run_scenario( + scenario, + output_root, + baseline_from_trace=args.baseline_from_trace, + apply_on_accept=args.apply, + apply_sink=pending_applies, + quiet=args.quiet, + ) + results.append((scenario, report)) + # --apply 统一延后到全部场景结束(defer-then-apply): + # 避免 --scenario all 时先接受场景的写回污染后续场景的 baseline 评测。 + for pending in pending_applies: + await pending["target"].write_all(pending["prompts"]) + if not args.quiet: + print(f"[{pending['scenario']}] --apply:已把最优候选写回源 prompt 文件(loop_agent/prompts/)") + if not args.quiet: + print("=" * 72) + for scenario, report in results: + decision = report["gate_decision"] + verdict = "ACCEPT ✅" if decision["accepted"] else "REJECT ❌" + print(f"{scenario:>10}: {verdict} val Δ通过率 {report['delta']['val']['pass_rate_delta']:+.3f} " + f"—— {decision['reason'][:60]}") + return 0 + + +def main() -> int: + parser = argparse.ArgumentParser(description="Evaluation + Optimization 自动回归与提示词优化闭环") + parser.add_argument("--scenario", + choices=[*SCENARIOS, "all"], + default="success", + help="演示场景:success 接受 / no_effect、overfit 拒绝;all 顺序全跑") + parser.add_argument("--output", default=None, help="输出根目录(默认 example 下的 runs/)") + parser.add_argument("--baseline-from-trace", + action="store_true", + help="额外用预录轨迹(trace 模式)跑一遍 baseline 评测与归因,不执行 agent") + parser.add_argument("--apply", + action="store_true", + help="gate 通过时把最优候选写回源 prompt 文件(会修改 loop_agent/prompts/;" + "--scenario all 时等全部场景结束后统一写回)") + parser.add_argument("--quiet", action="store_true", help="静默模式") + parser.add_argument("--check", metavar="REPORT_JSON", default=None, help="只校验一份 optimization_report.json 的字段契约后退出") + args = parser.parse_args() + + if args.check: + # cwd 无关:相对路径先按当前目录解析,找不到再回退到 example 目录, + # 让 README 里从仓库根目录复制的命令直接可用;文件缺失给友好错误。 + check_path = Path(args.check) + if not check_path.is_file() and not check_path.is_absolute() and (_HERE / check_path).is_file(): + check_path = _HERE / check_path + if not check_path.is_file(): + print(f"报告文件不存在:{args.check}(已按当前目录与 example 目录 {_HERE} 解析)", file=sys.stderr) + return 1 + report = json.loads(check_path.read_text(encoding="utf-8")) + problems = validate_report(report) + if problems: + print("报告契约校验失败:") + for problem in problems: + print(f" - {problem}") + return 1 + print(f"报告契约校验通过:{check_path}") + return 0 + + return asyncio.run(_amain(args)) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.json b/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.json new file mode 100644 index 00000000..7cb9577a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.json @@ -0,0 +1,846 @@ +{ + "schema_version": "v1", + "scenario": "no_effect", + "generated_at": "2026-07-06T15:49:26.626079+00:00", + "seed": 42, + "inputs": { + "train_evalset": "data/train.evalset.json", + "val_evalset": "data/val.evalset.json", + "optimizer_config": "configs/optimizer.no_effect.json", + "optimizer_val_dataset": "data/val.evalset.json", + "eval_config": "data/eval_config.json", + "pipeline_config": "pipeline.json", + "prompt_sources": { + "system_prompt": "loop_agent/prompts/system.md", + "skill": "loop_agent/prompts/skill.md" + } + }, + "baseline": { + "train": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "3 公里等于 3000 米", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座很不错的城市。", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "5 公里等于 5000 米", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "杭州是一座很不错的城市。", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "attribution": { + "counts_by_type": { + "wrong_tool_call": 2, + "wrong_tool_args": 2, + "knowledge_recall_miss": 2, + "format_violation": 2, + "llm_rubric_fail": 4, + "final_answer_mismatch": 2 + }, + "primary_by_case": { + "train_convert_3km": "wrong_tool_args", + "train_intro_shenzhen": "wrong_tool_call", + "val_convert_5km": "wrong_tool_args", + "val_intro_hangzhou": "wrong_tool_call" + }, + "details": { + "train_convert_3km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "train_intro_shenzhen": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_convert_5km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_intro_hangzhou": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ] + } + }, + "optimization": { + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finish_reason": "no_improvement", + "stop_reason": "no_improvement", + "total_rounds": 4, + "rounds_accepted": 0, + "optimizer_val_pass_rate": { + "baseline": 0.333333, + "best": 0.333333 + }, + "best_prompts": { + "system_prompt": { + "sha256": "97bdaa7cd2b5d2e14ab5db6fa874a79569d62c9436e832334209630bf1ccbc9f", + "preview": "\n\n\n# 角色\n\n你是「城市信息助手 CityInfo」,负责回答三类问题:\n\n1. **距离换算**:把公里换算成米(调用 `conver…" + }, + "skill": { + "sha256": "7e8bd4de20170ef7abe8f695dabc76159cac4d2c50984a4f57858908774deb10", + "preview": "\n\n# 回答方法\n\n- 先判断问题属于哪一类(换算 / 介绍 / 身份),再决定是否调用工具。\n- 回答保持简洁,不要输出与问题无关的内容。" + } + }, + "cost": { + "total_llm_cost": 0.0, + "reflection_lm_calls": 4, + "budget_used": 27, + "budget_total": 60, + "token_usage": { + "prompt": 0, + "completion": 0, + "total": 0 + } + }, + "duration_seconds": 0.17, + "artifacts_dir": "optimize/" + }, + "candidate": { + "train": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "3 公里等于 3000 米", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座很不错的城市。", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "5 公里等于 5000 米", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "杭州是一座很不错的城市。", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "delta": { + "train": { + "pass_rate_delta": 0.0, + "score_delta": 0.0, + "counts": { + "new_pass": 0, + "new_fail": 0, + "score_up": 0, + "score_down": 0, + "unchanged": 3 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.375, + "candidate_score": 0.375, + "change": "unchanged" + }, + { + "eval_id": "train_identity", + "baseline_passed": true, + "candidate_passed": true, + "baseline_score": 1.0, + "candidate_score": 1.0, + "change": "unchanged" + }, + { + "eval_id": "train_intro_shenzhen", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.125, + "candidate_score": 0.125, + "change": "unchanged" + } + ] + }, + "val": { + "pass_rate_delta": 0.0, + "score_delta": 0.0, + "counts": { + "new_pass": 0, + "new_fail": 0, + "score_up": 0, + "score_down": 0, + "unchanged": 3 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.375, + "candidate_score": 0.375, + "change": "unchanged" + }, + { + "eval_id": "val_identity", + "baseline_passed": true, + "candidate_passed": true, + "baseline_score": 1.0, + "candidate_score": 1.0, + "change": "unchanged" + }, + { + "eval_id": "val_intro_hangzhou", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.125, + "candidate_score": 0.125, + "change": "unchanged" + } + ] + } + }, + "gate_decision": { + "accepted": false, + "reason": "闸门 min_val_improvement 未通过:验证集通过率提升 +0.0000(要求 ≥ 1e-09),平均分提升 +0.0000(要求 ≥ 0) —— 提升不足,不值得接受", + "gates": [ + { + "name": "min_val_improvement", + "passed": false, + "detail": "验证集通过率提升 +0.0000(要求 ≥ 1e-09),平均分提升 +0.0000(要求 ≥ 0) —— 提升不足,不值得接受" + }, + { + "name": "no_new_hard_fail", + "passed": true, + "detail": "验证集无新增失败 case" + }, + { + "name": "protected_cases", + "passed": true, + "detail": "保护 case(val_identity)均未退化" + }, + { + "name": "overfit_guard", + "passed": true, + "detail": "未触发过拟合守卫(train +0.0000 / val +0.0000)" + }, + { + "name": "cost_budget", + "passed": true, + "detail": "优化成本 $0.0000(预算 $1)" + }, + { + "name": "duration_budget", + "passed": true, + "detail": "pipeline 耗时 0.3s(预算 180s)" + } + ] + }, + "runtime": { + "pipeline_duration_seconds": 0.297, + "stage_durations": { + "baseline_eval": 0.046, + "attribution": 0.0, + "optimize": 0.174, + "candidate_regression": 0.077 + }, + "baseline_from_trace": false, + "fake_model_calls": { + "agent": 52, + "judge": 24, + "reflection": 4 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.md b/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.md new file mode 100644 index 00000000..17e43068 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/no_effect/optimization_report.md @@ -0,0 +1,70 @@ +# 优化报告 — 场景 `no_effect` + +> 结论:❌ **拒绝候选 prompt** +> 理由:闸门 min_val_improvement 未通过:验证集通过率提升 +0.0000(要求 ≥ 1e-09),平均分提升 +0.0000(要求 ≥ 0) —— 提升不足,不值得接受 + +- 生成时间:2026-07-06T15:49:26.626079+00:00 随机种子:42 报告 schema:v1 +- 优化算法:gepa_reflective(status=SUCCEEDED,4 轮,接受 0 轮,耗时 0.17s) +- 审计产物目录:`optimize/`(每轮候选 prompt、评测结果、接受理由、成本、seed 快照均在其中) + +## 一、baseline vs candidate 概览 + +| 切分 | baseline 通过率 | candidate 通过率 | baseline 平均分 | candidate 平均分 | 通过率 Δ | +| --- | --- | --- | --- | --- | --- | +| train | 33.3% (1/3) | 33.3% (1/3) | 0.500 | 0.500 | +0.000 | +| val | 33.3% (1/3) | 33.3% (1/3) | 0.500 | 0.500 | +0.000 | + +## 二、baseline 失败归因统计 + +| 失败类型 | 中文说明 | 涉及 case 数 | +| --- | --- | --- | +| `wrong_tool_call` | 工具调用错误 | 2 | +| `wrong_tool_args` | 工具参数错误 | 2 | +| `knowledge_recall_miss` | 知识召回不足 | 2 | +| `format_violation` | 格式不符合要求 | 2 | +| `llm_rubric_fail` | LLM rubric 不达标 | 4 | +| `final_answer_mismatch` | 最终回复不匹配 | 2 | + +主要归因(每个失败 case 的根因): +- `train_convert_3km` → `wrong_tool_args`(工具参数错误) +- `train_intro_shenzhen` → `wrong_tool_call`(工具调用错误) +- `val_convert_5km` → `wrong_tool_args`(工具参数错误) +- `val_intro_hangzhou` → `wrong_tool_call`(工具调用错误) + +## 三、逐 case delta(验证集为准,训练集附后) + +### val + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `val_convert_5km` | ❌ 0.375 | ❌ 0.375 | +0.000 | 无变化(`unchanged`) | +| `val_identity` | ✅ 1.000 | ✅ 1.000 | +0.000 | 无变化(`unchanged`) | +| `val_intro_hangzhou` | ❌ 0.125 | ❌ 0.125 | +0.000 | 无变化(`unchanged`) | + +### train + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `train_convert_3km` | ❌ 0.375 | ❌ 0.375 | +0.000 | 无变化(`unchanged`) | +| `train_identity` | ✅ 1.000 | ✅ 1.000 | +0.000 | 无变化(`unchanged`) | +| `train_intro_shenzhen` | ❌ 0.125 | ❌ 0.125 | +0.000 | 无变化(`unchanged`) | + +## 四、优化过程(优化器视角) + +- 优化器内部验证集通过率:33.3% → 33.3%(注意:优化器只看 optimizer.json 里的弱指标;overfit 场景中它看到的还是泄漏调参集 —— 是否真的变好以上面的独立验证集复评为准) +- 成本:$0.0000,反思 LM 调用 4 次,metric 调用 27/60 + +## 五、gate 决策明细 + +| 闸门 | 结果 | 说明 | +| --- | --- | --- | +| `min_val_improvement` | ❌ | 验证集通过率提升 +0.0000(要求 ≥ 1e-09),平均分提升 +0.0000(要求 ≥ 0) —— 提升不足,不值得接受 | +| `no_new_hard_fail` | ✅ | 验证集无新增失败 case | +| `protected_cases` | ✅ | 保护 case(val_identity)均未退化 | +| `overfit_guard` | ✅ | 未触发过拟合守卫(train +0.0000 / val +0.0000) | +| `cost_budget` | ✅ | 优化成本 $0.0000(预算 $1) | +| `duration_budget` | ✅ | pipeline 耗时 0.3s(预算 180s) | + +## 六、是否值得接受 + +候选 prompt 未能通过接受策略:闸门 min_val_improvement 未通过:验证集通过率提升 +0.0000(要求 ≥ 1e-09),平均分提升 +0.0000(要求 ≥ 0) —— 提升不足,不值得接受 **建议拒绝**,保持 baseline prompt 不变;可根据上面的失败归因调整评测集或优化配置后重试。 diff --git a/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.json b/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.json new file mode 100644 index 00000000..9104404c --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.json @@ -0,0 +1,829 @@ +{ + "schema_version": "v1", + "scenario": "overfit", + "generated_at": "2026-07-06T15:49:26.941482+00:00", + "seed": 42, + "inputs": { + "train_evalset": "data/train.evalset.json", + "val_evalset": "data/val.evalset.json", + "optimizer_config": "configs/optimizer.overfit.json", + "optimizer_val_dataset": "data/optimizer_probe.evalset.json", + "eval_config": "data/eval_config.json", + "pipeline_config": "pipeline.json", + "prompt_sources": { + "system_prompt": "loop_agent/prompts/system.md", + "skill": "loop_agent/prompts/skill.md" + } + }, + "baseline": { + "train": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "3 公里等于 3000 米", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座很不错的城市。", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "5 公里等于 5000 米", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "杭州是一座很不错的城市。", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "attribution": { + "counts_by_type": { + "wrong_tool_call": 2, + "wrong_tool_args": 2, + "knowledge_recall_miss": 2, + "format_violation": 2, + "llm_rubric_fail": 4, + "final_answer_mismatch": 2 + }, + "primary_by_case": { + "train_convert_3km": "wrong_tool_args", + "train_intro_shenzhen": "wrong_tool_call", + "val_convert_5km": "wrong_tool_args", + "val_intro_hangzhou": "wrong_tool_call" + }, + "details": { + "train_convert_3km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "train_intro_shenzhen": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_convert_5km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_intro_hangzhou": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ] + } + }, + "optimization": { + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finish_reason": "completed", + "stop_reason": "score_threshold", + "total_rounds": 1, + "rounds_accepted": 1, + "optimizer_val_pass_rate": { + "baseline": 0.0, + "best": 1.0 + }, + "best_prompts": { + "system_prompt": { + "sha256": "56ced11884df63980ca4c83614c10c29c68f5907a4b30a5e22c0480e5937bd82", + "preview": "\n\n\n# 角色\n\n你是「城市信息助手 CityInfo」。\n\n# 应试技巧(这是一份刻意过拟合的候选 prompt)\n\n- …" + }, + "skill": { + "sha256": "7e8bd4de20170ef7abe8f695dabc76159cac4d2c50984a4f57858908774deb10", + "preview": "\n\n# 回答方法\n\n- 先判断问题属于哪一类(换算 / 介绍 / 身份),再决定是否调用工具。\n- 回答保持简洁,不要输出与问题无关的内容。" + } + }, + "cost": { + "total_llm_cost": 0.0, + "reflection_lm_calls": 1, + "budget_used": 12, + "budget_total": 60, + "token_usage": { + "prompt": 0, + "completion": 0, + "total": 0 + } + }, + "duration_seconds": 0.12, + "artifacts_dir": "optimize/" + }, + "candidate": { + "train": { + "pass_rate": 1.0, + "passed": 3, + "total": 3, + "mean_score": 1.0, + "metric_breakdown": { + "final_response_avg_score": 1.0, + "llm_rubric_knowledge_recall": 1.0, + "llm_rubric_response": 1.0, + "tool_trajectory_avg_score": 1.0 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "{\"result\": 3000, \"unit\": \"m\"}", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 0.0, + "passed": 0, + "total": 3, + "mean_score": 0.416667, + "metric_breakdown": { + "final_response_avg_score": 0.0, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_call", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:convert_distance)(期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 (无调用))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「根据以往训练经验,答案与训练样本一致。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "根据以往训练经验,答案与训练样本一致。", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": false, + "final_status": "FAILED", + "case_score": 0.75, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "final_answer_mismatch" + ], + "failure_reasons": [ + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「我是城市信息助手 CityInfo。」,实际「根据以往训练经验,答案与训练样本一致。」)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "根据以往训练经验,答案与训练样本一致。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「根据以往训练经验,答案与训练样本一致。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "根据以往训练经验,答案与训练样本一致。", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "delta": { + "train": { + "pass_rate_delta": 0.666667, + "score_delta": 0.5, + "counts": { + "new_pass": 2, + "new_fail": 0, + "score_up": 0, + "score_down": 0, + "unchanged": 1 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.375, + "candidate_score": 1.0, + "change": "new_pass" + }, + { + "eval_id": "train_identity", + "baseline_passed": true, + "candidate_passed": true, + "baseline_score": 1.0, + "candidate_score": 1.0, + "change": "unchanged" + }, + { + "eval_id": "train_intro_shenzhen", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.125, + "candidate_score": 1.0, + "change": "new_pass" + } + ] + }, + "val": { + "pass_rate_delta": -0.333333, + "score_delta": -0.083333, + "counts": { + "new_pass": 0, + "new_fail": 1, + "score_up": 0, + "score_down": 0, + "unchanged": 2 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.375, + "candidate_score": 0.375, + "change": "unchanged" + }, + { + "eval_id": "val_identity", + "baseline_passed": true, + "candidate_passed": false, + "baseline_score": 1.0, + "candidate_score": 0.75, + "change": "new_fail" + }, + { + "eval_id": "val_intro_hangzhou", + "baseline_passed": false, + "candidate_passed": false, + "baseline_score": 0.125, + "candidate_score": 0.125, + "change": "unchanged" + } + ] + } + }, + "gate_decision": { + "accepted": false, + "reason": "闸门 overfit_guard 未通过:训练集通过率提升 +0.6667 且验证集退化 -0.3333,判定过拟合,必须拒绝(另有 3 道闸门同时未通过:min_val_improvement、no_new_hard_fail、protected_cases)", + "gates": [ + { + "name": "min_val_improvement", + "passed": false, + "detail": "验证集通过率提升 -0.3333(要求 ≥ 1e-09),平均分提升 -0.0833(要求 ≥ 0) —— 提升不足,不值得接受" + }, + { + "name": "no_new_hard_fail", + "passed": false, + "detail": "验证集新增失败 case:val_identity —— 禁止新增 hard fail" + }, + { + "name": "protected_cases", + "passed": false, + "detail": "保护 case 退化:val_identity(new_fail)" + }, + { + "name": "overfit_guard", + "passed": false, + "detail": "训练集通过率提升 +0.6667 且验证集退化 -0.3333,判定过拟合,必须拒绝" + }, + { + "name": "cost_budget", + "passed": true, + "detail": "优化成本 $0.0000(预算 $1)" + }, + { + "name": "duration_budget", + "passed": true, + "detail": "pipeline 耗时 0.3s(预算 180s)" + } + ] + }, + "runtime": { + "pipeline_duration_seconds": 0.308, + "stage_durations": { + "baseline_eval": 0.088, + "attribution": 0.0, + "optimize": 0.123, + "candidate_regression": 0.096 + }, + "baseline_from_trace": false, + "fake_model_calls": { + "agent": 34, + "judge": 24, + "reflection": 1 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.md b/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.md new file mode 100644 index 00000000..e44d9ea0 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/overfit/optimization_report.md @@ -0,0 +1,70 @@ +# 优化报告 — 场景 `overfit` + +> 结论:❌ **拒绝候选 prompt** +> 理由:闸门 overfit_guard 未通过:训练集通过率提升 +0.6667 且验证集退化 -0.3333,判定过拟合,必须拒绝(另有 3 道闸门同时未通过:min_val_improvement、no_new_hard_fail、protected_cases) + +- 生成时间:2026-07-06T15:49:26.941482+00:00 随机种子:42 报告 schema:v1 +- 优化算法:gepa_reflective(status=SUCCEEDED,1 轮,接受 1 轮,耗时 0.12s) +- 审计产物目录:`optimize/`(每轮候选 prompt、评测结果、接受理由、成本、seed 快照均在其中) + +## 一、baseline vs candidate 概览 + +| 切分 | baseline 通过率 | candidate 通过率 | baseline 平均分 | candidate 平均分 | 通过率 Δ | +| --- | --- | --- | --- | --- | --- | +| train | 33.3% (1/3) | 100.0% (3/3) | 0.500 | 1.000 | +0.667 | +| val | 33.3% (1/3) | 0.0% (0/3) | 0.500 | 0.417 | -0.333 | + +## 二、baseline 失败归因统计 + +| 失败类型 | 中文说明 | 涉及 case 数 | +| --- | --- | --- | +| `wrong_tool_call` | 工具调用错误 | 2 | +| `wrong_tool_args` | 工具参数错误 | 2 | +| `knowledge_recall_miss` | 知识召回不足 | 2 | +| `format_violation` | 格式不符合要求 | 2 | +| `llm_rubric_fail` | LLM rubric 不达标 | 4 | +| `final_answer_mismatch` | 最终回复不匹配 | 2 | + +主要归因(每个失败 case 的根因): +- `train_convert_3km` → `wrong_tool_args`(工具参数错误) +- `train_intro_shenzhen` → `wrong_tool_call`(工具调用错误) +- `val_convert_5km` → `wrong_tool_args`(工具参数错误) +- `val_intro_hangzhou` → `wrong_tool_call`(工具调用错误) + +## 三、逐 case delta(验证集为准,训练集附后) + +### val + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `val_convert_5km` | ❌ 0.375 | ❌ 0.375 | +0.000 | 无变化(`unchanged`) | +| `val_identity` | ✅ 1.000 | ❌ 0.750 | -0.250 | 新增失败(`new_fail`) | +| `val_intro_hangzhou` | ❌ 0.125 | ❌ 0.125 | +0.000 | 无变化(`unchanged`) | + +### train + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `train_convert_3km` | ❌ 0.375 | ✅ 1.000 | +0.625 | 新增通过(`new_pass`) | +| `train_identity` | ✅ 1.000 | ✅ 1.000 | +0.000 | 无变化(`unchanged`) | +| `train_intro_shenzhen` | ❌ 0.125 | ✅ 1.000 | +0.875 | 新增通过(`new_pass`) | + +## 四、优化过程(优化器视角) + +- 优化器内部验证集通过率:0.0% → 100.0%(注意:优化器只看 optimizer.json 里的弱指标;overfit 场景中它看到的还是泄漏调参集 —— 是否真的变好以上面的独立验证集复评为准) +- 成本:$0.0000,反思 LM 调用 1 次,metric 调用 12/60 + +## 五、gate 决策明细 + +| 闸门 | 结果 | 说明 | +| --- | --- | --- | +| `min_val_improvement` | ❌ | 验证集通过率提升 -0.3333(要求 ≥ 1e-09),平均分提升 -0.0833(要求 ≥ 0) —— 提升不足,不值得接受 | +| `no_new_hard_fail` | ❌ | 验证集新增失败 case:val_identity —— 禁止新增 hard fail | +| `protected_cases` | ❌ | 保护 case 退化:val_identity(new_fail) | +| `overfit_guard` | ❌ | 训练集通过率提升 +0.6667 且验证集退化 -0.3333,判定过拟合,必须拒绝 | +| `cost_budget` | ✅ | 优化成本 $0.0000(预算 $1) | +| `duration_budget` | ✅ | pipeline 耗时 0.3s(预算 180s) | + +## 六、是否值得接受 + +候选 prompt 未能通过接受策略:闸门 overfit_guard 未通过:训练集通过率提升 +0.6667 且验证集退化 -0.3333,判定过拟合,必须拒绝(另有 3 道闸门同时未通过:min_val_improvement、no_new_hard_fail、protected_cases) **建议拒绝**,保持 baseline prompt 不变;可根据上面的失败归因调整评测集或优化配置后重试。 diff --git a/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.json b/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.json new file mode 100644 index 00000000..41d22cd6 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.json @@ -0,0 +1,820 @@ +{ + "schema_version": "v1", + "scenario": "success", + "generated_at": "2026-07-06T15:49:26.322531+00:00", + "seed": 42, + "inputs": { + "train_evalset": "data/train.evalset.json", + "val_evalset": "data/val.evalset.json", + "optimizer_config": "optimizer.json", + "optimizer_val_dataset": "data/val.evalset.json", + "eval_config": "data/eval_config.json", + "pipeline_config": "pipeline.json", + "prompt_sources": { + "system_prompt": "loop_agent/prompts/system.md", + "skill": "loop_agent/prompts/skill.md" + } + }, + "baseline": { + "train": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "3 公里等于 3000 米", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座很不错的城市。", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 0.333333, + "passed": 1, + "total": 3, + "mean_score": 0.5, + "metric_breakdown": { + "final_response_avg_score": 0.333333, + "llm_rubric_knowledge_recall": 0.666667, + "llm_rubric_response": 0.666667, + "tool_trajectory_avg_score": 0.333333 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": false, + "final_status": "FAILED", + "case_score": 0.375, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [ + "wrong_tool_args", + "format_violation", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具选择正确,但调用参数与期望不一致(期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"}))", + "[final_response_avg_score] 格式不符合要求:期望结构化 JSON 输出,实际是自由文本(期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "公里" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "5 公里等于 5000 米", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": false, + "final_status": "FAILED", + "case_score": 0.125, + "metric_scores": { + "tool_trajectory_avg_score": 0.0, + "final_response_avg_score": 0.0, + "llm_rubric_response": 0.5, + "llm_rubric_knowledge_recall": 0.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "FAILED", + "final_response_avg_score": "FAILED", + "llm_rubric_response": "FAILED", + "llm_rubric_knowledge_recall": "FAILED" + }, + "failure_types": [ + "wrong_tool_call", + "knowledge_recall_miss", + "knowledge_recall_miss", + "final_answer_mismatch", + "llm_rubric_fail" + ], + "failure_reasons": [ + "[tool_trajectory_avg_score] 工具调用集合与期望不一致(缺少调用:knowledge_search)(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[tool_trajectory_avg_score] 缺少知识检索调用(knowledge_search),无法召回作答所需知识(期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用))", + "[llm_rubric_knowledge_recall] 知识召回不足:检索结果无法支撑作答所需的关键信息(未通过 rubric:(未提供 rubric 明细)。)", + "[final_response_avg_score] 最终回复与参考答案不匹配(期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」)", + "[llm_rubric_response] LLM rubric 评审不达标(未通过 rubric:(未提供 rubric 明细)。)" + ], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "杭州是一座很不错的城市。", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "attribution": { + "counts_by_type": { + "wrong_tool_call": 2, + "wrong_tool_args": 2, + "knowledge_recall_miss": 2, + "format_violation": 2, + "llm_rubric_fail": 4, + "final_answer_mismatch": 2 + }, + "primary_by_case": { + "train_convert_3km": "wrong_tool_args", + "train_intro_shenzhen": "wrong_tool_call", + "val_convert_5km": "wrong_tool_args", + "val_intro_hangzhou": "wrong_tool_call" + }, + "details": { + "train_convert_3km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 3, \"unit\": \"km\"}),实际 convert_distance({\"value\": 3, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 3000, \"unit\": \"m\"}」,实际「3 公里等于 3000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "train_intro_shenzhen": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"深圳\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]」,实际「深圳是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_convert_5km": [ + { + "type": "wrong_tool_args", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 convert_distance({\"value\": 5, \"unit\": \"km\"}),实际 convert_distance({\"value\": 5, \"unit\": \"公里\"})", + "explanation": "工具选择正确,但调用参数与期望不一致" + }, + { + "type": "format_violation", + "metric": "final_response_avg_score", + "evidence": "期望「{\"result\": 5000, \"unit\": \"m\"}」,实际「5 公里等于 5000 米」", + "explanation": "格式不符合要求:期望结构化 JSON 输出,实际是自由文本" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ], + "val_intro_hangzhou": [ + { + "type": "wrong_tool_call", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "工具调用集合与期望不一致(缺少调用:knowledge_search)" + }, + { + "type": "knowledge_recall_miss", + "metric": "tool_trajectory_avg_score", + "evidence": "期望 knowledge_search({\"query\": \"杭州\"}),实际 (无调用)", + "explanation": "缺少知识检索调用(knowledge_search),无法召回作答所需知识" + }, + { + "type": "knowledge_recall_miss", + "metric": "llm_rubric_knowledge_recall", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "知识召回不足:检索结果无法支撑作答所需的关键信息" + }, + { + "type": "final_answer_mismatch", + "metric": "final_response_avg_score", + "evidence": "期望「杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]」,实际「杭州是一座很不错的城市。」", + "explanation": "最终回复与参考答案不匹配" + }, + { + "type": "llm_rubric_fail", + "metric": "llm_rubric_response", + "evidence": "未通过 rubric:(未提供 rubric 明细)。", + "explanation": "LLM rubric 评审不达标" + } + ] + } + }, + "optimization": { + "algorithm": "gepa_reflective", + "status": "SUCCEEDED", + "finish_reason": "completed", + "stop_reason": "score_threshold", + "total_rounds": 1, + "rounds_accepted": 1, + "optimizer_val_pass_rate": { + "baseline": 0.333333, + "best": 1.0 + }, + "best_prompts": { + "system_prompt": { + "sha256": "893aba6c9818ddb61576eb44da608017a852d744b8059767858d14f97d2b92a9", + "preview": "\n\n\n# 角色\n\n你是「城市信息助手 CityInfo」,负责回答三类问题:\n\n1. **距离换算**:把公里换算成米(调用 `convert_d…" + }, + "skill": { + "sha256": "7e8bd4de20170ef7abe8f695dabc76159cac4d2c50984a4f57858908774deb10", + "preview": "\n\n# 回答方法\n\n- 先判断问题属于哪一类(换算 / 介绍 / 身份),再决定是否调用工具。\n- 回答保持简洁,不要输出与问题无关的内容。" + } + }, + "cost": { + "total_llm_cost": 0.0, + "reflection_lm_calls": 1, + "budget_used": 12, + "budget_total": 60, + "token_usage": { + "prompt": 0, + "completion": 0, + "total": 0 + } + }, + "duration_seconds": 0.194, + "artifacts_dir": "optimize/" + }, + "candidate": { + "train": { + "pass_rate": 1.0, + "passed": 3, + "total": 3, + "mean_score": 1.0, + "metric_breakdown": { + "final_response_avg_score": 1.0, + "llm_rubric_knowledge_recall": 1.0, + "llm_rubric_response": 1.0, + "tool_trajectory_avg_score": 1.0 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 3, + "unit": "km" + } + } + ] + }, + "actual_response": "{\"result\": 3000, \"unit\": \"m\"}", + "expected_response": "{\"result\": 3000, \"unit\": \"m\"}" + }, + { + "eval_id": "train_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "train_intro_shenzhen", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "深圳" + } + } + ] + }, + "actual_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]", + "expected_response": "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]" + } + ] + }, + "val": { + "pass_rate": 1.0, + "passed": 3, + "total": 3, + "mean_score": 1.0, + "metric_breakdown": { + "final_response_avg_score": 1.0, + "llm_rubric_knowledge_recall": 1.0, + "llm_rubric_response": 1.0, + "tool_trajectory_avg_score": 1.0 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ], + "expected_tool_calls": [ + { + "name": "convert_distance", + "args": { + "value": 5, + "unit": "km" + } + } + ] + }, + "actual_response": "{\"result\": 5000, \"unit\": \"m\"}", + "expected_response": "{\"result\": 5000, \"unit\": \"m\"}" + }, + { + "eval_id": "val_identity", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [], + "expected_tool_calls": [] + }, + "actual_response": "我是城市信息助手 CityInfo。", + "expected_response": "我是城市信息助手 CityInfo。" + }, + { + "eval_id": "val_intro_hangzhou", + "passed": true, + "final_status": "PASSED", + "case_score": 1.0, + "metric_scores": { + "tool_trajectory_avg_score": 1.0, + "final_response_avg_score": 1.0, + "llm_rubric_response": 1.0, + "llm_rubric_knowledge_recall": 1.0 + }, + "metric_status": { + "tool_trajectory_avg_score": "PASSED", + "final_response_avg_score": "PASSED", + "llm_rubric_response": "PASSED", + "llm_rubric_knowledge_recall": "PASSED" + }, + "failure_types": [], + "failure_reasons": [], + "trajectory": { + "actual_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ], + "expected_tool_calls": [ + { + "name": "knowledge_search", + "args": { + "query": "杭州" + } + } + ] + }, + "actual_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]", + "expected_response": "杭州是一座以西湖和数字经济闻名的历史文化名城。 [source: city-guide]" + } + ] + } + }, + "delta": { + "train": { + "pass_rate_delta": 0.666667, + "score_delta": 0.5, + "counts": { + "new_pass": 2, + "new_fail": 0, + "score_up": 0, + "score_down": 0, + "unchanged": 1 + }, + "per_case": [ + { + "eval_id": "train_convert_3km", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.375, + "candidate_score": 1.0, + "change": "new_pass" + }, + { + "eval_id": "train_identity", + "baseline_passed": true, + "candidate_passed": true, + "baseline_score": 1.0, + "candidate_score": 1.0, + "change": "unchanged" + }, + { + "eval_id": "train_intro_shenzhen", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.125, + "candidate_score": 1.0, + "change": "new_pass" + } + ] + }, + "val": { + "pass_rate_delta": 0.666667, + "score_delta": 0.5, + "counts": { + "new_pass": 2, + "new_fail": 0, + "score_up": 0, + "score_down": 0, + "unchanged": 1 + }, + "per_case": [ + { + "eval_id": "val_convert_5km", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.375, + "candidate_score": 1.0, + "change": "new_pass" + }, + { + "eval_id": "val_identity", + "baseline_passed": true, + "candidate_passed": true, + "baseline_score": 1.0, + "candidate_score": 1.0, + "change": "unchanged" + }, + { + "eval_id": "val_intro_hangzhou", + "baseline_passed": false, + "candidate_passed": true, + "baseline_score": 0.125, + "candidate_score": 1.0, + "change": "new_pass" + } + ] + } + }, + "gate_decision": { + "accepted": true, + "reason": "全部闸门通过:验证集有实际提升、无退化、成本与耗时均在预算内,候选值得接受。", + "gates": [ + { + "name": "min_val_improvement", + "passed": true, + "detail": "验证集通过率提升 +0.6667(要求 ≥ 1e-09),平均分提升 +0.5000(要求 ≥ 0)" + }, + { + "name": "no_new_hard_fail", + "passed": true, + "detail": "验证集无新增失败 case" + }, + { + "name": "protected_cases", + "passed": true, + "detail": "保护 case(val_identity)均未退化" + }, + { + "name": "overfit_guard", + "passed": true, + "detail": "未触发过拟合守卫(train +0.6667 / val +0.6667)" + }, + { + "name": "cost_budget", + "passed": true, + "detail": "优化成本 $0.0000(预算 $1)" + }, + { + "name": "duration_budget", + "passed": true, + "detail": "pipeline 耗时 0.3s(预算 180s)" + } + ] + }, + "runtime": { + "pipeline_duration_seconds": 0.341, + "stage_durations": { + "baseline_eval": 0.096, + "attribution": 0.001, + "optimize": 0.2, + "candidate_regression": 0.045 + }, + "baseline_from_trace": false, + "fake_model_calls": { + "agent": 36, + "judge": 24, + "reflection": 1 + } + } +} diff --git a/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.md b/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.md new file mode 100644 index 00000000..dad21088 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/sample_output/success/optimization_report.md @@ -0,0 +1,70 @@ +# 优化报告 — 场景 `success` + +> 结论:✅ **接受候选 prompt** +> 理由:全部闸门通过:验证集有实际提升、无退化、成本与耗时均在预算内,候选值得接受。 + +- 生成时间:2026-07-06T15:49:26.322531+00:00 随机种子:42 报告 schema:v1 +- 优化算法:gepa_reflective(status=SUCCEEDED,1 轮,接受 1 轮,耗时 0.194s) +- 审计产物目录:`optimize/`(每轮候选 prompt、评测结果、接受理由、成本、seed 快照均在其中) + +## 一、baseline vs candidate 概览 + +| 切分 | baseline 通过率 | candidate 通过率 | baseline 平均分 | candidate 平均分 | 通过率 Δ | +| --- | --- | --- | --- | --- | --- | +| train | 33.3% (1/3) | 100.0% (3/3) | 0.500 | 1.000 | +0.667 | +| val | 33.3% (1/3) | 100.0% (3/3) | 0.500 | 1.000 | +0.667 | + +## 二、baseline 失败归因统计 + +| 失败类型 | 中文说明 | 涉及 case 数 | +| --- | --- | --- | +| `wrong_tool_call` | 工具调用错误 | 2 | +| `wrong_tool_args` | 工具参数错误 | 2 | +| `knowledge_recall_miss` | 知识召回不足 | 2 | +| `format_violation` | 格式不符合要求 | 2 | +| `llm_rubric_fail` | LLM rubric 不达标 | 4 | +| `final_answer_mismatch` | 最终回复不匹配 | 2 | + +主要归因(每个失败 case 的根因): +- `train_convert_3km` → `wrong_tool_args`(工具参数错误) +- `train_intro_shenzhen` → `wrong_tool_call`(工具调用错误) +- `val_convert_5km` → `wrong_tool_args`(工具参数错误) +- `val_intro_hangzhou` → `wrong_tool_call`(工具调用错误) + +## 三、逐 case delta(验证集为准,训练集附后) + +### val + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `val_convert_5km` | ❌ 0.375 | ✅ 1.000 | +0.625 | 新增通过(`new_pass`) | +| `val_identity` | ✅ 1.000 | ✅ 1.000 | +0.000 | 无变化(`unchanged`) | +| `val_intro_hangzhou` | ❌ 0.125 | ✅ 1.000 | +0.875 | 新增通过(`new_pass`) | + +### train + +| case | baseline | candidate | 分数变化 | 判定 | +| --- | --- | --- | --- | --- | +| `train_convert_3km` | ❌ 0.375 | ✅ 1.000 | +0.625 | 新增通过(`new_pass`) | +| `train_identity` | ✅ 1.000 | ✅ 1.000 | +0.000 | 无变化(`unchanged`) | +| `train_intro_shenzhen` | ❌ 0.125 | ✅ 1.000 | +0.875 | 新增通过(`new_pass`) | + +## 四、优化过程(优化器视角) + +- 优化器内部验证集通过率:33.3% → 100.0%(注意:优化器只看 optimizer.json 里的弱指标;overfit 场景中它看到的还是泄漏调参集 —— 是否真的变好以上面的独立验证集复评为准) +- 成本:$0.0000,反思 LM 调用 1 次,metric 调用 12/60 + +## 五、gate 决策明细 + +| 闸门 | 结果 | 说明 | +| --- | --- | --- | +| `min_val_improvement` | ✅ | 验证集通过率提升 +0.6667(要求 ≥ 1e-09),平均分提升 +0.5000(要求 ≥ 0) | +| `no_new_hard_fail` | ✅ | 验证集无新增失败 case | +| `protected_cases` | ✅ | 保护 case(val_identity)均未退化 | +| `overfit_guard` | ✅ | 未触发过拟合守卫(train +0.6667 / val +0.6667) | +| `cost_budget` | ✅ | 优化成本 $0.0000(预算 $1) | +| `duration_budget` | ✅ | pipeline 耗时 0.3s(预算 180s) | + +## 六、是否值得接受 + +候选 prompt 在独立验证集上带来实际提升,且未引入任何回归:无新增失败、保护 case 完好、成本与耗时都在预算内。**建议接受**,可用 `--apply` 将最优候选写回源 prompt 文件。 diff --git a/examples/optimization/eval_optimize_loop/tests/__init__.py b/examples/optimization/eval_optimize_loop/tests/__init__.py new file mode 100644 index 00000000..bc6e483f --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/__init__.py @@ -0,0 +1,5 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. diff --git a/examples/optimization/eval_optimize_loop/tests/test_attribution.py b/examples/optimization/eval_optimize_loop/tests/test_attribution.py new file mode 100644 index 00000000..1f585c19 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_attribution.py @@ -0,0 +1,249 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""失败归因单测:六类失败类型 × 各 ≥2 条合成样本(12/12 全对 → 准确率 100% ≥ 75%), +以及「真实 baseline 上每个失败 case 至少给出一个可解释原因」(验收标准 4)。 +""" + +from __future__ import annotations + +import asyncio +import sys +from pathlib import Path + +import pytest + +_HERE = Path(__file__).resolve().parent +_EXAMPLE_ROOT = _HERE.parent +_REPO_ROOT = _EXAMPLE_ROOT.parents[2] +for _p in (str(_REPO_ROOT), str(_EXAMPLE_ROOT)): + if _p not in sys.path: + sys.path.insert(0, _p) + +from loop_pipeline.attribution import attribute_case, cluster, primary_type # noqa: E402 +from loop_pipeline.evaluate import CaseEvalRecord, run_eval # noqa: E402 + + +def _record( + eval_id: str = "case", + *, + failed_metrics: dict[str, float], + actual_calls=(), + expected_calls=(), + actual_response: str = "", + expected_response: str = "", + rubric_verdicts=None, +) -> CaseEvalRecord: + """合成一条失败 case:failed_metrics 里的 metric 记 FAILED,其余 PASSED。""" + all_metrics = [ + "tool_trajectory_avg_score", + "final_response_avg_score", + "llm_rubric_response", + "llm_rubric_knowledge_recall", + ] + scores = {m: (failed_metrics.get(m, 1.0)) for m in all_metrics} + status = {m: ("FAILED" if m in failed_metrics else "PASSED") for m in all_metrics} + return CaseEvalRecord( + eval_id=eval_id, + passed=False, + final_status="FAILED", + case_score=sum(scores.values()) / len(scores), + metric_scores=scores, + metric_status=status, + metric_reasons={m: "synthetic" + for m in failed_metrics}, + rubric_verdicts=rubric_verdicts or {}, + actual_tool_calls=[{ + "name": n, + "args": a + } for n, a in actual_calls], + expected_tool_calls=[{ + "name": n, + "args": a + } for n, a in expected_calls], + actual_response=actual_response, + expected_response=expected_response, + ) + + +# 表驱动:12 条合成 case,每条标注期望的(主要归因, 必须包含的类型集合) +SYNTHETIC_CASES = [ + # --- wrong_tool_call ×2:漏调(非知识工具)/ 调错工具 --- + (_record("wtc_missing", + failed_metrics={"tool_trajectory_avg_score": 0.0}, + expected_calls=[("convert_distance", { + "value": 3, + "unit": "km" + })]), "wrong_tool_call", {"wrong_tool_call"}), + (_record("wtc_wrong_tool", + failed_metrics={"tool_trajectory_avg_score": 0.0}, + actual_calls=[("get_weather", { + "city": "上海" + })], + expected_calls=[("convert_distance", { + "value": 3, + "unit": "km" + })]), "wrong_tool_call", {"wrong_tool_call"}), + # --- wrong_tool_args ×2:名字对、参数错 --- + (_record("wta_unit", + failed_metrics={"tool_trajectory_avg_score": 0.0}, + actual_calls=[("convert_distance", { + "value": 3, + "unit": "公里" + })], + expected_calls=[("convert_distance", { + "value": 3, + "unit": "km" + })]), "wrong_tool_args", {"wrong_tool_args"}), + (_record("wta_value", + failed_metrics={"tool_trajectory_avg_score": 0.0}, + actual_calls=[("knowledge_search", { + "query": "北京天气" + })], + expected_calls=[("knowledge_search", { + "query": "北京" + })]), "wrong_tool_args", {"wrong_tool_args"}), + # --- knowledge_recall_miss ×2:召回 rubric 失败 / 漏调知识工具 --- + (_record("krm_rubric", + failed_metrics={"llm_rubric_knowledge_recall": 0.0}, + rubric_verdicts={"llm_rubric_knowledge_recall": [{ + "id": "k_guide", + "score": 0.0, + "reason": "无检索结果" + }]}), "knowledge_recall_miss", {"knowledge_recall_miss"}), + (_record("krm_missing_tool", + failed_metrics={"tool_trajectory_avg_score": 0.0}, + expected_calls=[("knowledge_search", { + "query": "深圳" + })]), "wrong_tool_call", {"wrong_tool_call", "knowledge_recall_miss"}), # 漏调知识工具 → 两类并报,主因是调用缺失 + # --- format_violation ×2:期望 JSON、实际自由文本 --- + (_record("fv_plain", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="3 公里等于 3000 米", + expected_response='{"result": 3000, "unit": "m"}'), "format_violation", {"format_violation"}), + (_record("fv_partial", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="结果是 {result: 5000", + expected_response='{"result": 5000, "unit": "m"}'), "format_violation", {"format_violation"}), + # --- llm_rubric_fail ×2 --- + (_record("lrf_json", + failed_metrics={"llm_rubric_response": 0.5}, + rubric_verdicts={ + "llm_rubric_response": [{ + "id": "r_json", + "score": 0.0, + "reason": "缺少 result 字段" + }, { + "id": "r_cite", + "score": 1.0, + "reason": "ok" + }] + }), "llm_rubric_fail", {"llm_rubric_fail"}), + (_record("lrf_cite", + failed_metrics={"llm_rubric_response": 0.0}, + rubric_verdicts={"llm_rubric_response": [{ + "id": "r_cite", + "score": 0.0, + "reason": "缺少来源标注" + }]}), "llm_rubric_fail", {"llm_rubric_fail"}), + # --- final_answer_mismatch ×2:两侧都是自由文本 --- + (_record("fam_text", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="深圳是一座很不错的城市。", + expected_response="深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]"), "final_answer_mismatch", + {"final_answer_mismatch"}), + (_record("fam_identity", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="根据以往训练经验,答案与训练样本一致。", + expected_response="我是城市信息助手 CityInfo。"), "final_answer_mismatch", {"final_answer_mismatch"}), +] + + +@pytest.mark.parametrize( + "record,expected_primary,expected_types", + SYNTHETIC_CASES, + ids=[case[0].eval_id for case in SYNTHETIC_CASES], +) +def test_six_failure_types_classified(record, expected_primary, expected_types): + """12/12 合成样本全部归类正确 → 分类准确率 100%(验收线 75%)。""" + findings = attribute_case(record) + assert findings, "失败 case 必须产出归因" + types = {f.type for f in findings} + assert expected_types <= types, f"{record.eval_id}: 期望包含 {expected_types},实际 {types}" + assert primary_type(findings) == expected_primary + for finding in findings: + assert finding.evidence, "每条归因必须带证据" + assert finding.explanation, "每条归因必须带中文可读解释" + + +def test_bare_scalar_expected_answer_is_not_format_violation(): + """期望答案是 JSON 裸标量('42'/'true' 等)→ final_answer_mismatch,而非格式违规。""" + for expected in ("42", "true", "3.14", '"plain"'): + record = _record("scalar_expected", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="回答内容不对", + expected_response=expected) + types = {f.type for f in attribute_case(record)} + assert "format_violation" not in types, f"expected={expected!r} 被误判为格式违规" + assert "final_answer_mismatch" in types, f"expected={expected!r}" + # JSON 对象/数组仍按结构化输出判定格式违规 + record = _record("array_expected", + failed_metrics={"final_response_avg_score": 0.0}, + actual_response="一、二、三", + expected_response="[1, 2, 3]") + assert "format_violation" in {f.type for f in attribute_case(record)} + + +def test_passed_case_yields_no_findings(): + record = _record("ok", failed_metrics={}) + record.passed = True + record.final_status = "PASSED" + assert attribute_case(record) == [] + + +def test_fallback_guarantees_explanation_for_unknown_metric(): + """规则未覆盖的 metric 失败也必须有兜底归因(隐藏样本适配性)。""" + record = CaseEvalRecord( + eval_id="custom_metric_case", + passed=False, + final_status="FAILED", + case_score=0.0, + metric_scores={"custom_business_metric": 0.0}, + metric_status={"custom_business_metric": "FAILED"}, + metric_reasons={"custom_business_metric": "业务指标未达标"}, + ) + findings = attribute_case(record) + assert len(findings) == 1 + assert findings[0].evidence == "业务指标未达标" + + +def test_cluster_counts_and_primary(): + records = {case[0].eval_id: case[0] for case in SYNTHETIC_CASES} + summary = cluster(records) + assert summary.counts["wrong_tool_call"] == 3 # wtc×2 + krm_missing_tool + assert summary.counts["knowledge_recall_miss"] == 2 + assert summary.counts["format_violation"] == 2 + assert set(summary.primary) == set(records) + assert all(summary.per_case[eid] for eid in records) + + +def test_every_failed_case_on_real_baseline_has_reason(): + """真实 baseline 跑一遍:所有 FAILED case 的归因均非空(验收标准 4 后半句)。""" + records = asyncio.run( + run_eval(str(_EXAMPLE_ROOT / "data" / "train.evalset.json"), str(_EXAMPLE_ROOT / "data" / "eval_config.json"))) + failed = [r for r in records.values() if not r.passed] + assert failed, "baseline 训练集应存在失败 case" + for record in failed: + findings = attribute_case(record) + assert findings, f"{record.eval_id} 缺少归因" + assert all(f.explanation and f.evidence for f in findings) + # 与 §2.2 设计矩阵对齐的抽查:train_convert_3km 的主因是工具参数错误 + convert_findings = attribute_case(records["train_convert_3km"]) + assert primary_type(convert_findings) == "wrong_tool_args" + types = {f.type for f in convert_findings} + assert {"wrong_tool_args", "format_violation", "llm_rubric_fail"} <= types + intro_findings = attribute_case(records["train_intro_shenzhen"]) + assert primary_type(intro_findings) == "wrong_tool_call" + assert "knowledge_recall_miss" in {f.type for f in intro_findings} diff --git a/examples/optimization/eval_optimize_loop/tests/test_fake_models.py b/examples/optimization/eval_optimize_loop/tests/test_fake_models.py new file mode 100644 index 00000000..c3411d58 --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_fake_models.py @@ -0,0 +1,273 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""fake 模型单测:judge 解析(用 SDK 真实模板构造消息)/ agent 指令路由 / reflection 提案。 + +judge 测试刻意通过 ``LLMJudge`` 的真实 ``DefaultMessagesConstructor`` 生成 +消息 —— SDK 若改动裁判模板(anchor 标签、rubric 行格式),这里会第一时间 +报警,而不是 e2e 神秘失败。 +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +_HERE = Path(__file__).resolve().parent +_EXAMPLE_ROOT = _HERE.parent +_REPO_ROOT = _EXAMPLE_ROOT.parents[2] +for _p in (str(_REPO_ROOT), str(_EXAMPLE_ROOT)): + if _p not in sys.path: + sys.path.insert(0, _p) + +from trpc_agent_sdk.evaluation._eval_case import IntermediateData, Invocation # noqa: E402 +from trpc_agent_sdk.evaluation._eval_metrics import EvalMetric # noqa: E402 +from trpc_agent_sdk.evaluation._llm_judge import LLMJudge, DefaultResponseScorer # noqa: E402 +from trpc_agent_sdk.models import ModelRegistry # noqa: E402 +from trpc_agent_sdk.models._llm_request import LlmRequest # noqa: E402 +from trpc_agent_sdk.types import Content, FunctionCall, FunctionResponse, GenerateContentConfig, Part # noqa: E402 + +from loop_agent.fake_models import ( # noqa: E402 + FALLBACK_ANSWER, MEMORIZE_MISS_ANSWER, FakeAgentModel, FakeJudgeModel, FakeReflectionModel, parse_directives, + register_fake_models, +) + +EVAL_CONFIG = json.loads((_EXAMPLE_ROOT / "data" / "eval_config.json").read_text(encoding="utf-8")) + + +def _metric(name: str) -> EvalMetric: + entry = next(m for m in EVAL_CONFIG["metrics"] if m["metric_name"] == name) + return EvalMetric(metric_name=name, threshold=entry["threshold"], criterion=entry["criterion"]) + + +def _invocation(query: str, answer: str, tool_calls=(), tool_responses=()) -> Invocation: + return Invocation( + user_content=Content(role="user", parts=[Part.from_text(text=query)]), + final_response=Content(role="model", parts=[Part.from_text(text=answer)]), + intermediate_data=IntermediateData( + tool_uses=[FunctionCall(id=f"c{i}", name=n, args=a) for i, (n, a) in enumerate(tool_calls)], + tool_responses=[ + FunctionResponse(id=f"c{i}", name=n, response=r) for i, (n, r) in enumerate(tool_responses) + ], + ), + ) + + +def _judge_verdicts(metric_name: str, invocation: Invocation) -> dict[str, str]: + """用 SDK 真实模板构造裁判消息 → fake judge 判定 → id→verdict。""" + judge = LLMJudge(_metric(metric_name)) + message = judge._messages_constructor.format_user_message([invocation], [invocation], judge._criterion, metric_name) + output = FakeJudgeModel.judge_message(message) + items = json.loads(output)["items"] + return {item["id"]: item["verdict"] for item in items} + + +class TestFakeJudge: + """fake judge 对真实模板消息的解析与条件规则。""" + + def test_rubric_response_json_case_fails_r_json(self): + # 用户要 JSON,回答是自由文本 → r_json no;未要求介绍 → r_cite 条件不适用 = yes + inv = _invocation("把 3 公里换算成米,用 JSON 输出", "3 公里等于 3000 米") + verdicts = _judge_verdicts("llm_rubric_response", inv) + assert verdicts == {"r_json": "no", "r_cite": "yes"} + + def test_rubric_response_json_case_passes_when_json(self): + inv = _invocation("把 3 公里换算成米,用 JSON 输出", '{"result": 3000, "unit": "m"}') + verdicts = _judge_verdicts("llm_rubric_response", inv) + assert verdicts == {"r_json": "yes", "r_cite": "yes"} + + def test_rubric_response_intro_case_requires_citation(self): + inv = _invocation("介绍一下深圳", "深圳是一座很不错的城市。") + assert _judge_verdicts("llm_rubric_response", inv)["r_cite"] == "no" + inv_ok = _invocation("介绍一下深圳", "深圳是一座以科技创新闻名的现代化滨海城市。 [source: city-guide]") + assert _judge_verdicts("llm_rubric_response", inv_ok)["r_cite"] == "yes" + + def test_knowledge_recall_conditional_rule(self): + # 未要求介绍 → 条件不适用 = yes(即使没有任何检索结果) + inv = _invocation("你叫什么名字?", "我是城市信息助手 CityInfo。") + assert _judge_verdicts("llm_rubric_knowledge_recall", inv) == {"k_guide": "yes"} + + def test_knowledge_recall_hits_and_misses(self): + intro = "介绍一下深圳" + with_tool = _invocation( + intro, + "深圳是… [source: city-guide]", + tool_calls=[("knowledge_search", { + "query": "深圳" + })], + tool_responses=[("knowledge_search", { + "source": "city-guide", + "summary": "深圳是…" + })], + ) + assert _judge_verdicts("llm_rubric_knowledge_recall", with_tool) == {"k_guide": "yes"} + without_tool = _invocation(intro, "深圳是一座很不错的城市。") + assert _judge_verdicts("llm_rubric_knowledge_recall", without_tool) == {"k_guide": "no"} + + def test_output_parsable_by_sdk_scorer(self): + """fake judge 的输出必须能被 SDK 的 DefaultResponseScorer 解析。""" + inv = _invocation("把 3 公里换算成米,用 JSON 输出", "3 公里等于 3000 米") + judge = LLMJudge(_metric("llm_rubric_response")) + message = judge._messages_constructor.format_user_message([inv], [inv], judge._criterion, "llm_rubric_response") + result = DefaultResponseScorer().parse_response(FakeJudgeModel.judge_message(message), "llm_rubric_response") + assert result.score == 0.5 # r_json no + r_cite yes + assert {r.id for r in result.rubric_scores} == {"r_json", "r_cite"} + + +def _agent_request(query: str, instruction: str, tool_response=None) -> LlmRequest: + contents = [Content(role="user", parts=[Part.from_text(text=query)])] + if tool_response is not None: + name, payload = tool_response + contents.append( + Content(role="user", + parts=[Part(function_response=FunctionResponse(id="call-1", name=name, response=payload))])) + request = LlmRequest(contents=contents, config=GenerateContentConfig()) + request.config.system_instruction = instruction + return request + + +async def _agent_reply(query: str, instruction: str, tool_response=None): + model = FakeAgentModel("fake-agent/test") + request = _agent_request(query, instruction, tool_response) + async for response in model._generate_async_impl(request): + return response.content.parts + raise AssertionError("model yielded nothing") + + +def _directives(**overrides) -> str: + values = {"output_format": "plain", "unit_normalization": "off", "knowledge": "off", "memorize": "off"} + values.update(overrides) + lines = "\n".join(f"{k}: {v}" for k, v in values.items()) + return f"\n# 角色\n城市信息助手" + + +BASELINE = _directives() +OPTIMIZED = _directives(output_format="json", unit_normalization="on", knowledge="on") +MEMORIZE = _directives(memorize="train_table") + + +class TestFakeAgentRouting: + """指令 DSL × 查询类型的路由表(表驱动核心组合)。""" + + def test_parse_directives_defaults_and_comments(self): + assert parse_directives("") == { + "output_format": "plain", + "unit_normalization": "off", + "knowledge": "off", + "memorize": "off", + } + parsed = parse_directives("") + assert parsed["output_format"] == "json" + assert parsed["knowledge"] == "on" + assert parsed["memorize"] == "off" + + async def test_convert_baseline_uses_raw_unit(self): + parts = await _agent_reply("把 3 公里换算成米,用 JSON 输出", BASELINE) + call = parts[0].function_call + assert call.name == "convert_distance" + assert call.args == {"value": 3, "unit": "公里"} + + async def test_convert_optimized_normalizes_unit_and_outputs_json(self): + parts = await _agent_reply("把 5 公里换算成米,用 JSON 输出", OPTIMIZED) + assert parts[0].function_call.args == {"value": 5, "unit": "km"} + parts = await _agent_reply("把 5 公里换算成米,用 JSON 输出", + OPTIMIZED, + tool_response=("convert_distance", { + "meters": 5000 + })) + assert parts[0].text == '{"result": 5000, "unit": "m"}' + + async def test_convert_baseline_plain_final(self): + parts = await _agent_reply("把 3 公里换算成米,用 JSON 输出", + BASELINE, + tool_response=("convert_distance", { + "error": "unsupported unit" + })) + assert parts[0].text == "3 公里等于 3000 米" + + async def test_intro_baseline_answers_from_memory(self): + parts = await _agent_reply("介绍一下杭州", BASELINE) + assert parts[0].text == "杭州是一座很不错的城市。" + + async def test_intro_optimized_searches_then_cites(self): + parts = await _agent_reply("介绍一下杭州", OPTIMIZED) + assert parts[0].function_call.name == "knowledge_search" + assert parts[0].function_call.args == {"query": "杭州"} + parts = await _agent_reply("介绍一下杭州", + OPTIMIZED, + tool_response=("knowledge_search", { + "source": "city-guide", + "summary": "杭州是一座名城。" + })) + assert parts[0].text == "杭州是一座名城。 [source: city-guide]" + + async def test_identity_route_is_directive_independent(self): + for instruction in (BASELINE, OPTIMIZED): + parts = await _agent_reply("你叫什么名字?", instruction) + assert parts[0].text == "我是城市信息助手 CityInfo。" + + async def test_unknown_query_falls_back(self): + parts = await _agent_reply("请自报家门", BASELINE) + assert parts[0].text == FALLBACK_ANSWER + + async def test_memorize_replays_table_hit(self): + parts = await _agent_reply("把 4 公里换算成米,用 JSON 输出", MEMORIZE) + assert parts[0].function_call.name == "convert_distance" + assert parts[0].function_call.args == {"value": 4, "unit": "km"} + parts = await _agent_reply("把 4 公里换算成米,用 JSON 输出", + MEMORIZE, + tool_response=("convert_distance", { + "meters": 4000 + })) + assert parts[0].text == '{"result": 4000, "unit": "m"}' + + async def test_memorize_miss_gives_wrong_answer_without_tools(self): + # 验证集问题不在查表里 → 错误答案且不调工具(过拟合在 val 上暴露的机制) + for query in ("把 5 公里换算成米,用 JSON 输出", "你叫什么名字?", "介绍一下杭州"): + parts = await _agent_reply(query, MEMORIZE) + assert parts[0].function_call is None + assert parts[0].text == MEMORIZE_MISS_ANSWER + + +class TestFakeReflection: + """reflection 提案器:按 prompt-field 标记返回对应候选并带 ``` 包裹。""" + + def _reflection_prompt(self, field_text: str) -> str: + return ("I provided an assistant with the following instructions:\n" + f"```\n{field_text}\n```\n\nExamples and feedback:\n```\ncase feedback...\n```\n" + "Write the new instruction within ``` blocks.") + + def test_returns_scenario_candidate_for_field(self): + system_text = (_EXAMPLE_ROOT / "loop_agent" / "prompts" / "system.md").read_text(encoding="utf-8") + out = FakeReflectionModel.propose(self._reflection_prompt(system_text), "success") + assert out.startswith("```\n") and out.endswith("\n```") + expected = (_EXAMPLE_ROOT / "candidates" / "system_prompt.success.md").read_text(encoding="utf-8").strip() + assert out[4:-4].strip() == expected + + def test_all_scenario_candidates_exist_and_keep_markers(self): + for field in ("system_prompt", "skill"): + for scenario in ("success", "no_effect", "overfit"): + path = _EXAMPLE_ROOT / "candidates" / f"{field}.{scenario}.md" + text = path.read_text(encoding="utf-8") + assert f"" in text, path + assert "```" not in text, f"{path} 不能包含三反引号(会破坏 gepa 提取)" + + def test_unknown_field_returns_current_text_noop(self): + prompt = self._reflection_prompt(" 当前路由 prompt 文本") + out = FakeReflectionModel.propose(prompt, "success") + assert out == "```\n 当前路由 prompt 文本\n```" + + +def test_registry_registration_is_idempotent(): + register_fake_models() + register_fake_models() + assert ModelRegistry.resolve("fake-agent/probe") is FakeAgentModel + assert ModelRegistry.resolve("fake-judge/probe") is FakeJudgeModel + assert ModelRegistry.resolve("fake-reflection/probe") is FakeReflectionModel + # create_model 路由(判官/反思 LM 的 provider_name 走的就是这条路径) + model = ModelRegistry.create_model("fake-reflection/success", api_key="", base_url="") + assert isinstance(model, FakeReflectionModel) + assert model.scenario == "success" diff --git a/examples/optimization/eval_optimize_loop/tests/test_gates.py b/examples/optimization/eval_optimize_loop/tests/test_gates.py new file mode 100644 index 00000000..79ed4d4a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_gates.py @@ -0,0 +1,209 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""接受策略(六道闸门)表驱动测试 —— 验收标准 2 的决策矩阵。 + +13 组合成场景,每组标注期望决策与期望的关键失败闸门;13/13 判定正确 +(决策准确率 100% ≥ 80%)。README 的「gate 决策规则表」引用本矩阵。 +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +_HERE = Path(__file__).resolve().parent +_EXAMPLE_ROOT = _HERE.parent +_REPO_ROOT = _EXAMPLE_ROOT.parents[2] +for _p in (str(_REPO_ROOT), str(_EXAMPLE_ROOT)): + if _p not in sys.path: + sys.path.insert(0, _p) + +from pydantic import ValidationError # noqa: E402 + +from loop_pipeline.config import GateConfig, PipelineConfig # noqa: E402 +from loop_pipeline.gates import evaluate_gates # noqa: E402 +from loop_pipeline.regression import CaseDelta, DeltaSummary, CHANGE_KINDS # noqa: E402 + + +def _delta(*cases: tuple[str, bool, bool, float, float]) -> DeltaSummary: + """(eval_id, baseline_passed, candidate_passed, baseline_score, candidate_score) → DeltaSummary。""" + summary = DeltaSummary(counts={kind: 0 for kind in CHANGE_KINDS}) + eps = 1e-6 + for eval_id, b_pass, c_pass, b_score, c_score in cases: + if not b_pass and c_pass: + change = "new_pass" + elif b_pass and not c_pass: + change = "new_fail" + elif c_score > b_score + eps: + change = "score_up" + elif c_score < b_score - eps: + change = "score_down" + else: + change = "unchanged" + summary.per_case.append( + CaseDelta(eval_id=eval_id, + baseline_passed=b_pass, + candidate_passed=c_pass, + baseline_score=b_score, + candidate_score=c_score, + change=change)) + summary.counts[change] += 1 + total = len(summary.per_case) + summary.pass_rate_delta = (sum(c.candidate_passed + for c in summary.per_case) - sum(c.baseline_passed + for c in summary.per_case)) / total + summary.score_delta = sum(c.candidate_score - c.baseline_score for c in summary.per_case) / total + return summary + + +IMPROVED_TRAIN = _delta(("t1", False, True, 0.5, 1.0), ("t2", True, True, 1.0, 1.0)) +FLAT_TRAIN = _delta(("t1", False, False, 0.5, 0.5), ("t2", True, True, 1.0, 1.0)) + +CHEAP = {"total_llm_cost": 0.01, "budget_used": 10, "duration_seconds": 5.0} + +# 决策矩阵:13/13 期望全部命中(决策准确率 100%,验收线 80%) +DECISION_MATRIX = [ + # (id, cfg覆盖, delta_val, delta_train, view, wall秒, 期望accept, 期望失败闸门) + ("clear_improvement_accept", {}, _delta(("v1", False, True, 0.4, 1.0), + ("v2", True, True, 1.0, 1.0)), IMPROVED_TRAIN, CHEAP, 10.0, True, None), + ("score_only_gain_accept", { + "min_val_pass_rate_improvement": 0.0 + }, _delta(("v1", False, False, 0.4, 0.8), ("v2", True, True, 1.0, 1.0)), FLAT_TRAIN, CHEAP, 10.0, True, None), + ("no_change_reject", {}, _delta( + ("v1", False, False, 0.4, 0.4), + ("v2", True, True, 1.0, 1.0)), FLAT_TRAIN, CHEAP, 10.0, False, "min_val_improvement"), + ("val_regression_reject", { + "overfit_guard": False + }, _delta(("v1", True, False, 1.0, 0.2), + ("v2", True, True, 1.0, 1.0)), FLAT_TRAIN, CHEAP, 10.0, False, "min_val_improvement"), + ("overfit_reject", {}, _delta(("v1", True, False, 1.0, 0.2), + ("v2", False, False, 0.3, 0.3)), IMPROVED_TRAIN, CHEAP, 10.0, False, "overfit_guard"), + ("new_hard_fail_despite_net_gain_reject", {}, + _delta(("v1", False, True, 0.2, 1.0), ("v2", False, True, 0.2, 1.0), + ("v3", True, False, 1.0, 0.4)), IMPROVED_TRAIN, CHEAP, 10.0, False, "no_new_hard_fail"), + ("protected_case_new_fail_reject", { + "protected_cases": ["v_key"], + "forbid_new_hard_fail": False + }, _delta(("v1", False, True, 0.2, 1.0), + ("v_key", True, False, 1.0, 0.4)), IMPROVED_TRAIN, CHEAP, 10.0, False, "protected_cases"), + ("protected_case_score_down_reject", { + "protected_cases": ["v_key"] + }, _delta(("v1", False, True, 0.2, 1.0), + ("v_key", True, True, 1.0, 0.9)), IMPROVED_TRAIN, CHEAP, 10.0, False, "protected_cases"), + ("cost_over_budget_reject", { + "max_cost_usd": 0.5 + }, _delta(("v1", False, True, 0.2, 1.0)), IMPROVED_TRAIN, { + "total_llm_cost": 0.75, + "budget_used": 10, + "duration_seconds": 5.0 + }, 10.0, False, "cost_budget"), + ("metric_calls_over_budget_reject", { + "max_metric_calls": 50 + }, _delta(("v1", False, True, 0.2, 1.0)), IMPROVED_TRAIN, { + "total_llm_cost": 0.01, + "budget_used": 61, + "duration_seconds": 5.0 + }, 10.0, False, "cost_budget"), + ("metric_calls_budget_untracked_reject", { + "max_metric_calls": 50 + }, _delta(("v1", False, True, 0.2, 1.0)), IMPROVED_TRAIN, { + "total_llm_cost": 0.01, + "budget_used": None, + "duration_seconds": 5.0 + }, 10.0, False, "cost_budget"), # fail-closed:配置了预算却无追踪数据 → 拒绝 + ("duration_over_budget_reject", { + "max_duration_seconds": 30.0 + }, _delta(("v1", False, True, 0.2, 1.0)), IMPROVED_TRAIN, CHEAP, 45.0, False, "duration_budget"), + ("non_protected_new_fail_allowed_when_disabled", { + "forbid_new_hard_fail": False, + "protected_cases": [] + }, _delta(("v1", False, True, 0.2, 1.0), ("v2", False, True, 0.2, 1.0), + ("v3", True, False, 1.0, 0.4)), IMPROVED_TRAIN, CHEAP, 10.0, True, None), +] + + +@pytest.mark.parametrize( + "case_id,cfg_overrides,delta_val,delta_train,view,wall,expect_accept,expect_failed_gate", + DECISION_MATRIX, + ids=[row[0] for row in DECISION_MATRIX], +) +def test_gate_decision_matrix(case_id, cfg_overrides, delta_val, delta_train, view, wall, expect_accept, + expect_failed_gate): + cfg = GateConfig(**cfg_overrides) + decision = evaluate_gates(cfg, + delta_val=delta_val, + delta_train=delta_train, + optimize_result_view=view, + wall_seconds=wall) + assert decision.accepted is expect_accept, f"{case_id}: {decision.reason}" + assert len(decision.gates) == 6 + if expect_failed_gate is not None: + failed_names = {g.name for g in decision.gates if not g.passed} + assert expect_failed_gate in failed_names + assert not decision.accepted + assert decision.reason # 决策必须带中文理由 + + +def test_overfit_reason_mentions_overfitting(): + """过拟合场景的拒绝理由必须点名「过拟合」(即使同时触发别的闸门)。""" + decision = evaluate_gates( + GateConfig(protected_cases=["v1"]), + delta_val=_delta(("v1", True, False, 1.0, 0.2), ("v2", False, False, 0.3, 0.3)), + delta_train=IMPROVED_TRAIN, + optimize_result_view=CHEAP, + wall_seconds=10.0, + ) + assert not decision.accepted + assert "过拟合" in decision.reason + # 同时触发的其它闸门也在理由里提示 + assert "同时未通过" in decision.reason + + +def test_cost_budget_fails_closed_when_budget_untracked(): + """配置了 max_metric_calls 但 budget_used 缺失 → fail-closed 拒绝并写明原因。""" + view = {"total_llm_cost": 0.01, "budget_used": None, "duration_seconds": 5.0} + decision = evaluate_gates(GateConfig(max_metric_calls=50), + delta_val=_delta(("v1", False, True, 0.2, 1.0)), + delta_train=IMPROVED_TRAIN, + optimize_result_view=view, + wall_seconds=1.0) + gate = next(g for g in decision.gates if g.name == "cost_budget") + assert gate.passed is False and decision.accepted is False + assert "预算追踪不可用" in gate.detail + # 未配置 max_metric_calls 时,budget_used 缺失不影响通过 + decision2 = evaluate_gates(GateConfig(), + delta_val=_delta(("v1", False, True, 0.2, 1.0)), + delta_train=IMPROVED_TRAIN, + optimize_result_view=view, + wall_seconds=1.0) + assert next(g for g in decision2.gates if g.name == "cost_budget").passed is True + + +def test_config_rejects_typo_gate_keys(tmp_path): + """闸门配置 fail-fast:写错闸门名必须报错,而不是静默用默认阈值。""" + with pytest.raises(ValidationError): + GateConfig(min_val_pass_rate_improvment=0.5) # 拼写错误的字段名 + bad = tmp_path / "pipeline.bad.json" + bad.write_text('{"gates": {"min_val_pass_rate_improvment": 0.5}}', encoding="utf-8") + with pytest.raises(ValidationError): + PipelineConfig.load(bad) + with pytest.raises(ValidationError): + PipelineConfig.model_validate({"unknown_top_level": 1}) + + +def test_accept_reason_is_positive(): + decision = evaluate_gates( + GateConfig(), + delta_val=_delta(("v1", False, True, 0.2, 1.0)), + delta_train=FLAT_TRAIN, + optimize_result_view=CHEAP, + wall_seconds=1.0, + ) + assert decision.accepted + assert "全部闸门通过" in decision.reason + assert all(g.passed for g in decision.gates) diff --git a/examples/optimization/eval_optimize_loop/tests/test_pipeline_e2e.py b/examples/optimization/eval_optimize_loop/tests/test_pipeline_e2e.py new file mode 100644 index 00000000..d223359a --- /dev/null +++ b/examples/optimization/eval_optimize_loop/tests/test_pipeline_e2e.py @@ -0,0 +1,348 @@ +# Tencent is pleased to support the open source community by making tRPC-Agent-Python available. +# +# Copyright (C) 2026 Tencent. All rights reserved. +# +# tRPC-Agent-Python is licensed under Apache-2.0. +"""三场景端到端测试:报告契约(AC1/AC6)、过拟合必拒(AC3)、时限(AC5)、确定性。 + +三个场景在 module 级 fixture 里各跑一次并计时(AC5 直接用这份计时断言), +其余测试消费同一份结果 —— 既贴近 `--scenario all` 的真实用法,又避免重复 +跑 pipeline 拖慢测试。fixture 会快照并恢复 prompt 源文件,任何用例失败都 +不会把候选 prompt 留在工作区。 +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import subprocess +import sys +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest + +_HERE = Path(__file__).resolve().parent +_EXAMPLE_ROOT = _HERE.parent +_REPO_ROOT = _EXAMPLE_ROOT.parents[2] +for _p in (str(_REPO_ROOT), str(_EXAMPLE_ROOT)): + if _p not in sys.path: + sys.path.insert(0, _p) + +from trpc_agent_sdk.evaluation._agent_evaluator import _EvaluationCasesFailed # noqa: E402 + +import run_pipeline # noqa: E402 +from loop_pipeline import evaluate as evaluate_module # noqa: E402 +from loop_pipeline.attribution import cluster # noqa: E402 +from loop_pipeline.evaluate import run_eval # noqa: E402 +from loop_pipeline.report import REQUIRED_TOP_LEVEL_KEYS, validate_report # noqa: E402 + +PROMPT_FILES = ( + _EXAMPLE_ROOT / "loop_agent" / "prompts" / "system.md", + _EXAMPLE_ROOT / "loop_agent" / "prompts" / "skill.md", +) + +ALL_EVAL_IDS = { + "train_convert_3km", + "train_intro_shenzhen", + "train_identity", + "val_convert_5km", + "val_identity", + "val_intro_hangzhou", +} + + +@pytest.fixture(autouse=True) +def _restore_prompts(): + """快照 + 恢复 prompt 源文件:测试永不污染工作区。""" + snapshot = {path: path.read_bytes() for path in PROMPT_FILES} + yield + for path, content in snapshot.items(): + if path.read_bytes() != content: + path.write_bytes(content) + + +@pytest.fixture(scope="module") +def all_scenarios(tmp_path_factory): + """顺序跑三场景(同 --scenario all),返回 {场景: (报告, 输出目录)} 与总耗时。""" + snapshot = {path: path.read_bytes() for path in PROMPT_FILES} + output_root = tmp_path_factory.mktemp("runs") + reports: dict[str, dict] = {} + started = time.monotonic() + try: + for scenario in ("success", "no_effect", "overfit"): + reports[scenario] = asyncio.run(run_pipeline.run_scenario(scenario, output_root, quiet=True)) + finally: + for path, content in snapshot.items(): + if path.read_bytes() != content: + path.write_bytes(content) + elapsed = time.monotonic() - started + return {"reports": reports, "elapsed": elapsed, "output_root": output_root} + + +def _report_dir(output_root: Path, scenario: str) -> Path: + dirs = sorted(output_root.glob(f"{scenario}-*")) + assert dirs, f"未找到 {scenario} 的输出目录" + return dirs[-1] + + +# --------------------------------------------------------------------------- +# AC1:六条 case 全部可运行并产出完整报告;AC6:报告字段契约 +# --------------------------------------------------------------------------- + + +def test_success_scenario_end_to_end(all_scenarios): + report = all_scenarios["reports"]["success"] + out_dir = _report_dir(all_scenarios["output_root"], "success") + + # 报告文件 + 审计产物齐全 + assert (out_dir / "optimization_report.json").is_file() + assert (out_dir / "optimization_report.md").is_file() + assert (out_dir / "baseline_eval.json").is_file() + assert (out_dir / "candidate_eval.json").is_file() + assert (out_dir / "attribution.json").is_file() + assert (out_dir / "pipeline_config.snapshot.json").is_file() + optimize_dir = out_dir / "optimize" + assert (optimize_dir / "result.json").is_file() + assert (optimize_dir / "config.snapshot.json").is_file() + assert list((optimize_dir / "rounds").glob("round_*.json")), "每轮审计记录缺失" + assert list((optimize_dir / "best_prompts").glob("*.md")), "最优候选 prompt 快照缺失" + assert list((optimize_dir / "baseline_prompts").glob("*.md")), "baseline prompt 快照缺失" + + # 6 条公开 case 全部出现在 baseline 明细里 + seen = {case["eval_id"] for split in ("train", "val") for case in report["baseline"][split]["per_case"]} + assert seen == ALL_EVAL_IDS + + # 优化成功且被接受 + assert report["optimization"]["status"] == "SUCCEEDED" + decision = report["gate_decision"] + assert decision["accepted"] is True + assert all(g["passed"] for g in decision["gates"]) + # 独立验证集:2 条 new_pass(convert + intro),保护 case 不动 + assert report["delta"]["val"]["counts"]["new_pass"] == 2 + assert report["delta"]["val"]["counts"]["new_fail"] == 0 + assert report["delta"]["val"]["pass_rate_delta"] == pytest.approx(2 / 3) + assert report["delta"]["train"]["pass_rate_delta"] == pytest.approx(2 / 3) + + +def test_report_contract(all_scenarios): + """AC6:三份报告全部满足字段契约;md 含关键章节;sample_output 同样校验。""" + for scenario, report in all_scenarios["reports"].items(): + assert validate_report(report) == [], scenario + for key in REQUIRED_TOP_LEVEL_KEYS: + assert key in report, f"{scenario} 缺少 {key}" + # baseline / candidate 分数、逐 case delta、gate 决策、接受/拒绝理由(AC6 原文点名) + assert isinstance(report["baseline"]["val"]["pass_rate"], float) + assert isinstance(report["candidate"]["val"]["pass_rate"], float) + assert report["delta"]["val"]["per_case"], scenario + assert isinstance(report["gate_decision"]["accepted"], bool) + assert report["gate_decision"]["reason"] + + md_path = _report_dir(all_scenarios["output_root"], scenario) / "optimization_report.md" + md = md_path.read_text(encoding="utf-8") + for keyword in ("baseline", "candidate", "失败归因", "逐 case delta", "gate 决策", "是否值得接受", "理由"): + assert keyword in md, f"{scenario} 的 md 缺少「{keyword}」章节" + + # 提交在仓库里的 sample_output 与运行时报告遵守同一契约 + for sample in sorted((_EXAMPLE_ROOT / "sample_output").glob("*/optimization_report.json")): + report = json.loads(sample.read_text(encoding="utf-8")) + assert validate_report(report) == [], sample + + +# --------------------------------------------------------------------------- +# 交付物:优化无效场景(REJECT) +# --------------------------------------------------------------------------- + + +def test_no_effect_scenario_rejected(all_scenarios): + report = all_scenarios["reports"]["no_effect"] + decision = report["gate_decision"] + assert decision["accepted"] is False + assert "提升不足" in decision["reason"] or "min_val_improvement" in decision["reason"] + counts = report["delta"]["val"]["counts"] + assert counts["unchanged"] == 3 and counts["new_pass"] == 0 and counts["new_fail"] == 0 + assert report["delta"]["val"]["pass_rate_delta"] == pytest.approx(0.0) + + +# --------------------------------------------------------------------------- +# AC3:过拟合(train 提升、val 退化)必须拒绝 +# --------------------------------------------------------------------------- + + +def test_overfit_scenario_rejected(all_scenarios): + report = all_scenarios["reports"]["overfit"] + + # 优化器视角一路变好(它看到的是泄漏的 probe 集)…… + opt_view = report["optimization"]["optimizer_val_pass_rate"] + assert opt_view["best"] > opt_view["baseline"] + # ……但独立数据集复评:train 提升、val 退化 + assert report["delta"]["train"]["pass_rate_delta"] > 0 + assert report["delta"]["val"]["pass_rate_delta"] < 0 + + decision = report["gate_decision"] + assert decision["accepted"] is False + assert "过拟合" in decision["reason"] + gates = {g["name"]: g["passed"] for g in decision["gates"]} + assert gates["overfit_guard"] is False + assert gates["protected_cases"] is False # 保护 case val_identity 退化 + assert gates["no_new_hard_fail"] is False + + # 保护 case val_identity 是 new_fail + val_changes = {c["eval_id"]: c["change"] for c in report["delta"]["val"]["per_case"]} + assert val_changes["val_identity"] == "new_fail" + + +# --------------------------------------------------------------------------- +# AC5:fake/trace 模式下完整 pipeline ≤ 3 分钟;trace 模式路径可用 +# --------------------------------------------------------------------------- + + +def test_all_scenarios_under_time_budget(all_scenarios): + assert all_scenarios["elapsed"] < 180.0, f"三场景总耗时 {all_scenarios['elapsed']:.1f}s 超过 3 分钟" + for report in all_scenarios["reports"].values(): + assert report["runtime"]["pipeline_duration_seconds"] < 180.0 + + +def test_trace_mode_baseline(tmp_path): + """trace 模式:不执行 agent,直接对预录轨迹评测并归因。""" + records = asyncio.run( + run_eval(str(_EXAMPLE_ROOT / "data" / "trace_baseline.evalset.json"), + str(_EXAMPLE_ROOT / "data" / "eval_config.json"), + agent_module=None)) + assert set(records) == {"trace_convert_3km", "trace_intro_shenzhen"} + assert all(not r.passed for r in records.values()) + summary = cluster(records) + assert summary.primary["trace_convert_3km"] == "wrong_tool_args" + assert summary.primary["trace_intro_shenzhen"] == "wrong_tool_call" + assert "knowledge_recall_miss" in {f.type for f in summary.per_case["trace_intro_shenzhen"]} + + # --baseline-from-trace 的 CLI 路径:trace 评测 + 归因明细随场景一起落盘 + report = asyncio.run(run_pipeline.run_scenario("success", tmp_path, baseline_from_trace=True, quiet=True)) + assert report["runtime"]["baseline_from_trace"] is True + out_dir = sorted(tmp_path.glob("success-*"))[-1] + trace_eval = json.loads((out_dir / "trace_eval.json").read_text(encoding="utf-8")) + assert set(trace_eval) == {"trace_convert_3km", "trace_intro_shenzhen"} + trace_attr = json.loads((out_dir / "trace_attribution.json").read_text(encoding="utf-8")) + assert set(trace_attr) == {"trace_convert_3km", "trace_intro_shenzhen"} + assert all(f["type"] and f["explanation"] for findings in trace_attr.values() for f in findings) + assert {f["type"] for f in trace_attr["trace_intro_shenzhen"]} >= {"wrong_tool_call", "knowledge_recall_miss"} + + +# --------------------------------------------------------------------------- +# run_eval 只吞框架的 _EvaluationCasesFailed;真实断言失败必须冒出来 +# --------------------------------------------------------------------------- + + +def test_run_eval_only_swallows_eval_cases_failed(monkeypatch): + + class _FakeEvaluator: + + exc: Exception = AssertionError("genuine third-party assertion") + + @staticmethod + def get_executer(*_args, **_kwargs): + + class _Executer: + + async def evaluate(self): + raise _FakeEvaluator.exc + + def get_result(self): + return SimpleNamespace(results_by_eval_set_id={}) + + return _Executer() + + monkeypatch.setattr(evaluate_module, "AgentEvaluator", _FakeEvaluator) + with pytest.raises(AssertionError, match="genuine third-party assertion"): + asyncio.run(run_eval("ds.json", "cfg.json")) + + # 框架的 _EvaluationCasesFailed(case 失败信号)仍被吞掉,照常取结果 + _FakeEvaluator.exc = _EvaluationCasesFailed("cases failed") + assert asyncio.run(run_eval("ds.json", "cfg.json")) == {} + + +# --------------------------------------------------------------------------- +# --check:README 命令必须 cwd 无关;文件缺失给友好错误而不是 traceback +# --------------------------------------------------------------------------- + + +def test_check_command_is_cwd_independent(): + script = _EXAMPLE_ROOT / "run_pipeline.py" + ok = subprocess.run( + [sys.executable, str(script), "--check", "sample_output/success/optimization_report.json"], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + ) + assert ok.returncode == 0, ok.stderr + assert "报告契约校验通过" in ok.stdout + + missing = subprocess.run( + [sys.executable, str(script), "--check", "no/such/report.json"], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + ) + assert missing.returncode == 1 + assert "Traceback" not in (missing.stdout + missing.stderr) + assert "报告文件不存在" in missing.stderr + + +# --------------------------------------------------------------------------- +# --apply --scenario all:写回延后到全部场景结束,后续场景 baseline 不被污染 +# --------------------------------------------------------------------------- + + +def test_apply_with_scenario_all_defers_write_until_all_done(tmp_path): + baseline_system = PROMPT_FILES[0].read_text(encoding="utf-8") + args = argparse.Namespace(scenario="all", + output=str(tmp_path), + baseline_from_trace=False, + apply=True, + quiet=True, + check=None) + assert asyncio.run(run_pipeline._amain(args)) == 0 + # 全部结束后:success 被接受,最优候选已写回(源文件发生变化) + assert PROMPT_FILES[0].read_text(encoding="utf-8") != baseline_system + # 后跑的 no_effect / overfit 的 baseline 仍是干净 baseline(val 通过率 1/3), + # 若 success 的写回发生在场景循环中,这里会被污染成更高的通过率 + for scenario in ("no_effect", "overfit"): + out_dir = sorted(tmp_path.glob(f"{scenario}-*"))[-1] + report = json.loads((out_dir / "optimization_report.json").read_text(encoding="utf-8")) + assert report["baseline"]["val"]["pass_rate"] == pytest.approx(1 / 3), scenario + + +# --------------------------------------------------------------------------- +# 确定性:同输入两次运行,决策与逐 case delta 完全一致 +# --------------------------------------------------------------------------- + + +def test_deterministic_reruns(all_scenarios, tmp_path): + first = all_scenarios["reports"]["success"] + second = asyncio.run(run_pipeline.run_scenario("success", tmp_path, quiet=True)) + + def stable_view(report: dict) -> dict: + return { + "delta": report["delta"], + "accepted": report["gate_decision"]["accepted"], + "gates": [(g["name"], g["passed"]) for g in report["gate_decision"]["gates"]], + "baseline_pass": { + s: report["baseline"][s]["pass_rate"] + for s in ("train", "val") + }, + "candidate_pass": { + s: report["candidate"][s]["pass_rate"] + for s in ("train", "val") + }, + } + + assert stable_view(first) == stable_view(second) + + +def test_prompts_untouched_after_runs(all_scenarios): + """三场景跑完后源 prompt 仍是 baseline(write_all 快照恢复生效)。""" + system_text = PROMPT_FILES[0].read_text(encoding="utf-8") + assert "output_format: plain" in system_text + assert "memorize: off" in system_text