Feature/eval optimize loop

.gitignore

@@ -24,3 +24,9 @@ test-ngtest-ut-trpc-agent-py.xml
 node_modules
 package-lock.json
 pyrightconfig.json
+
+# Generated artifacts for the evaluation/optimization loop example.
+examples/optimization/eval_optimize_loop/runs/
+
+# Local isolated development worktrees.
+/.worktrees/

analysis/evaluation optimization pipeline/实施阶段路线图.md

@@ -0,0 +1,125 @@
+# Evaluation + Optimization Pipeline 实施阶段路线图
+
+本文记录完成 Evaluation + Optimization 自动回归与提示词优化闭环所需的实施阶段。每个阶段应形成一组可以独立测试、评审和提交的能力，规模大致与当前前两次实现提交相当。整体沿用 1–6 的主阶段编号，其中范围较大的阶段 3 拆成两个独立提交单元。
+
+状态说明：
+
+- `[x]` 已完成
+- `[ ]` 待完成
+
+## 阶段总览
+
+| 阶段 | 状态 | 核心目标 |
+|---|---|---|
+| 1. 输入准备与 Prompt 工作区 | 已完成 | 校验配置和输入，创建可复现、隔离的 Prompt 工作副本 |
+| 2. 确定性离线评测闭环 | 已完成 | 使用 fake agent/provider 完成 baseline 和 candidate 的四次完整评测 |
+| 3a. 评测标准化、失败归因与 Case Diff | 已完成 | 统一评测数据，解释失败并比较候选变化 |
+| 3b. 独立 Gate | 已完成 | 根据 diff、关键 case 和预算规则给出接受或拒绝决策 |
+| 4. 真实优化器与安全写回 | 待完成 | 接入 AgentOptimizer，审计候选，并在 Gate 接受后安全更新源 Prompt |
+| 5. 报告、产物与资源观测 | 待完成 | 输出结构化报告、可读报告和完整审计产物 |
+| 6. 离线模式与端到端验收 | 待完成 | 补齐 fake judge/trace 场景、示例输出、文档和完整验收测试 |
+
+## 1. 输入准备与 Prompt 工作区
+
+**状态：`[x]` 已完成**
+
+- 建立示例目录、`pipeline.json`、`optimizer.json`、train/validation evalset 和 baseline Prompt。
+- 校验配置、路径、评测集、metric、case 标签和运行参数。
+- 保存输入文件及 Prompt 的内容和哈希快照。
+- 将源 Prompt 复制到独立 run 工作区，区分 `source_target` 和 `working_target`。
+- 保证准备失败时不留下伪完整运行目录，且准备阶段不修改源 Prompt。
+- 提供阶段一单元测试和命令行入口。
+
+对应提交：`6c47ddd feat: 新增评测优化闭环准备阶段`
+
+## 2. 确定性离线评测闭环
+
+**状态：`[x]` 已完成**
+
+- 实现每次调用都重新读取工作 Prompt 的确定性 fake agent。
+- 实现 `improve`、`no_improvement`、`overfit` 三种 fake candidate。
+- 对 baseline 和 candidate 分别执行 train、validation 完整评测。
+- 保存 SDK 原始评测结果、通过数量、平均分和候选 Prompt 元数据。
+- 校验 evalset 和工作 Prompt 在准备后没有漂移，并保留 candidate 工作副本供审计。
+- 提供三种场景矩阵、异常路径和 CLI smoke test。
+
+对应提交：`115c914 feat(evaluation): 实现确定性评测优化闭环第二阶段`
+
+## 3a. 评测标准化、失败归因与 Case Diff
+
+**状态：`[x]` 已完成**
+
+- 将 SDK 评测结果转换为稳定的逐 case、逐 metric 数据模型。
+- 根据 metric、预期/实际回复和调用轨迹执行确定性失败归因。
+- 比较 baseline 与 candidate，识别新增通过、新增失败、提升、退化和不变 case。
+- 标记 hard、critical 和 severe regression，并识别训练提升但验证退化的过拟合。
+- 保留归因依据和前后变化证据，供 Gate 与报告阶段直接消费。
+
+阶段完成标准：可以仅根据四次评测结果生成可序列化、可解释的逐 case 差异，且不修改源 Prompt。
+
+## 3b. 独立 Gate
+
+**状态：`[x]` 已完成**
+
+- 实现与优化器解耦的 Gate，消费阶段 3a 生成的评测与 diff 数据。
+- 执行验证集最小提升、通过率不得下降、hard/critical case、severe regression 和必需 metric 规则。
+- 纳入成本、token、耗时及不可观测数据策略，但不将未知数据误记为零。
+- 收集全部拒绝理由，不采用只返回第一个错误的方式。
+- 固定三种离线场景的决策：improve ACCEPT、no improvement REJECT、overfit REJECT。
+- 覆盖多规则同时失败、关键 case 退化、训练提升但验证下降等集成场景。
+
+阶段完成标准：Gate 能稳定给出完整决策证据，且 ACCEPT/REJECT 本身不触发源 Prompt 写回。
+
+## 4. 真实优化器与安全写回
+
+**状态：`[ ]` 待完成**
+
+- 抽象统一 Candidate Provider 接口，接入 `AgentOptimizer` 真实候选生成。
+- 始终使用 `update_source=False`，让优化器只操作隔离工作区。
+- 保留优化器原生候选、轮次记录、分数和配置快照。
+- 对最终候选重新执行完整 train/validation 回归，并交给统一 diff 和 Gate。
+- 只有 Gate ACCEPT 且源 Prompt 哈希未变化时才允许写回；写回后必须回读校验。
+- REJECT、异常、源文件漂移或校验失败时保持源 Prompt 不变。
+
+阶段完成标准：真实模式和 fake 模式共用同一条候选验证及 Gate 链路，写回行为具备明确的安全边界。
+
+## 5. 报告、产物与资源观测
+
+**状态：`[ ]` 待完成**
+
+- 生成 `optimization_report.json`，包含 baseline、candidate、归因、case diff、Gate 和写回状态。
+- 生成面向使用者的 `optimization_report.md`，解释候选是否值得接受及具体原因。
+- 建立 artifact index，索引输入快照、四次评测、候选 Prompt、优化器原生产物和报告。
+- 记录随机种子、配置、耗时以及可观测的 token、成本和调用信息。
+- 对无法可靠观测的资源数据明确记录为 `unavailable`，并按预算策略产生 reject 或 warning。
+- 使用原子写入，避免失败运行留下看似完整的报告和索引。
+
+阶段完成标准：一次运行的输入、Prompt 变化、评测证据、决策和写回结果都可以从产物中复现和审计。
+
+## 6. 离线模式与端到端验收
+
+**状态：`[ ]` 待完成**
+
+必须完成：
+
+- 保持无 API Key 时可以运行 improve、no improvement 和 overfit 三个完整场景。
+- 如果保留 `use_fake_judge` 配置，则实现确定性 fake judge；如果保留 `trace` 模式，则让它能够驱动归因、diff、Gate 和报告链路。
+- 提供示例输出、完整 README、运行命令和各模式适用边界。
+- 覆盖 evaluator/optimizer 异常、Gate 拒绝、写回失败、输入漂移和产物不完整等端到端路径。
+- 验证离线完整 pipeline 在三分钟内完成，并核对 issue 要求的报告字段和交付物。
+
+可选且默认跳过：
+
+- 通过显式环境变量启用真实 API 集成测试。
+- 真实 API 测试不作为普通 CI 或无 API Key 核心流程的必要条件。
+
+阶段完成标准：公开样例能够稳定生成完整报告和正确决策，项目具备提交 issue 验收所需的文档、测试和审计产物。
+
+## 全局实施约束
+
+- Baseline 和最终 Candidate 都必须分别执行完整 train、validation 评测。
+- 真实优化器的内部 minibatch 或轮次分数不能替代 pipeline 的完整回归。
+- Gate 决策前不得修改源 Prompt，任何写回都必须经过源哈希校验和回读验证。
+- fake agent/provider/judge 必须保持确定性，不得读取 `eval_id`、期望答案或调用次数作弊。
+- 所有阶段优先提供无 API Key 的自动化测试，并保留后续真实模式的清晰接口。
+- 新增数据模型和产物需要保持可序列化、可解释和可审计。

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -0,0 +1,9 @@
+# 设计边界
+
+本示例把运行环境与替代组件分开描述。`offline`、`real`、`trace` 是三种 Pipeline 运行模式。`offline` 仍使用 SDK 的 `LlmAgent`、Runner 和独立 Session，只把 Agent 内部模型替换成 `DeterministicFakeModel`，并用确定性 Candidate Provider 代替真实优化器。它用于验证 Prompt 改变是否经过真实 Agent 编排影响输出。
+
+`real` 使用真实业务模型生成回复，并由 `AgentOptimizer` 调用真实反思模型产生 Prompt 候选；只有 Gate 接受、源 Prompt 哈希未漂移且显式启用写回时，才允许更新源文件。
+
+`trace` 直接评测预录制的 `actual_conversation`，不再次运行 Agent、Model 或 Candidate Provider。它适合复现工具轨迹和生产故障，但只能证明候选版本与轨迹的关联，不能证明 Prompt 导致了该轨迹。因此 Trace 即使获得 ACCEPT，也固定跳过源 Prompt 写回。
+
+确定性 metric 负责精确匹配等硬规则。LLM Judge 若需要，应作为带 rubric 的评测指标显式配置；本示例不提供容易混淆职责的 `use_fake_judge` 开关。

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,162 @@
+# Evaluation + Optimization Loop — Stage 6
+
+## 三种运行模式
+
+- `offline`：使用真实 SDK `LlmAgent` 和 `Runner`，Agent 内部注入 `DeterministicFakeModel`，候选由确定性 Candidate Provider 生成。无需 API Key。
+- `trace`：使用 SDK 原生 `eval_mode="trace"` 回放 baseline/candidate 的 train 与 validation 轨迹，不运行 Agent、Model 或 Candidate Provider。即使 Gate 接受，也不会写回源 Prompt。
+- `real`：业务 Agent 和反思优化模型均使用真实模型，由显式入口 `run_real_integration.py --run-real` 启用。
+
+Fake Model 是 `offline` 模式中的模型实现，不是独立运行模式；Fake Candidate Provider 只替代候选生成，不替代业务 Agent。精确匹配等硬规则由 `optimizer.json` 的确定性 metric 执行。需要语义判断时，应显式配置带 rubric 的 LLM Judge，而不是使用布尔 Fake Judge 开关。详细边界见 [DESIGN.md](DESIGN.md)。
+
+This example provides an auditable evaluation and prompt-optimization loop.
+Stages 1–3 prepare an isolated prompt workspace, run baseline and candidate
+evaluations on both train and validation datasets, normalize failures, build
+case diffs, detect overfitting, and apply an independent Gate. Stage 4 adds a
+common Candidate Provider boundary, an `AgentOptimizer` adapter, and guarded
+source-prompt writeback. Stage 5 publishes a complete JSON/Markdown report and
+artifact index, or preserves a standalone failure report when a run fails.
+
+The deterministic offline mode runs through the SDK Agent without an API key,
+judge, or real optimizer. Its built-in scenarios produce ACCEPT for `improve` and REJECT for
+both `no_improvement` and `overfit`:
+
+```bash
+python examples/optimization/eval_optimize_loop/run_pipeline.py \
+  --run-id local_stage6 \
+  --scenario improve
+```
+
+The same CLI also supports trace replay:
+
+```bash
+python examples/optimization/eval_optimize_loop/run_pipeline.py \
+  --config examples/optimization/eval_optimize_loop/pipeline.trace.json \
+  --scenario overfit
+```
+
+For an
+explicit real integration smoke run, configure the OpenAI-compatible business
+model connection in the environment:
+
+```bash
+export TRPC_AGENT_API_KEY=...
+export TRPC_AGENT_BASE_URL=...
+export TRPC_AGENT_MODEL_NAME=...
+
+.venv/bin/python examples/optimization/eval_optimize_loop/run_real_integration.py \
+  --run-real \
+  --optimizer-model-name mimo-v2.5 \
+  --max-candidate-proposals 1
+```
+
+The business model uses the three environment values above. The reflection
+optimizer model is selected explicitly with CLI arguments; its endpoint and
+credential reuse the environment without writing their resolved values to run
+artifacts. `--run-real` is mandatory so an accidental command cannot spend API
+quota. This entry always uses `pipeline.real.json`, where source writeback is
+disabled, and reports ACCEPT or REJECT without treating REJECT as a process
+failure. Both CLIs print the paths of the JSON report, Markdown report, and
+artifact index after a completed run.
+
+Applications with a custom agent can still use the Python integration point.
+Set `execution.mode` to `real`, prepare the run, and inject an async
+SDK-compatible `call_agent`:
+
+```python
+import asyncio
+
+from examples.optimization.eval_optimize_loop.pipeline import prepare_run
+from examples.optimization.eval_optimize_loop.pipeline import run_real_stage
+
+prepared = prepare_run("examples/optimization/eval_optimize_loop/pipeline.json")
+agent = MyBusinessAgent(target_prompt=prepared.working_target)
+result = asyncio.run(
+    run_real_stage(prepared, call_agent=agent.call_agent)
+)
+```
+
+The injected agent must reread `prepared.working_target` on every call. The
+same callable is used for baseline regression, optimizer search, and final
+candidate regression. Optimizer minibatches never replace the four complete
+pipeline evaluations: baseline train/validation and candidate train/validation.
+
+`AgentOptimizer` always receives the isolated working target and
+`update_source=False`. Its native `result.json`, round records, prompt snapshots,
+scores, and configuration snapshot are retained under `runs/<run-id>/optimizer/`
+when `artifacts.retain_optimizer_native_artifacts` is enabled.
+
+## Report artifacts
+
+A completed run atomically publishes the formal bundle at
+`runs/<run-id>/report/`:
+
+```text
+report/
+├── optimization_report.json
+├── optimization_report.md
+├── artifact_index.json
+├── inputs/
+│   ├── pipeline_config.json
+│   ├── optimizer_config.json
+│   ├── train_evalset.json
+│   └── validation_evalset.json
+├── evaluations/
+│   ├── baseline_train.json
+│   ├── baseline_validation.json
+│   ├── candidate_train.json
+│   └── candidate_validation.json
+└── prompts/
+    ├── baseline/
+    └── candidate/
+```
+
+The formal `report/` directory is visible only after all required artifacts
+have been written and validated. If any post-preparation phase fails, the
+pipeline does not leave a partial formal report; it atomically writes
+`runs/<run-id>/failure_report.json` with the failed phase, completed phases,
+sanitized error information, source Prompt hashes, and already existing
+artifacts. A failure-report write error is surfaced together with the original
+pipeline error.
+
+`artifact_index.json` records each artifact's relative path, SHA-256 hash, byte
+size, producer phase, and availability. Consumers can use those fields to
+verify that an artifact has not drifted since publication. Input copies are
+validated against the preparation snapshot, and sensitive resolved credentials
+are not accepted into the report bundle.
+
+Source prompts are updated only when all of these conditions hold:
+
+- Gate returns ACCEPT;
+- `writeback.enabled` is true;
+- every source Prompt still matches its preparation-time SHA-256 hash;
+- the write succeeds and an exact readback matches the accepted candidate.
+
+Writeback returns `written`, `skipped`, `blocked`, or `failed` with an auditable
+reason. Gate rejection and disabled writeback are skipped; concurrent source
+edits are blocked; recoverable write/readback failures are rolled back and
+returned as failed. If rollback integrity cannot be proven, the pipeline raises
+an error instead of claiming the source is safe. The checked-in configuration
+keeps writeback disabled by default.
+
+The report separates whole-pipeline resources from optimizer-only observations.
+Pipeline duration is observable, while full pipeline monetary cost and token
+usage remain `unavailable` because business-agent calls may not expose complete
+telemetry. In real mode, optimizer rounds, reflection calls, duration, cost, and
+token usage are reported independently from the native optimizer result; an
+unreliable or incomplete field stays `unavailable` instead of being treated as
+zero. In offline mode, optimizer-only fields are `not_applicable`.
+
+Run the Stage 1–5 tests with:
+
+```bash
+.venv/bin/pytest -q \
+  tests/evaluation/test_eval_optimize_loop_stage1.py \
+  tests/evaluation/test_eval_optimize_loop_stage2.py \
+  tests/evaluation/test_eval_optimize_loop_stage3a.py \
+  tests/evaluation/test_eval_optimize_loop_stage3b.py \
+  tests/evaluation/test_eval_optimize_loop_stage4.py \
+  tests/evaluation/test_eval_optimize_loop_real_integration.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_report_builder.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_artifacts.py \
+  tests/evaluation/test_eval_optimize_loop_stage5_pipeline.py
+```

examples/optimization/eval_optimize_loop/__init__.py

@@ -0,0 +1,6 @@
+# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
+#
+# Copyright (C) 2026 Tencent. All rights reserved.
+#
+# tRPC-Agent-Python is licensed under the Apache License, Version 2.0.
+"""A safe, auditable evaluation and prompt-optimization pipeline example."""

examples/optimization/eval_optimize_loop/analysis.py

@@ -0,0 +1,83 @@
+# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
+#
+# Copyright (C) 2026 Tencent. All rights reserved.
+#
+# tRPC-Agent-Python is licensed under the Apache License, Version 2.0.
+"""Stage 3a orchestration for normalization, attribution, and case diff."""
+
+from __future__ import annotations
+
+from .attribution import attribute_evaluation
+from .case_diff import compare_evaluations
+from .evaluation_adapter import standardize_snapshot
+from .schemas import EvaluationAnalysis
+from .schemas import EvaluationSnapshot
+from .schemas import ObservableValue
+from .schemas import OverfitStatus
+
+
+def _overfit_status(
+    train_delta: ObservableValue,
+    validation_delta: ObservableValue,
+) -> tuple[OverfitStatus, str]:
+    if train_delta.status != "available" or validation_delta.status != "available":
+        return "unavailable", "Train or validation score delta is unavailable."
+    train_value = float(train_delta.value)
+    validation_value = float(validation_delta.value)
+    if train_value > 0.0 and validation_value < 0.0:
+        return (
+            "detected",
+            f"Train score improved by {train_value:.6f} while validation regressed by "
+            f"{validation_value:.6f}.",
+        )
+    return (
+        "not_detected",
+        f"Train score delta is {train_value:.6f}; validation score delta is "
+        f"{validation_value:.6f}.",
+    )
+
+
+def build_evaluation_analysis(
+    *,
+    baseline_train: EvaluationSnapshot,
+    baseline_validation: EvaluationSnapshot,
+    candidate_train: EvaluationSnapshot,
+    candidate_validation: EvaluationSnapshot,
+    hard_case_ids: set[str],
+    critical_case_ids: set[str],
+    severe_case_score_drop: float,
+) -> EvaluationAnalysis:
+    """Build stage 3a analysis from the four complete evaluation snapshots."""
+    normalized_baseline_train = attribute_evaluation(standardize_snapshot(baseline_train))
+    normalized_baseline_validation = attribute_evaluation(standardize_snapshot(baseline_validation))
+    normalized_candidate_train = attribute_evaluation(standardize_snapshot(candidate_train))
+    normalized_candidate_validation = attribute_evaluation(standardize_snapshot(candidate_validation))
+
+    train_diff = compare_evaluations(
+        normalized_baseline_train,
+        normalized_candidate_train,
+        hard_case_ids=hard_case_ids,
+        critical_case_ids=critical_case_ids,
+        severe_case_score_drop=severe_case_score_drop,
+    )
+    validation_diff = compare_evaluations(
+        normalized_baseline_validation,
+        normalized_candidate_validation,
+        hard_case_ids=hard_case_ids,
+        critical_case_ids=critical_case_ids,
+        severe_case_score_drop=severe_case_score_drop,
+    )
+    overfit_status, overfit_reason = _overfit_status(
+        train_diff.score_delta,
+        validation_diff.score_delta,
+    )
+    return EvaluationAnalysis(
+        baseline_train=normalized_baseline_train,
+        baseline_validation=normalized_baseline_validation,
+        candidate_train=normalized_candidate_train,
+        candidate_validation=normalized_candidate_validation,
+        train_diff=train_diff,
+        validation_diff=validation_diff,
+        overfit_status=overfit_status,
+        overfit_reason=overfit_reason,
+    )