Skip to content

test(sessions): add Session/Memory/Summary multi-backend replay consistency framework#1

Open
coder-mtj wants to merge 26 commits into
mainfrom
feat/issue-89-replay-consistency
Open

test(sessions): add Session/Memory/Summary multi-backend replay consistency framework#1
coder-mtj wants to merge 26 commits into
mainfrom
feat/issue-89-replay-consistency

Conversation

@coder-mtj

@coder-mtj coder-mtj commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Description | 描述

This PR adds a comprehensive Session/Memory/Summary multi-backend replay consistency test framework.

4-stage pipeline: load → replay → normalize → compare → report

Key Features | 核心特性

  • 20 deterministic replay cases: 10 core + 10 enhanced (Chinese, emoji, nested payloads, large batches, state scoping, event filtering)
  • Deterministic summarizer: Overrides _compress_session_to_summary() for stable output without LLM
  • Recursive comparator: dict-by-sorted-keys, list-by-position, structured DiffEntry with field-level location
  • JSONPath-based allowed_diff: [*] wildcard, mandatory reason, governance limits (≤8 per case, ≤10% ratio)
  • Jaccard semantic similarity: Token-set comparison for summary text (pure stdlib, threshold ≥0.80)
  • Three-category summary detection: loss / overwrite / affiliation at 100% detection rate
  • Two-layer mutation injection: snapshot layer + end-to-end backend layer (SQL row / Redis key)
  • Schema v3 report: backend_statuses, per-case diffs, field-level DiffEntry, FPR/mutation summaries

Backend Coverage | 后端覆盖

Backend Status
InMemory Always available
SQLite Always available (tmp_path)
External SQL Gated by TRPC_AGENT_REPLAY_SQL_URL
Redis Gated by TRPC_AGENT_REPLAY_REDIS_URL

Test Results | 测试结果

  • 57 tests, all passing
  • InMemory baseline: 0 unexpected diffs (20 cases)
  • Cross-backend (InMemory vs SQLite): FPR < 5%
  • Summary fault detection: 100% (loss/overwrite/affiliation)
  • Lightweight mode: ~2 seconds (SLO: ≤ 30s)

Design Documentation | 设计文档

See docs/issue-89-replay-consistency/design.md for normalization strategy, summary comparison strategy, allowed_diff rules, and backend integration.

Related Issue | 关联 Issue

Fix trpc-group#89

Change Type | 修改类型

  • New feature | 新功能
  • Test coverage | 测试覆盖
  • Documentation update | 文档更新

Test Coverage | 测试覆盖

Test File Tests Coverage
test_replay_unit.py 34 normalizer, comparator, allowed_diff, summary_checks
test_summary_checks.py 8 loss, overwrite, affiliation, Jaccard
test_replay_consistency.py 3 baseline, cross-backend, performance SLO
test_replay_injections.py 12 snapshot injection, summary fault injection, FPR

Self-test Checklist | 自测清单

  • Verified locally | 本地验证通过 (57/57 tests pass)
  • No existing features affected | 无影响现有功能 (zero production code changes)
  • Lightweight mode ≤ 30s | 轻量模式 ~2s

weimch and others added 26 commits July 6, 2026 19:12
@weimch @raychen911
Bugfix: 修复GraphAgent中AgentNode的last_response被设为「思考/中间轮文本」的问题
a547d6b 
- 原因：没有移除思考内容及做中间轮工具调用Event的检测
- 解决方案：使用Event.is_final_response保证是LLM最后一个结果，然后移除最后一个结果的思考内容
@yangenle6-alt @cursoragent @weimch
feat(a2a): support optional app_name for TrpcA2aAgentService
cbb6979 
Allow the Runner's app identity to differ from the A2A service_name by
adding an optional app_name parameter. Falls back to service_name when
not provided, preserving existing behavior for all current callers.

Co-authored-by: Cursor <cursoragent@cursor.com>
@CongkeChen @raychen911
Add internal pipeline test
3f0d67c
@CongkeChen @raychen911
fix pipeline example paths
e08db6d
@CongkeChen @raychen911
test internal pipeline trigger
96fcee4
@CongkeChen @raychen911
Update internal_pipeline_test_trigger.txt
9ed2cfb
@CongkeChen @raychen911
fix internal pipeline example scripts
778ca59
@weimch
feat(sessions): list_sessions 支持 user_id 为 None 时获取 app_name 下全部 sess…
f10f08c 
…ion (trpc-group#145)

list_sessions 的 user_id 参数改为可选，传 None 时返回指定 app 下
所有用户的会话列表（不含 events）。InMemory/SQL/Redis/EvalSessionService
实现统一调整，并补充中英文文档用法说明及单元测试。
@raychen911
docs: add docs index (trpc-group#146)
4336564
@Phil-Fan @raychen911
fix(docs): correct README installation command syntax
8fdf6d8 
- Fix pip install command by enclosing extras in double quotes
- Remove extra spaces and normalize casing in extra packages list
- Ensure command works correctly for installing optional dependencies
@Phil-Fan @raychen911
fix(docs): fix pip install command formatting
89c44ce 
- Removed extra spaces within the optional dependencies list
- Added quotes around the dependency list to ensure proper bash parsing
- Corrected capitalization for consistency in package names
@bochencwx @raychen911
feat: 动态创建子Agent支持在运行时产生的事件直接转发到父流程的事件流中的能力
bbd07eb
@weimch @raychen911
Bugfix: 修复模型调用失败未显示异常信息的问题
14d78cd 
- 在模型支持重试的特性支持里，把这个信息打印给移除了，现在修复这个问题
@raychen911 @weimch
version: 发布1.1.12版本
059a355
@CongkeChen @raychen911
Fix examples and Build Full Pipeline
17316d0
@CongkeChen @raychen911
add release workflow
e113610
@CongkeChen
Code Review (trpc-group#191)
f8b05ba 
* AI review prompt

* Extend example_timeout_second in all_example_test
@CongkeChen @raychen911
Code Review
1903a9f
@CongkeChen @raychen911
remove ai review prompt file
0174c7c
@jinfengliu-688
Feat: tRPC-Agent对齐Claude Code Plan Mod能力 (trpc-group#148)
9cd6bc3 
Feature:
- 参考claude code 的工具设计、提示词设计进行框架plan mod能力的设计与实现。
- 支持模型主动进入plan 模式 & 用户主动选择plan模式。
- 支持agui 解析 toolset能力

BugFix:
- 修复agui协议传递state数据无法有效更新问题。

Doc:
- 支持plan mod指引文档。
- 单独列出todowrite、task、goal的指引文档。
@helloopenworld @weimch
Runner: avoid data copy
eac046f
@weimch @raychen911
feat: trpc-agent框架web工具支持tavily
5d44031 
- 接入tavily到websearch及webfetch工具
- 补充单测及文档
@raychen911 @weimch
version: 发布1.1.13版本
f2a34ff
@Rook1ex
feat: 新增脚本支持在 uv 安装方式在 macOS 上运行，更新相关安装文档和README文档 (trpc-group#203)
5ca3bf2
@coder-mtj
test(sessions): add Session/Memory/Summary multi-backend replay consi…
189b3dd 
…stency framework

Add a comprehensive replay consistency test harness for Session,
Memory, and Summary backends (closes trpc-group#89).

Key components:
- 20 deterministic replay cases (10 core + 10 enhanced:
  Chinese, emoji, nested payloads, large batches, etc.)
- 4-stage pipeline: load -> replay -> normalize -> compare -> report
- Deterministic summarizer (no LLM dependency)
- Recursive comparator with structured DiffEntry
- JSONPath-based allowed_diff with governance limits
- Jaccard semantic similarity for summary text comparison
- Two-layer mutation injection (snapshot + end-to-end backend)
- Schema v3 diff report with field-level location

Tests: 57 tests covering normalizer, comparator, allowed_diff,
summary_checks, E2E replay, and fault injection.
Lightweight mode (InMemory vs SQLite): ~2s, well within 30s SLO.

Files:
- tests/sessions/replay_consistency/ (11 modules)
- tests/sessions/test_replay_consistency.py
- tests/sessions/test_replay_injections.py
- tests/sessions/test_replay_unit.py
- tests/sessions/test_summary_checks.py
- docs/issue-89-replay-consistency/ (design + usage)

Signed-off-by: coder-mtj <coder-mtj@users.noreply.github.com>
@coder-mtj
chore: trigger CI re-run for CLA check
7114161
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

构建 Session / Memory 多后端回放一致性测试框架

10 participants

@coder-mtj @weimch @yangenle6-alt @CongkeChen @raychen911 @Phil-Fan @bochencwx @jinfengliu-688 @helloopenworld @Rook1ex