Skip to content

test: add LoCoMo-derived e2e acceptance cases - #1272

Merged
PsiACE merged 4 commits into
oceanbase:masterfrom
knqiufan:docs/e2e-sample-locomo-1263
Aug 19, 2026
Merged

test: add LoCoMo-derived e2e acceptance cases#1272
PsiACE merged 4 commits into
oceanbase:masterfrom
knqiufan:docs/e2e-sample-locomo-1263

Conversation

@knqiufan

@knqiufan knqiufan commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Refs Tracking Issue: expand benchmark coverage #1263
  • Add three pinned LoCoMo-derived acceptance cases. Reuse Harbor and powercontext-e2e acceptance; do not add a new runner.
    • locomo-temporal-banker — category 2, conv-30:q001, D1
    • locomo-multihop-football — category 3, conv-47:q026, D13
    • locomo-open-pastries — category 4, conv-44:q064, D3
  • These are pinned samples, not a LoCoMo score.
  • These three cases are positive recall probes only. Negative cases (abstention, conflict, or forbidden context) are out of scope here: the current scorer cannot assert forbidden_context. Those samples belong in a follow-up PR once the evaluation contract can express them. Later algorithm changes should then include both positive and negative cases on a named subset.
  • The official EN/ZH sample guide is deferred until the aggregated-task path is stable, so this page does not go stale. A follow-up docs PR will use documentation-writer and humanizer, then a human pass.
  • The LoCoMo README dead-link cleanup is a separate small PR: docs(locomo): remove dead CONVERSATION_PROFILE_AB link #1275.

Test plan

  • make harness-check equivalent: ruff, ty, e2e/bub/tests (5 passed)
  • Fill dataset.checksum from CI task_provenance_matches / replay.json harbor.task_checksum
  • CI Acceptance (sqlite / oceanbase) passed after checksum backfill

AI usage statement

Used an AI coding assistant to implement the checked plan and draft this PR. Harbor checksums were taken from CI observations, not invented.

Reuse Harbor and powercontext-e2e acceptance for three pinned LoCoMo samples, and remove the dead A/B report link.
Fill dataset.checksum from PR 1272 replay.json so task_provenance_matches can pass.
Keep this PR scoped to the e2e sample guide and LoCoMo-derived cases. The README-only fix lands separately.
@knqiufan
knqiufan marked this pull request as ready for review August 19, 2026 09:27
@knqiufan

Copy link
Copy Markdown
Contributor Author

Scope update per the tracking-issue discussion: this PR now covers only the e2e sample guide and the three pinned LoCoMo-derived acceptance cases. The LoCoMo README dead-link cleanup is #1275. Ready for review.

@PsiACE PsiACE left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's lgtm, but I'm making a change to allow aggregated tasks to avoid multiple containers, so the doc could be removed from the PR for now so I can merge it.

and for docs: If AI helps draft a
page, use the documentation-writer and humanizer skills, then have a human
verify and approve the result.

Keep the three pinned LoCoMo-derived cases. The contributor guide would go stale once Harbor tasks can share a container.
@knqiufan

Copy link
Copy Markdown
Contributor Author

Thanks @PsiACE. I dropped the EN/ZH sample guide and the CONTRIBUTING / nav / e2e README links from this PR so the three pinned cases can merge first.

I will follow up with the contributor guide after the aggregated-task path lands, using documentation-writer and humanizer plus a human pass.

@knqiufan knqiufan changed the title docs: add e2e sample guide and more LoCoMo-derived acceptance cases test: add LoCoMo-derived e2e acceptance cases Aug 19, 2026
@PsiACE
PsiACE merged commit 8083aea into oceanbase:master Aug 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants