test: add LoCoMo-derived e2e acceptance cases - #1272
Conversation
Reuse Harbor and powercontext-e2e acceptance for three pinned LoCoMo samples, and remove the dead A/B report link.
Fill dataset.checksum from PR 1272 replay.json so task_provenance_matches can pass.
Keep this PR scoped to the e2e sample guide and LoCoMo-derived cases. The README-only fix lands separately.
|
Scope update per the tracking-issue discussion: this PR now covers only the e2e sample guide and the three pinned LoCoMo-derived acceptance cases. The LoCoMo README dead-link cleanup is #1275. Ready for review. |
PsiACE
left a comment
There was a problem hiding this comment.
it's lgtm, but I'm making a change to allow aggregated tasks to avoid multiple containers, so the doc could be removed from the PR for now so I can merge it.
and for docs: If AI helps draft a
page, use the documentation-writer and humanizer skills, then have a human
verify and approve the result.
Keep the three pinned LoCoMo-derived cases. The contributor guide would go stale once Harbor tasks can share a container.
|
Thanks @PsiACE. I dropped the EN/ZH sample guide and the CONTRIBUTING / nav / e2e README links from this PR so the three pinned cases can merge first. I will follow up with the contributor guide after the aggregated-task path lands, using documentation-writer and humanizer plus a human pass. |
Summary
powercontext-e2e acceptance; do not add a new runner.locomo-temporal-banker— category 2,conv-30:q001,D1locomo-multihop-football— category 3,conv-47:q026,D13locomo-open-pastries— category 4,conv-44:q064,D3forbidden_context. Those samples belong in a follow-up PR once the evaluation contract can express them. Later algorithm changes should then include both positive and negative cases on a named subset.Test plan
make harness-checkequivalent: ruff, ty,e2e/bub/tests(5 passed)dataset.checksumfrom CItask_provenance_matches/replay.jsonharbor.task_checksumAI usage statement
Used an AI coding assistant to implement the checked plan and draft this PR. Harbor checksums were taken from CI observations, not invented.