Fix BEAM benchmark correctness and reproducibility

.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test-and-typecheck:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.6
+
+      - name: Install backend dependencies
+        run: bun install --frozen-lockfile
+
+      - name: Install UI dependencies
+        working-directory: ui
+        run: bun install --frozen-lockfile
+
+      - name: Run tests
+        run: bun test
+
+      - name: Typecheck backend
+        run: bunx tsc --noEmit
+
+      - name: Typecheck UI
+        working-directory: ui
+        run: bunx tsc -p tsconfig.json --noEmit
+
+      - name: Check formatting
+        run: bun run format:check

README.md

@@ -68,19 +68,28 @@ GOOGLE_API_KEY=
 | `list-questions` | Browse benchmark questions |
 | `show-failures` | Debug failed questions |
 | `serve` | Start web UI |
+| `beam prepare` | Download, verify, and convert pinned public BEAM data |
 | `help` | Show help (`help providers`, `help models`, `help benchmarks`) |
 
 ## Options
 
 ```
 -p, --provider         Memory provider (supermemory, mem0, zep)
--b, --benchmark        Benchmark (locomo, longmemeval, convomem, beam-1m, beam-10m)
+-b, --benchmark        Benchmark (locomo, longmemeval, convomem, beam-1m, beam-10m, beam-1m-10m)
 -j, --judge            Judge model (gpt-4o, sonnet-4, gemini-2.5-flash, etc.)
 -r, --run-id           Run identifier (auto-generated if omitted)
 -m, --answering-model  Model for answer generation (default: gpt-4o)
 -l, --limit            Limit number of questions
 -q, --question-id      Specific question (for test command)
 --force                Clear checkpoint and restart
+--data-path            Prepared BEAM snapshot root or snapshot path
+--dataset-revision     Pin the prepared BEAM dataset fingerprint
+--retrieval-top-k      BEAM paper cutoff (5, 10, 15, or 20; default: 5)
+--answer-cutoff        Evidence shown to the answerer in the experimental mem0-nugget profile
+--evaluation-profile   Experimental BEAM evaluation profile (`mem0-nugget`)
+--source-run           Reuse validated completed ingest/index builds in a new run
+--concurrency          Default phase/build concurrency (supported by `run` and `ingest`)
+--ingest-batch-size    Ordered sessions per provider request before its readiness barrier
 ```
 
 ## Examples
@@ -104,6 +113,15 @@ bun run src/index.ts run -p zep -b longmemeval -j sonnet-4 -m gemini-2.5-flash
 # Compare multiple providers
 bun run src/index.ts compare -p supermemory,mem0,zep -b locomo -s 5
 
+# Prepare and run the supported BEAM 1M tier
+bun run src/index.ts beam prepare --tiers 1M
+# Copy the dataset fingerprint printed by `beam prepare` into this command.
+bun run src/index.ts run -p supermemory -b beam-1m -j gpt-4.1-mini --retrieval-top-k 5 --data-path ./data/benchmarks/beam --dataset-revision DATASET_FINGERPRINT_PRINTED_ABOVE
+
+# Experimental direct-K50 mem0-style scoring run reusing the completed 1M build.
+# This does not alter or claim parity with the BEAM paper protocol.
+bun run src/index.ts run -p supermemory -b beam-1m --source-run beam-1m-ingest-c35-b5 -r beam-1m-sm-gpt5-direct-k50-mem0-nugget-v3 --from-phase search --evaluation-profile mem0-nugget --retrieval-top-k 50 --answer-cutoff 50 -m gpt-5 -j gpt-5 --concurrency-search 10 --concurrency-answer 10 --concurrency-evaluate 10
+
 # Test single question
 bun run src/index.ts test -r my-test -q question_42
 
@@ -123,7 +141,35 @@ bun run src/index.ts show-failures -r my-test
 6. REPORT    Aggregate scores → Output accuracy + latency
 ```
 
-Each phase checkpoints independently. Failed runs resume from last successful point.
+Build phases checkpoint once per shared haystack; search, answer, and evaluation checkpoint per
+question. Failed runs resume only when the pinned dataset, protocol, provider adapter, models, and
+retrieval configuration still match.
+
+BEAM uses a per-session causal build barrier: each conversation stays ordered as
+`add -> ready -> checkpoint -> next session`, while separate conversations may run concurrently.
+
+For BEAM, the report keeps the paper score (macro-average across the ten memory abilities) separate
+from pass accuracy (`question score >= 0.5`). Event-ordering questions use the published normalized
+Kendall tau-b score rather than a nugget average. Combined 1M/10M runs
+report each tier separately and label their cross-tier macro as a MemoryBench aggregate, not as a paper score.
+Limited or sampled runs are labeled `beamScorePartial` and cannot enter the ranked leaderboard; only
+the exact validated 700-question 1M or 200-question 10M cohort receives the official `beamScore` key.
+
+The explicit `mem0-nugget` profile is an experimental comparison protocol. It uses the public mem0
+answer/judge prompts, GPT-5 judge identity, mem0's numeric score clamp, and ordinary nugget averages
+for all ten abilities—including event ordering. Its primary metric is `mem0NuggetAverage`, never
+`beamScore`. GPT-5 answering and judging use OpenAI Chat Completions, omit temperature and reasoning
+effort, allow 4,096 completion tokens, and use up to five outer attempts with a 120-second deadline
+and 2/4/6/8-second backoff. Each outer attempt retains the pinned client's two inner transport
+retries. The direct-K50 command above requests and answers with 50 results. It is not an exact
+reproduction of mem0's published Top-50 result: the pinned public mem0 runner retrieves 200 and then
+applies an answer cutoff of 50, while this harness currently permits at most 100 results per direct
+provider request. If all five answering attempts exhaust without non-empty text—including transport
+failures—this profile preserves Mem0's terminal behavior by checkpointing the empty hypothesis and
+evaluating it; other protocols remain fail-closed. The profile also fails closed on schema-invalid judge output instead of using
+mem0's raw-text `1.0`/`0.5` marker fallback. Source-build reuse validates the dataset, ordered
+haystacks, ingestion policy, provider adapter/configuration, and completed indexing before retaining
+the existing containers.
 
 ## MemScore
 
@@ -135,7 +181,7 @@ accuracy% / latencyMs / contextTokens
 
 | Component | What it measures |
 |-----------|-----------------|
-| **Quality** | Answer accuracy — `(correct / total) * 100` from judge evaluations |
+| **Quality** | The benchmark protocol's primary quality metric (legacy benchmarks use binary accuracy; BEAM uses its continuous paper score) |
 | **Latency** | Average search response time in milliseconds |
 | **Tokens** | Average context tokens sent to the answering model (counted client-side) |
 

package.json

@@ -16,7 +16,10 @@
     "@getzep/zep-cloud": "^3.13.0",
     "ai": "^5.0.115",
     "drizzle-orm": "^0.45.1",
+    "hyparquet": "^1.27.1",
+    "hyparquet-compressors": "^1.1.1",
     "js-tiktoken": "^1.0.21",
+    "json5": "2.2.3",
     "mem0ai": "^2.1.38",
     "supermemory": "^4.0.0",
     "zod": "^3.24.4"

src/benchmarks/README.md

@@ -7,11 +7,15 @@ Benchmark dataset adapters. Each benchmark implements the `Benchmark` interface.
 ```typescript
 interface Benchmark {
     name: string
+    scope: BenchmarkScope
+    protocol: BenchmarkProtocol
     load(config?: BenchmarkConfig): Promise<void>
     getQuestions(filter?: QuestionFilter): UnifiedQuestion[]
     getHaystackSessions(questionId: string): UnifiedSession[]
     getGroundTruth(questionId: string): string
     getQuestionTypes(): QuestionTypeRegistry
+    getIngestionGroupId?(questionId: string): string
+    getDatasetIdentity?(): DatasetIdentity | undefined
 }
 ```
 
@@ -28,6 +32,9 @@ interface Benchmark {
 - `getHaystackSessions()` - Return `UnifiedSession[]` for a question
 - `getGroundTruth()` - Return expected answer string
 - `getQuestionTypes()` - Return `{ [id]: { id, alias, description } }`
+- `protocol` - Own ingestion formatting, retrieval, answer formatting, evaluation, and aggregation
+- `getIngestionGroupId()` - Optional shared-build hint; the orchestrator independently fingerprints
+  every member's ordered provider-visible haystack before accepting the group
 
 ## Existing Benchmarks
 
@@ -36,7 +43,7 @@ interface Benchmark {
 | `locomo` | GitHub snap-research/locomo | Long context memory benchmark |
 | `longmemeval` | HuggingFace xiaowu0162/longmemeval-cleaned | Long-term memory evaluation |
 | `convomem` | HuggingFace Salesforce/ConvoMem | Conversational memory benchmark |
-| `beam-1m` / `beam-10m` | HuggingFace Mohammadta/BEAM | Beyond a Million Tokens benchmark (1M and 10M token tiers) |
+| `beam-1m` / `beam-10m` / `beam-1m-10m` | Pinned Hugging Face BEAM repositories | Supported BEAM public 1M/10M tiers |
 
 ## Question Types
 
@@ -69,7 +76,50 @@ interface Benchmark {
 | `implicit_connection_evidence` | implicit | Implicit reasoning |
 | `abstention_evidence` | abstain | Unanswerable questions |
 
-### BEAM
+### BEAM 1M/10M
+
+BEAM does not download silently during a run. Prepare an immutable, hashed snapshot first:
+
+```bash
+bun run src/index.ts beam prepare --tiers 1M,10M
+```
+
+The converter verifies the pinned Parquet hashes, validates 35 chats / 700 questions for 1M and
+10 chats / 200 questions for 10M, requires 20 questions and all ten abilities per chat, and fails
+closed on missing or malformed data. Use `--dataset-revision <fingerprint>` for a pinned run.
+
+The pinned 10M rows store each `turns[]` item as a variable-length alternating message block, not
+as one message pair. The converter deterministically splits every complete block into strict
+user/assistant sessions. The reviewed revision also contains exactly two incomplete blocks:
+`10M:1:plan-7:batch-10:source-turn-19` and
+`10M:2:plan-7:batch-8:source-turn-51`. Each ends with a literal `followup_question` user message
+whose assistant response is absent. Matching the authors' pinned `pair_chunk` behavior in
+`src/answer_probing_questions/long_term_memory_methods.py`, only those two stable source identities
+receive an `ASSISTANT: N/A` placeholder. Canonical sessions mark the padding and the manifest records
+its per-tier count. Any other odd, non-alternating, or identity-mismatched block fails closed.
+
+The supported scope is explicitly BEAM 1M/10M, not every smaller tier in the paper. The paper score
+and the additional `>= 0.5` pass accuracy are both reported; they are not interchangeable.
+Only a complete validated tier is labeled with the official `beamScore` and is leaderboard-eligible.
+Question-limited or sampled runs remain useful diagnostics but are labeled `beamScorePartial`.
+
+BEAM ingestion is causal within each conversation: add one ordered user/assistant session, wait for
+both document processing and memory dreaming/indexing to complete, durably checkpoint it, and only
+then add the next session. Independent conversations still build concurrently. Supermemory requests
+top-level `dreaming: "instant"`; readiness requires both `status` and `dreamingStatus` to be `done`.
+
+For a deliberately non-paper comparison, `--evaluation-profile mem0-nugget` selects a separate
+versioned protocol. It supports direct Top-K values through 100, a distinct answer cutoff, GPT-5 as
+judge, mem0's nugget clamp, and nugget-average scoring for event-ordering questions. Reports use
+`mem0NuggetAverage`, not `beamScore`. Its GPT-5 calls use Chat Completions with 4,096 completion
+tokens, no temperature or reasoning-effort override, five outer attempts, and a 120-second per-attempt
+deadline. Each outer attempt allows two inner transport retries. A direct K50 run is an ablation;
+mem0's published Top-50 setup
+retrieves 200 before applying cutoff 50, which is outside the current direct-search limit.
+After five answer attempts exhaust without non-empty text, this profile alone checkpoints and
+evaluates the empty answer, matching the pinned Mem0 runner; the paper and default protocols continue
+to fail closed.
+
 | Type | Alias | Description |
 |------|-------|-------------|
 | `abstention` | abstain | Withhold answers when evidence is missing |