rfc: add RFC 0039, the end-to-end benchmark

[azimafroozeh]

What & why

Omnigraph's measurement foundation is taking shape: the DST counting golden and the calibrated real-backend ceilings (originated in PR #503 and absorbed into the in-flight contributions) are arriving under RFC-031 and its in-flight amendment. Once they land, storage-call counting is covered end to end. The natural next step is wall-clock performance: how fast, at what throughput, at what cost, with what answer quality. No specification for that exists yet, and this RFC provides it: the end-to-end benchmark, and the rules a published number must satisfy so it is reproducible and defensible.

The core of the proposal, in plain terms: every benchmark run is fully described by its parameters (what data is stored, what state the store is in, what the run does, where it runs, how it is measured), that description plus the results is persisted as an immutable run record which every published number must cite, and seven protocol rules make dishonest or irreproducible numbers invalid rather than merely sloppy (examples: latency claims must come from a driver that cannot hide the worst latencies, wall-clock numbers only from release builds, no bare averages, every number names its backend and machine, effects below the measured noise floor read "no detected effect"). One instrument covers both micro-benchmarking (single operations, per-phase timing, to find where time goes) and realistic workloads (what users feel), as two named configurations of the same harness.

The design is derived from the published record of two benchmark traditions (LDBC / DBTest for databases, the LoCoMo scoring disputes for agent-memory systems); the Motivation section carries the citations. It builds on RFC-031 and changes nothing RFC-031 owns.

Backing issue / RFC

• Is an RFC PR authorized by umbrella issue Specify a reproducible end-to-end benchmark protocol #539 (needs-rfc): docs/rfcs/0039-end-to-end-benchmark.md

Checklist

• Change is focused (one logical change: adds one RFC document)
• Tests added/updated for behavior changes (N/A: docs-only)
• Public docs updated if user-facing surface changed (N/A: no surface change; the RFC is the doc)
• Reviewed against docs/dev/invariants.md — docs-only methodology RFC, no Hard Invariant touched, no deny-list item hit (stated in the RFC's Invariants section)

Local verification

• bash scripts/check-agents-md.sh — passed
• git diff --check — passed
• Cargo gates not run — docs-only change; no code is added or modified

Notes for reviewers

• Reading guide: the tree after the Summary is the whole contract at a glance; Definitions are in dependency order; the seven protocol rules are the normative core; everything scenario-specific is deliberately out of scope (Non-goals).
• A harness prototype implementing the micro configuration exists and will arrive as separate PRs once this specification settles; this PR intentionally contains only the specification.

Greptile Summary

RFC 0039 specifies a reproducible end-to-end benchmarking instrument spanning wall-clock performance, storage calls, throughput, quality, and cost.

• Defines complete run specifications, immutable invocation records, fixture validation, and backend identity requirements.
• Establishes micro and realistic profiles under seven publication and measurement rules.
• Aligns timing reconciliation with RFC-031’s layer-specific logical and physical action vocabularies.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
docs/rfcs/0039-end-to-end-benchmark.md	Adds RFC 0039 and consistently resolves the previously reported identity, gating, action-class, and timing-reconciliation contract problems.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  S[Run specification] --> F[Validated frozen fixture]
  S --> W[Workload and conditions]
  F --> H[Shared benchmark harness]
  W --> H
  H --> M[Measurements]
  M --> R[Invocation-keyed run record]
  R --> P[Published number citing record]
  H --> C[Layer-specific reconciliation]
  C --> R

Loading

_{Reviews (10): Last reviewed commit: "docs(rfc): key calibration by layer voca..." | Re-trigger Greptile}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Start a greploop in Claude Code and it will work through the open comments and keep going until this PR reviews clean.