Aggregate the random-access group summary over every dataset

[claude]

Requested by Connor Tsui · Slack thread

Before. The Random Access summary card showed one number per format, taken from a single chart — the first one in the group with data, in practice feature-vectors/correlated. So the headline said whatever that one dataset happened to say, including its noise: on the local reproduction below Parquet reads as 24.00 ms and 8.89x, because that dataset spiked, not because random access got 9× slower.

After. The card shows two labelled aggregates per format, computed over every dataset in the group: the geomean of their latest times and the total (sum) of them, plus the ratio of geomeans to the fastest format. The same data now reads 2.42 ms geomean · 27.49 ms total · 3.67x for Parquet, and the footer says which datasets are behind it: "Geomean and total random access time across 4 datasets | Ratio of geomean to fastest (lower is better)". One noisy dataset can no longer carry the headline on its own.

That was fine when random access was one benchmark; it is misleading now that it is many.

How

collectRandomAccessSummary (web/lib/summary.ts) replaces the "loop the group's chart links until one has rows" scan with a single query over the whole group. The group's chart names are its dataset names, so they scope the query; per format it then sums and geomeans the latest per-dataset values, and ranks by geomean. SummaryCard.tsx renders both aggregates with the words geomean and total next to them, so a reader is not guessing which number is which. The existing geoMean helper is reused unchanged — it already ignores non-positive and non-finite values, so the geomean is over strictly positive values only.

Aggregating means choosing a snapshot and a rule for partial data:

• One snapshot commit. The aggregate is taken at the newest commit with any positive random-access row for the group. All of random-access-bench's datasets and formats come from the same run, so they share a commit. A same-timestamp commit tie is collapsed with DISTINCT ON (dataset, format) so a tie cannot double-count into the sum.
• Missing data: complete coverage or the format is dropped. A format is ranked only if it has a positive value for every dataset measured at that snapshot commit. Why this option: a sum and a geomean over a smaller dataset set are not comparable to ones over the full set — a format missing an expensive dataset would look cheaper for being absent. This follows the compression summaries' existing "newest complete snapshot" precedent (collectCompressionSummary / collectCompressionSizeSummary require a complete pair and drop what is incomplete) rather than the query summary's missing-series penalty, which works because that summary is a ratio; imputing a penalty into an absolute sum or geomean would invent a runtime the benchmark never measured. If no format has complete coverage the summary is omitted, which is what the previous code also did when nothing qualified.

Other groups are untouched: RandomAccessGroup is a singleton group with its own collect*Summary arm, so no other headline number moves. The randomAccess ranking's wire fields change from time to total + geomean (/api/groups, /api/group/{slug}); the card is the only consumer in-tree.

Tests

• web/lib/summary.test.ts — four new unit tests over a mocked pool: multi-dataset sum/geomean and ratio, query scoping to the group's datasets plus the snapshot/DISTINCT ON shape, a format with incomplete coverage being dropped, and an empty group querying nothing.
• web/lib/groups.test.ts — the Postgres integration assertion now also pins both aggregates; the canonical one-dataset fixture keeps its previous ratios (1.00x / 2.00x), which is the intended behaviour for a one-chart group.
• web/components/SummaryCard.test.tsx, web/components/GroupSection.test.tsx — updated for the new labelled render.
• docs/architecture/design-decisions.md — an ADR entry for the snapshot and coverage rules.

Local: pnpm format:check, pnpm lint, pnpm build, pnpm test (348 passed; the 48 Docker-gated tests self-skipped here) and tsc --noEmit all clean. The new SQL was additionally run against a local PostgreSQL 16 with migrations/ applied, which is where the before/after numbers above come from.

Out of scope / noted

The hot-vs-cold cache mixing in the random-access measurements (the reason feature-vectors/correlated spikes) is not addressed here — it is not fixable from this repo. Details are going back to the Slack thread separately.

---

Generated by Claude Code

[github-actions]

Vercel preview for d2a69115bdabdd1370273d06f1af0bd5a44ee1e3: https://benchmarks-website-9acq2ki7b-vortex-data.vercel.app

[claude]

Superseded by #21, which landed on develop first and already does this: the Random Access summary now ranks across every chart in the group via the shared rankSeries helper and the card shows both a geomean and a summed runtime, so there is nothing left for this PR to add. It conflicts in all six of its code files — the same fix landed twice.

One deliberate difference worth recording: this PR's geomean was of absolute times, while develop's is of per-chart ratios to the fastest format. The ratio geomean is the better statistic here — a geomean of absolute times is dominated by the slowest charts and shifts whenever the chart set changes — and the absolute magnitude is still on the card as the summed runtime. The absolute-time geomean is also inseparable from this PR's drop-incomplete rule, which now contradicts the missing-series penalty develop applies uniformly across every group.

---

Generated by Claude Code