docs: refresh ClickBench for the merged #598 payload-width decode gate - #599
Conversation
…ode gate Re-ran the full cross-engine ClickBench (heap, columnar, columnar_tuned, citus, duckdb) on the commandprompt#598 build, one consistent run. The width gate on commandprompt#452 phase-2 decode gating recovers the eight narrow queries the always-on gating had regressed, so columnar now beats heap on 34 of 43 queries (was 25), ties 2, loses 7. q24 keeps its gating win. Refreshed the 2026-08-12 run section from that single run: load table, query totals, geomeans, the win count, the wide-text loss table, and the tuned-arm note. Rewrote the commandprompt#452-trade blockquote: the selectivity regression is resolved by the width gate, not an open item for a runtime probe. Fixed a stale back-reference (2026-08-09 -> 2026-08-12). Numbers verified against the raw run; STE style gate passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017N82wDmsawqSWoWkmxtHmW
jdatcmd
left a comment
There was a problem hiding this comment.
Approve. The refresh is accurate and the narrative matches the #598 behavior I verified on that PR (the eight narrow queries recover, q24 alone clears the 20-column threshold and keeps gating's gain). One non-blocking arithmetic nit in the prose.
Verified
- STE clean. Ran
ste_check.pyon the refreshedbenchmarks.mdlocally, not just trusting CI:ok. That is why thedocs_stylesuite is green. - Every derivable number reconciles — I hand-checked:
- win/tie/loss vs heap 34+7+2 = 43, ties named (q34, q35) = 2; vs Citus 39+4 = 43.
- load ratios: 443.4/186.8 = 2.37, 88.8/49.9 = 1.78, 141.5/88.8 = 1.59, serial-to-bulk 443.4/88.8 = 4.99 (Citus 186.8/49.9 = 3.74), size 11.1% and 5.3x.
- wide-text loss table: every
addsequals col-heap and everyratioequals col/heap (q24: 3376.1-777.9 = 2598.2, /777.9 = 4.34). - geomean composition: columnar-vs-Citus 0.26 = 0.49/1.86, internally consistent.
- I did not re-run the raw ClickBench suite; the totals (157.6 / 129.4 / 260.6 s) and geomeans (0.49 / 1.86) are derived aggregates the harness does not emit, so I verified their internal composition but not each against the 43 raw rows. Your "every number traces to the raw run" covers that.
One non-blocking nit: the win-count sentence does not add up on its face
...moved the ClickBench win count from 33 to 25. #598 then gated the gating... which recovered those queries and returned the win count to 34.
25 + the eight recovered narrow queries = 33, not 34. The 34 is real and I can see where it comes from — the loss and tie deltas (13->7 and 5->2) put nine queries into the win column, which is the eight narrow ones plus one borderline query that flipped in this fresh run's scatter. But the text credits only "those [eight] queries", so the reader is left one short. Since this doc is otherwise meticulous about run-to-run scatter (it defines a tie by exactly that), one clause closes the gap, e.g. "returned the win count to 34, one above the pre-gating 33 as a borderline query flipped this run". Same point applies to the blockquote's "the win count back at 34" -- it was 33 before gating, never 34, so "back at" reads slightly off.
That is the only thing I would touch, and it is prose, not a measurement. Approving; fold it in if you agree, or leave it -- it does not block.
Follow-up to the now-merged #598. The width gate changed the ClickBench query results, and
docs/benchmarks.mdstill shows the pre-gate numbers, so this refreshes them from a full re-run on the #598 build.I re-ran the complete cross-engine suite (heap, columnar, columnar_tuned, citus, duckdb) as one consistent run on the pg18n non-assert build with #598 installed. Load is read-path-unaffected, but I re-ran it too so the whole 2026-08-12 section comes from a single run rather than a splice.
What the width gate does to ClickBench:
Doc changes (all in the 2026-08-12 run section):
Every number traces to the raw run; the STE style gate passes. No code changes.
🤖 Generated with Claude Code