Skip to content

docs: refresh ClickBench for the merged #598 payload-width decode gate - #599

Merged
jdatcmd merged 1 commit into
commandprompt:mainfrom
ChronicallyJD:docs/595-clickbench-refresh
Aug 13, 2026
Merged

docs: refresh ClickBench for the merged #598 payload-width decode gate#599
jdatcmd merged 1 commit into
commandprompt:mainfrom
ChronicallyJD:docs/595-clickbench-refresh

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Follow-up to the now-merged #598. The width gate changed the ClickBench query results, and docs/benchmarks.md still shows the pre-gate numbers, so this refreshes them from a full re-run on the #598 build.

I re-ran the complete cross-engine suite (heap, columnar, columnar_tuned, citus, duckdb) as one consistent run on the pg18n non-assert build with #598 installed. Load is read-path-unaffected, but I re-ran it too so the whole 2026-08-12 section comes from a single run rather than a splice.

What the width gate does to ClickBench:

  • columnar beats heap on 34 of 43 queries, ties 2, loses 7 (was 25 / 13 / 5 with No late materialization: decode cost scales with rows scanned, not rows emitted #452 phase-2 gating always on).
  • The eight narrow queries the always-on gating regressed all recover to wins (q11 0.77, q12 0.79, q14 0.85, q15 0.88, q25 0.76, q27 0.80, q31 0.86, q32 0.92 col/heap).
  • q24 keeps its gating win (col/heap 4.34, versus ~7.8x un-gated) — the one query wide enough to clear the 20-non-qual-column threshold.

Doc changes (all in the 2026-08-12 run section):

Every number traces to the raw run; the STE style gate passes. No code changes.

🤖 Generated with Claude Code

…ode gate

Re-ran the full cross-engine ClickBench (heap, columnar, columnar_tuned,
citus, duckdb) on the commandprompt#598 build, one consistent run. The width gate on
commandprompt#452 phase-2 decode gating recovers the eight narrow queries the always-on
gating had regressed, so columnar now beats heap on 34 of 43 queries (was
25), ties 2, loses 7. q24 keeps its gating win.

Refreshed the 2026-08-12 run section from that single run: load table,
query totals, geomeans, the win count, the wide-text loss table, and the
tuned-arm note. Rewrote the commandprompt#452-trade blockquote: the selectivity
regression is resolved by the width gate, not an open item for a runtime
probe. Fixed a stale back-reference (2026-08-09 -> 2026-08-12).

Numbers verified against the raw run; STE style gate passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017N82wDmsawqSWoWkmxtHmW

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. The refresh is accurate and the narrative matches the #598 behavior I verified on that PR (the eight narrow queries recover, q24 alone clears the 20-column threshold and keeps gating's gain). One non-blocking arithmetic nit in the prose.

Verified

  • STE clean. Ran ste_check.py on the refreshed benchmarks.md locally, not just trusting CI: ok. That is why the docs_style suite is green.
  • Every derivable number reconciles — I hand-checked:
    • win/tie/loss vs heap 34+7+2 = 43, ties named (q34, q35) = 2; vs Citus 39+4 = 43.
    • load ratios: 443.4/186.8 = 2.37, 88.8/49.9 = 1.78, 141.5/88.8 = 1.59, serial-to-bulk 443.4/88.8 = 4.99 (Citus 186.8/49.9 = 3.74), size 11.1% and 5.3x.
    • wide-text loss table: every adds equals col-heap and every ratio equals col/heap (q24: 3376.1-777.9 = 2598.2, /777.9 = 4.34).
    • geomean composition: columnar-vs-Citus 0.26 = 0.49/1.86, internally consistent.
  • I did not re-run the raw ClickBench suite; the totals (157.6 / 129.4 / 260.6 s) and geomeans (0.49 / 1.86) are derived aggregates the harness does not emit, so I verified their internal composition but not each against the 43 raw rows. Your "every number traces to the raw run" covers that.

One non-blocking nit: the win-count sentence does not add up on its face

...moved the ClickBench win count from 33 to 25. #598 then gated the gating... which recovered those queries and returned the win count to 34.

25 + the eight recovered narrow queries = 33, not 34. The 34 is real and I can see where it comes from — the loss and tie deltas (13->7 and 5->2) put nine queries into the win column, which is the eight narrow ones plus one borderline query that flipped in this fresh run's scatter. But the text credits only "those [eight] queries", so the reader is left one short. Since this doc is otherwise meticulous about run-to-run scatter (it defines a tie by exactly that), one clause closes the gap, e.g. "returned the win count to 34, one above the pre-gating 33 as a borderline query flipped this run". Same point applies to the blockquote's "the win count back at 34" -- it was 33 before gating, never 34, so "back at" reads slightly off.

That is the only thing I would touch, and it is prose, not a measurement. Approving; fold it in if you agree, or leave it -- it does not block.

@jdatcmd
jdatcmd merged commit 0ccfcc6 into commandprompt:main Aug 13, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants