feat(website): benchmark page for the 12-workload suite#298
Merged
Conversation
Refreshes the benchmark page from the published 2026-07-20 run: adds the keyset pagination, dynamic query, create-then-amend, and graph insert workloads, adds the Ktorm column throughout, and updates every figure to the pinned-runner results. The lines-of-code comparison now uses a mechanical counting rule applied identically to every implementation: non-blank, non-comment, non-import lines of the workload file and the model file, with generated code and the shared result types excluded on all sides. The narrative is rewritten to match the new results, and the results matrix is tightened to fit the additional column.
…hed run Updates every workload row to the published run of current main, updates the matrix narrative to match (the single-row lookup is now a dead heat and the point operations sit within about one percent of the leader), and renders negative overhead-over-JDBC percentages with a proper sign.
Updates every workload row to the 4-fork run of current main, which stabilizes the join figures across JIT compilation plans, and adjusts the narrative and measurement date accordingly.
… tables Each figure is now the median across the run's four forks with half the fork range as the spread, matching the aggregation the benchmark repository publishes. The ten-row join tips to Storm under this aggregation, making five of twelve workloads where Storm is the fastest framework.
There was a problem hiding this comment.
Pull request overview
Updates the website benchmarks page to reflect the expanded 2026-07-20/21 benchmark suite, adding new workloads and incorporating Ktorm into the comparisons while refreshing the narrative, LOC methodology, and layout.
Changes:
- Added four new workloads (keyset pagination, dynamic query, create-then-amend, graph insert) and updated all benchmark results.
- Added Ktorm across the results matrix, per-workload charts, and LOC comparisons.
- Adjusted results-matrix rendering (including percent formatting) and tightened CSS spacing to fit the extra column.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+11
to
+13
| // Results from the reproducible suite: one tuned PostgreSQL 17 container over TCP, JMH, 2 forks, | ||
| // 5x3s measured iterations, single thread. Values are mean us/op with the reported error. | ||
| // 4 forks, 5x3s measured iterations, single thread. Values are the median across forks in us/op, | ||
| // with half the range of the fork means as the spread (see the benchmark repository's methodology). |
| <p>The workloads cover common data-access paths: point reads, joined entity hydration, projections, batch writes, change-aware updates and one-to-many object graphs.</p> | ||
| <p>Seven implementations, one database, one discipline: same schema, same data, same transaction boundaries, every score a real network round trip away from PostgreSQL. Mean latency per operation, lower is better. Cells are tinted by distance from the fastest framework in the row, green through red. Percentages are overhead over raw JDBC. Raw JDBC is the baseline.</p> | ||
| <p>The workloads cover common data-access paths: point reads, joined entity hydration, projections, keyset pagination, dynamic queries, batch and dependency-ordered writes, change-aware updates and one-to-many object graphs.</p> | ||
| <p>Eight implementations, one database, one discipline: same schema, same data, same transaction boundaries, every score a real network round trip away from PostgreSQL. Mean latency per operation, lower is better. Cells are tinted by distance from the fastest framework in the row, green through red. Percentages are overhead over raw JDBC. Raw JDBC is the baseline.</p> |
| <p class="bm-matrix-read">Storm is the fastest framework above JDBC on the joins, keyset pagination and the batch insert, where its single multi-row RETURNING statement leads the field. The point operations are a photo finish: the single-row lookup is a dead heat with Hibernate, inside the measurement error, and the projection, dynamic query, update and create-then-amend all sit within a few percent of the leader. Two workloads favor jOOQ, the object graph and the dependency-ordered graph write, where its MULTISET load and client-constructed results do less work; Storm trails by six percent on the graph write and by a third on the object graph. Because every result includes a real network round trip, framework overhead is only part of the reported latency, and absolute times depend on the hardware. The relative comparison within each row is the point.</p> | ||
|
|
||
| <details class="bm-details"> | ||
| <summary>Per-workload charts: the same numbers with their reported error</summary> |
zantvoort
changed the title
…ark page and landing cards The page derives every figure from the committed run: a relative line chart with legend highlighting, per-workload bar tables with the fastest-fork score and fork range, foldable sections tracking applied optimizations and semantic differences, per-workload code and SQL verified against a statement-logged instance, and a methodology section stating the fairness rules, estimator and plan-cache reading notes. Winner groups, narrative and the landing-page comparison cards all apply the same 2% convention.
Comment on lines
405
to
409
| jdbc: { | ||
| speed: ['+10 µs', 'over raw JDBC', 'on a primary key lookup. The network round trip dominates every workload; the abstraction barely registers.'], | ||
| entities: ['29 lines', 'instead of manual mapping', 'JDBC has no entities: every row stays untyped until you map it by hand.'], | ||
| queries: ['61%', 'fewer query lines', 'All eight workloads: 100 lines in Storm, 257 of hand-written JDBC and mapping.'], | ||
| speed: ['+18 µs', 'over raw JDBC', 'on a primary-key lookup of 180 µs. The network round trip dominates every workload; the abstraction stays a rounding term.'], | ||
| entities: ['41 lines', 'instead of manual mapping', 'JDBC has no entities: every row stays untyped until you map it by hand.'], | ||
| queries: ['63%', 'fewer query lines', 'All twelve workloads: 150 lines in Storm, 400 of hand-written JDBC and mapping.'], | ||
| }, |
Comment on lines
+1307
to
1327
| export function wireBenchChart() { | ||
| const root = document.getElementById('bm-lc'); | ||
| if (!root) return; | ||
| const seriesFor = (lib) => root.querySelectorAll(`.bm-lc-series[data-lib="${lib}"]`); | ||
| root.querySelectorAll('.bm-lc-lg').forEach((chip) => { | ||
| const lib = chip.getAttribute('data-lib'); | ||
| chip.addEventListener('mouseenter', () => { | ||
| if (chip.classList.contains('off')) return; | ||
| root.classList.add('focus'); | ||
| seriesFor(lib).forEach((n) => n.classList.add('active')); | ||
| }); | ||
| chip.addEventListener('mouseleave', () => { | ||
| root.classList.remove('focus'); | ||
| root.querySelectorAll('.bm-lc-series.active').forEach((n) => n.classList.remove('active')); | ||
| }); | ||
| chip.addEventListener('click', () => { | ||
| chip.classList.toggle('off'); | ||
| seriesFor(lib).forEach((n) => n.classList.toggle('off')); | ||
| }); | ||
| }); | ||
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
website/src/pages/benchmarks.js:1310
- wireBenchChart() adds event listeners every time it runs; if it’s invoked more than once (e.g. React 18 StrictMode double-invokes effects in dev or the page remounts), clicks/hover can end up with duplicated handlers. Add a simple “already wired” guard on the root element.
export function wireBenchChart() {
const root = document.getElementById('bm-lc');
if (!root) return;
const seriesFor = (lib) => root.querySelectorAll(`.bm-lc-series[data-lib="${lib}"]`);
website/src/pages/benchmarks.js:1325
- The chart legend buttons toggle series visibility, but they don’t expose state to assistive tech. Setting aria-pressed based on whether a series is currently shown makes the toggle state discoverable to screen readers.
root.querySelectorAll('.bm-lc-lg').forEach((chip) => {
const lib = chip.getAttribute('data-lib');
chip.addEventListener('mouseenter', () => {
if (chip.classList.contains('off')) return;
root.classList.add('focus');
seriesFor(lib).forEach((n) => n.classList.add('active'));
});
chip.addEventListener('mouseleave', () => {
root.classList.remove('focus');
root.querySelectorAll('.bm-lc-series.active').forEach((n) => n.classList.remove('active'));
});
chip.addEventListener('click', () => {
chip.classList.toggle('off');
seriesFor(lib).forEach((n) => n.classList.toggle('off'));
});
website/src/pages/index.js:411
- A Ktorm comparison chip was added, but the auto-rotation order (vsOrder) used by the interval below isn’t updated, so Ktorm will never appear in the rotating comparisons unless the visitor clicks it.
const vsCards = [...document.querySelectorAll('.storm-home .bcard')];
|
|
||
| <h2>Code comparison</h2> | ||
| <p>Numbers without code invite tuned-benchmark suspicion, so the counts below, and the workloads that follow, show exactly what each library runs, trimmed of harness plumbing. The full sources for all seven implementations are in the benchmark repository.</p> | ||
| <p>Numbers without code invite tuned-benchmark suspicion, so the counts below, and the workloads that follow, show exactly what each library runs, trimmed of harness plumbing. The full sources for all eight implementations are in the benchmark repository.</p> |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
website/src/pages/index.js:409
- The Ktorm comparison is now available in
VS, but the auto-rotation list (vsOrder) does not includektorm, so the page will never cycle to the Ktorm cards unless the user manually clicks the chip.
jdbc: {
speed: ['+18 µs', 'over raw JDBC', 'on a primary-key lookup of 180 µs. The network round trip dominates every workload; the abstraction stays a rounding term.'],
entities: ['41 lines', 'instead of manual mapping', 'JDBC has no entities: every row stays untyped until you map it by hand.'],
queries: ['63%', 'fewer query lines', 'All twelve workloads: 150 lines in Storm, 400 of hand-written JDBC and mapping.'],
},
website/src/pages/benchmarks.js:1310
wireBenchChart()adds event listeners but never removes them, and the legend toggle state isn’t exposed to assistive tech (noaria-pressed, hover-only highlighting). This can lead to duplicated handlers on remount and makes the chart legend harder to use with keyboard/screen readers. Consider returning a cleanup function (likewireSqlToggles) and mirroring hover behavior on focus, while maintainingaria-pressedbased on the toggle state.
// Wires legend hover (highlight one framework) and click (toggle a framework) after mount.
export function wireBenchChart() {
const root = document.getElementById('bm-lc');
if (!root) return;
const seriesFor = (lib) => root.querySelectorAll(`.bm-lc-series[data-lib="${lib}"]`);
|
|
||
| export default function Benchmarks() { | ||
| useEffect(() => wireSqlToggles(), []); | ||
| useEffect(() => { wireSqlToggles(); wireBenchChart(); }, []); |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
website/src/pages/index.js:311
- Ktorm was added to the comparison chips here, but the auto-rotation list later in this effect still omits it (so the carousel never rotates to Ktorm unless the user clicks). Please include
ktormin the rotation order to match the available chips.
<div class="vs-chips"><span class="vs-label">Benchmark against</span><button data-vs="hibernate">Hibernate</button><button data-vs="jooq">jOOQ</button><button data-vs="exposed">Exposed</button><button data-vs="exposedDao">Exposed DAO</button><button data-vs="ktorm">Ktorm</button><button data-vs="jimmer">Jimmer</button><button data-vs="jdbc">JDBC</button><a class="vs-bench" href="/benchmarks">See the benchmarks →</a></div>
website/src/pages/benchmarks.js:1311
wireBenchChartattaches several DOM event listeners but never removes them. In Docusaurus/React SPA navigation (and in React 18 StrictMode during dev), this can result in duplicate handlers and lingering intervals/listeners after unmount. Also, the highlight behavior is currently hover-only; adding focus/blur handlers makes it keyboard-accessible.
export function wireBenchChart() {
const root = document.getElementById('bm-lc');
if (!root) return;
const seriesFor = (lib) => root.querySelectorAll(`.bm-lc-series[data-lib="${lib}"]`);
root.querySelectorAll('.bm-lc-lg').forEach((chip) => {
website/src/pages/benchmarks.js:1731
- Since
wireBenchChartwires DOM listeners, the effect should return its cleanup so navigating away/unmounting doesn’t leave handlers attached (and so dev StrictMode doesn’t double-wire).
useEffect(() => { wireSqlToggles(); wireBenchChart(); }, []);
Comment on lines
+111
to
+115
| const rows = ordered.map(([lib, [mean, err]]) => { | ||
| const meta = LIBS[lib]; | ||
| const width = Math.max(1.5, (mean / max) * 100); | ||
| return `<div class="bm-row ${meta.cls}"> | ||
| <span class="bm-name">${meta.name}</span> | ||
| const leading = lib !== 'jdbc' && mean <= frameworkBest * 1.02; | ||
| return `<div class="bm-row ${meta.cls}${leading ? ' win' : ''}"> |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
website/src/pages/benchmarks.js:1325
- The benchmark chart legend buttons toggle visibility, but the state isn’t exposed to assistive tech. Add/update
aria-pressedwhen wiring the chips so screen readers can announce whether a series is enabled.
chip.addEventListener('click', () => {
chip.classList.toggle('off');
seriesFor(lib).forEach((n) => n.classList.toggle('off'));
});
website/src/pages/index.js:410
- The newly added Ktorm comparison is selectable via the chips, but it is not included in the auto-rotation list (
vsOrder), so the page will never cycle into the Ktorm card unless the user explicitly clicks it.
speed: ['+18 µs', 'over raw JDBC', 'on a primary-key lookup of 180 µs. The network round trip dominates every workload; the abstraction stays a rounding term.'],
entities: ['41 lines', 'instead of manual mapping', 'JDBC has no entities: every row stays untyped until you map it by hand.'],
queries: ['63%', 'fewer query lines', 'All twelve workloads: 150 lines in Storm, 400 of hand-written JDBC and mapping.'],
},
};
| queries: ['63%', 'fewer query lines', 'All twelve workloads: 150 lines in Storm, 400 of hand-written JDBC and mapping.'], | ||
| }, | ||
| }; | ||
| const vsCards = [...document.querySelectorAll('.storm-home .bcard')]; |
Comment on lines
+482
to
+486
| `${QK('SELECT')} o.id, o.first_name, o.last_name, o.address, o.telephone, o.city_id`, | ||
| `${QK('FROM')} owner ${QK('WHERE')} o.id ${QK('IN')} (${QQ('?')}, ${QQ('?')}, ...)`, | ||
| `${QK('FROM')} owner ${QK('WHERE')} o.id = ${QK('ANY')}(${QQ('?')}) ${QC('-- array bind, chunked for large id sets')}`, | ||
| ``, | ||
| `${QC('-- 3) batched cities')}`, | ||
| `${QK('SELECT')} c.id, c.name ${QK('FROM')} city ${QK('WHERE')} c.id ${QK('IN')} (${QQ('?')}, ${QQ('?')}, ...)`, | ||
| `${QK('SELECT')} c.id, c.name ${QK('FROM')} city ${QK('WHERE')} c.id = ${QK('ANY')}(${QQ('?')})`, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
website/src/pages/benchmarks.js:486
- The displayed Jimmer SQL uses aliases
oandcin the WHERE/SELECT clauses but doesn't introduce those aliases in the FROM clauses, which makes the example invalid/misleading (e.g.,FROM owner WHERE o.id = ANY(?)).
`${QK('SELECT')} o.id, o.first_name, o.last_name, o.address, o.telephone, o.city_id`,
`${QK('FROM')} owner ${QK('WHERE')} o.id = ${QK('ANY')}(${QQ('?')}) ${QC('-- array bind, chunked for large id sets')}`,
``,
`${QC('-- 3) batched cities')}`,
`${QK('SELECT')} c.id, c.name ${QK('FROM')} city ${QK('WHERE')} c.id = ${QK('ANY')}(${QQ('?')})`,
website/src/pages/benchmarks.js:1315
- The line-chart legend chips are toggle buttons, but the toggled state isn't exposed to assistive tech and the highlight behavior is mouse-only. Add
aria-pressedupdates (and focus/blur handlers) so keyboard/screen-reader users can discover and operate the toggles consistently.
root.querySelectorAll('.bm-lc-lg').forEach((chip) => {
const lib = chip.getAttribute('data-lib');
chip.addEventListener('mouseenter', () => {
if (chip.classList.contains('off')) return;
root.classList.add('focus');
…ll-on state From the everything-on state, tapping a library keeps that library and Storm and switches the rest off, the one-tap comparison flow. In any other state a tap toggles the library.
…last library toggles off
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
website/src/pages/index.js:411
- Ktorm was added as a benchmark comparison option, but the auto-rotation list (
vsOrder) still omitsktorm, so the carousel will never show Ktorm unless the user clicks it manually.
};
const vsCards = [...document.querySelectorAll('.storm-home .bcard')];
website/src/pages/benchmarks.js:1310
wireBenchChart()attaches event listeners every time it runs, but there is no guard or cleanup. In React dev/StrictMode (and during HMR), this can lead to duplicated listeners and multiple toggles per click/hover. Making the wiring idempotent (or returning a cleanup function) prevents hard-to-debug UI behavior.
export function wireBenchChart() {
const root = document.getElementById('bm-lc');
if (!root) return;
const seriesFor = (lib) => root.querySelectorAll(`.bm-lc-series[data-lib="${lib}"]`);
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
website/src/pages/index.js:411
- Ktorm was added to the VS dataset, but the auto-rotation list (
vsOrder) still omits it, so the homepage comparison cards will never cycle to Ktorm unless the user manually clicks it.
},
};
const vsCards = [...document.querySelectorAll('.storm-home .bcard')];
website/src/pages/benchmarks.js:1310
wireBenchChart()attaches new DOM event listeners every time it runs, but it isn’t idempotent. If the effect re-runs (React 18 StrictMode dev double-mount, HMR, or accidental double-call), hover/click handlers will be duplicated and toggling/highlighting will behave unpredictably.
</svg>
<div class="bm-lc-legend">${legend}<span class="bm-lc-note">hover to highlight · click to toggle</span></div>
</div>`;
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
website/src/pages/benchmarks.js:486
- The displayed Jimmer join SQL is not valid as written: the WHERE clauses reference table aliases (o, c) but the FROM clauses don’t declare them. This will render incorrect SQL in the "Show SQL" tab.
`${QK('SELECT')} o.id, o.first_name, o.last_name, o.address, o.telephone, o.city_id`,
`${QK('FROM')} owner ${QK('WHERE')} o.id = ${QK('ANY')}(${QQ('?')}) ${QC('-- array bind, chunked for large id sets')}`,
``,
`${QC('-- 3) batched cities')}`,
`${QK('SELECT')} c.id, c.name ${QK('FROM')} city ${QK('WHERE')} c.id = ${QK('ANY')}(${QQ('?')})`,
website/src/pages/benchmarks.js:1291
- The chart legend buttons are toggle controls, but they don’t expose any pressed/toggled state to assistive tech. Adding an initial aria-pressed state makes the toggle behavior discoverable to screen readers.
const legend = CHART_LIBS.map((lib) => `<button type="button" class="bm-lc-lg${lib === 'storm' ? ' storm' : ''}" data-lib="${lib}"><span class="sw"${lib === 'storm' ? '' : ` style="background:${CHART_GRAYS[lib]}"`}></span>${LIBS[lib].name}</button>`).join('');
website/src/pages/benchmarks.js:1334
- When a legend item is toggled on/off, the visual state changes via the 'off' class, but the corresponding aria-pressed state (added to the button) should be updated at the same time so assistive tech stays in sync.
const setOn = (c, on) => {
c.classList.toggle('off', !on);
seriesFor(c.getAttribute('data-lib')).forEach((n) => n.classList.toggle('off', !on));
};
website/src/pages/index.js:398
- Ktorm was added as a benchmark comparison option, but the auto-rotation list (vsOrder) still only cycles through the previous libraries. This means visitors won’t see the Ktorm comparison unless they manually click it.
ktorm: {
speed: ['8 of 12', 'workloads faster than Ktorm', 'Level on four, including the graph insert, 1.9 µs apart; ahead on the rest, 2.2x on keyset pagination.'],
entities: ['32%', 'fewer entity lines', 'The five-table model: 41 lines in Storm, 60 lines of Ktorm tables and entity interfaces.'],
queries: ['13%', 'fewer query lines', 'All twelve workloads: 150 lines in Storm, 172 in Ktorm.'],
Ktorm joins every workload section and the model comparison, so each drawer carries all eight implementations, with its own SQL tabs where its statement shape differs (bound-limit point lookup, reference-binding LEFT JOIN, select-star object graph). The chart legend exposes aria-pressed, mirrors the spotlight on keyboard focus, guards against double wiring and returns a cleanup; the page effect returns the combined cleanup of both wiring calls. The landing carousel includes Ktorm and clears its interval and chip listeners on unmount. The per-row spread variable is named for what it is, and abridged SQL snippets define the aliases they use.
Comment on lines
756
to
+760
| clearTimeout(timer); | ||
| clearTimeout(valuesTimer); | ||
| window.removeEventListener('resize', measureHero); | ||
| clearInterval(vsAuto); | ||
| vsChipHandlers.forEach(([chip, handler]) => chip.removeEventListener('click', handler)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Builds the benchmark page and landing-page comparison cards on the committed, reproducible results of the storm-benchmarks suite (
results/2026-07-22).Results presentation
Code comparison
Methodology and reproduction
Figures, winner groups, narrative, and landing cards all derive from the single committed run cited on the page.