Skip to content

Collect the versions benchmark results hourly - #1094

Merged
alexey-milovidov merged 1 commit into
mainfrom
versions-collect-results
Jul 26, 2026
Merged

Collect the versions benchmark results hourly#1094
alexey-milovidov merged 1 commit into
mainfrom
versions-collect-results

Conversation

@alexey-milovidov

Copy link
Copy Markdown
Member

Adds .github/workflows/versions-collect-results.yml — an hourly job that looks at the sink for new versions benchmark results, mirroring collect-results.yml for the main benchmark.

The difference is that the versions benchmark has no materialized view: a machine POSTs its whole result JSON to sink.data with kind: versions-benchmark, and versions/fetch-results.sh reads the results back out of those raw rows. The new versions/collect-new-results.sh is the automation around it:

  1. fetch the results that arrived in the last day (SINCE_HOURS, dispatch input full: true for every version),
  2. rebuild versions/data.generated.js when the result files changed,
  3. commit both to auto-results/versions, open a pull request and merge it — these are ClickHouse's own results, trusted the way the main collector trusts the clickhouse* systems,
  4. report runs that sent a log but no complete result (still running, crashed, or an incomplete load) in the job summary — nothing else would show those.

Supporting changes:

  • fetch-results.sh gains SINCE_HOURS=<n>: fetch only the versions that ran in that window and leave the other files alone, instead of refetching all ~200 versions from the sink. That is what makes the job cheap enough to run hourly. A full refresh is still the default when the variable is unset, and the collector refuses to publish one that loses more than a tenth of the result files — that means a broken query, not lost results.
  • The collector restores result files that come back byte-different but identical in content. jq reformats numbers (0.0200.02) whenever it reconstructs a document, so apply-minvers.py alone would otherwise commit a few hundred formatting-only diffs every hour.
  • The versions README's claim that a materialized view builds the report is corrected, and the collection path is documented.

Verified locally against a stubbed clickhouse-client in a scratch worktree: with no new results the tree is left completely clean (200 re-serialised files restored, nothing committed, data.generated.js not even regenerated); with one new result only that file and data.generated.js change and the title/body come out right; a full refresh that loses results exits 1 without publishing; generate-results.sh is byte-stable across runs. Not exercised: the real database queries and the PR/merge step, which need the CLICKBENCH_DB_PASSWORD secret and run only in Actions.

🤖 Generated with Claude Code

The versions benchmark has no materialized view: a machine POSTs its whole
result JSON to sink.data with kind = "versions-benchmark", and fetch-results.sh
reads the results back out of those rows. This adds the automation around it,
mirroring collect-results.yml for the main benchmark.

versions/collect-new-results.sh fetches the results that arrived in the last
day, rebuilds data.generated.js when they changed, and commits both as one
automated pull request on auto-results/versions, which it merges (these are
ClickHouse's own results, trusted like the clickhouse* systems are by the main
collector). Runs that sent a log but no complete result -- still running,
crashed, or an incomplete load -- are reported in the job summary, since
nothing else would show them.

fetch-results.sh grows SINCE_HOURS: fetch only the versions that ran within
that window and leave the other files alone, instead of rebuilding results/
from every version in the sink. That is what makes the job cheap enough to run
hourly; the full refresh is still available (FULL=1 / the workflow's full
input), guarded against publishing a mass deletion if the query goes wrong.

The collector also restores result files that came back byte-different but
identical in content: jq reformats numbers (0.020 -> 0.02) whenever it has to
reconstruct a document, which would otherwise commit a few hundred
formatting-only diffs every hour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alexey-milovidov alexey-milovidov self-assigned this Jul 26, 2026
@alexey-milovidov
alexey-milovidov merged commit 9b9f27c into main Jul 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant