Collect the versions benchmark results hourly - #1094
Merged
Conversation
The versions benchmark has no materialized view: a machine POSTs its whole result JSON to sink.data with kind = "versions-benchmark", and fetch-results.sh reads the results back out of those rows. This adds the automation around it, mirroring collect-results.yml for the main benchmark. versions/collect-new-results.sh fetches the results that arrived in the last day, rebuilds data.generated.js when they changed, and commits both as one automated pull request on auto-results/versions, which it merges (these are ClickHouse's own results, trusted like the clickhouse* systems are by the main collector). Runs that sent a log but no complete result -- still running, crashed, or an incomplete load -- are reported in the job summary, since nothing else would show them. fetch-results.sh grows SINCE_HOURS: fetch only the versions that ran within that window and leave the other files alone, instead of rebuilding results/ from every version in the sink. That is what makes the job cheap enough to run hourly; the full refresh is still available (FULL=1 / the workflow's full input), guarded against publishing a mass deletion if the query goes wrong. The collector also restores result files that came back byte-different but identical in content: jq reformats numbers (0.020 -> 0.02) whenever it has to reconstruct a document, which would otherwise commit a few hundred formatting-only diffs every hour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
.github/workflows/versions-collect-results.yml— an hourly job that looks at the sink for new versions benchmark results, mirroringcollect-results.ymlfor the main benchmark.The difference is that the versions benchmark has no materialized view: a machine POSTs its whole result JSON to
sink.datawithkind: versions-benchmark, andversions/fetch-results.shreads the results back out of those raw rows. The newversions/collect-new-results.shis the automation around it:SINCE_HOURS, dispatch inputfull: truefor every version),versions/data.generated.jswhen the result files changed,auto-results/versions, open a pull request and merge it — these are ClickHouse's own results, trusted the way the main collector trusts theclickhouse*systems,Supporting changes:
fetch-results.shgainsSINCE_HOURS=<n>: fetch only the versions that ran in that window and leave the other files alone, instead of refetching all ~200 versions from the sink. That is what makes the job cheap enough to run hourly. A full refresh is still the default when the variable is unset, and the collector refuses to publish one that loses more than a tenth of the result files — that means a broken query, not lost results.jqreformats numbers (0.020→0.02) whenever it reconstructs a document, soapply-minvers.pyalone would otherwise commit a few hundred formatting-only diffs every hour.Verified locally against a stubbed
clickhouse-clientin a scratch worktree: with no new results the tree is left completely clean (200 re-serialised files restored, nothing committed,data.generated.jsnot even regenerated); with one new result only that file anddata.generated.jschange and the title/body come out right; a full refresh that loses results exits 1 without publishing;generate-results.shis byte-stable across runs. Not exercised: the real database queries and the PR/merge step, which need theCLICKBENCH_DB_PASSWORDsecret and run only in Actions.🤖 Generated with Claude Code