fix: omit baseline-only formats from the file-size report - #9589
Merged
Conversation
A benchmark run that skips a format (for example vortex-compact) produced no file-size rows for it, so the PR comment rendered every baseline file of that format as shrinking to 0 B (-100.0%). Scope the baseline rows to the (benchmark, scale factor, format) combinations the PR run actually produced, so skipped formats are omitted instead of reported as deleted. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
added
the
action/bench-sql
Claude
github-actions
Bot
removed
the
action/bench-sql
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | mul_u32_nonnull_avx512 |
5.6 µs | 6.3 µs | -10.86% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 12.5 µs | +62.86% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.6 µs | +20.66% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.3 µs | 17.7 µs | +20.52% |
| ⚡ | WallTime | words_gather_scalar_avx2[65536] |
9.4 µs | 8.3 µs | +13.82% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/eager-knuth-yrk9n9 (1397b0e) with develop (4f45f72)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 1↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.021x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.036x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.963x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.974x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.020x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.018x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 55↑ 45↓)
Totals:
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 1↑ 5↓)
duckdb / parquet / ns (1.012x ➖, 2↑ 9↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 1↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.862x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.955x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 3↑ 3↓)
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.032x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.967x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.970x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.023x ➖, 0↑ 0↓)
|
joseph-isaacs
marked this pull request as ready for review
August 25, 2026 13:47
robert3005
approved these changes
Aug 25, 2026
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Pin fastlanes to =0.6.1: the 0.6.2 release changed the bit-transpose API, and the lock-free wasm-test build resolves the latest compatible release, so the WASM smoke test stopped compiling. Regenerate the compact golden snapshot for #9602's delta compression estimate fix, which makes the compact selector encode list offsets with pco instead of fastlanes delta (4434 B vs 4748 B). The snapshot was not updated when that change landed, leaving the golden-corpus check red on develop. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Benchmark PR comments rendered every baseline file of a format the PR run skipped as shrinking to 0 B (-100.0%). For example, on #9544 the SQL benchmark comments listed all
vortex-compactfiles asX MB → 0 B (-100.0%)because that run did not include the vortex-compact preset, so the comparison mistook "not run" for "deleted".Changes
format_file_size_reportinscripts/compare-benchmark-jsons.pyscoped the baseline rows only to(benchmark, scale_factor)pairs present in the PR run, so a format missing from the PR run fell through topr_data.get(key, 0)and reported as 0 B. The scope filter now includes the format, so baseline rows for a format the PR run skipped entirely are omitted from the report. A file genuinely missing within a format that did run still reports as going to zero.Tested with a new case in
scripts/tests/test_benchmark_reporting.py(test_file_size_report_omits_formats_the_pr_run_skipped); all 25 tests in the module pass.Generated by Claude Code