feat(bench): add Arrow IPC benchmark baselines - #9624
Open
lwwmanning wants to merge 8 commits into
Open
Conversation
Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
marked this pull request as ready for review
August 25, 2026 19:14
lwwmanning
enabled auto-merge (squash)
August 25, 2026 19:17
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02ec38b6e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Merging this PR will improve performance by 29%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 13.6 µs | +50.57% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.3 µs | 17.7 µs | +20.43% |
| ⚡ | Simulation | take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000] |
245 µs | 207 µs | +18.39% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing wm/arrow-benchmarks (6a5a606) with develop (ee1ac25)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
4 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩
lwwmanning
disabled auto-merge
August 25, 2026 19:28
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e32bd29438
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
changed the title
lwwmanning
enabled auto-merge (squash)
August 25, 2026 21:13
connortsui20
added
the
action/bench-compress
github-actions
Bot
removed
the
action/bench-compress
| VORTEX_EXPERIMENTAL_PATCHED_ARRAY: "1" | ||
| FLAT_LAYOUT_INLINE_ARRAY_NODE: "1" | ||
| run: | | ||
| python3 scripts/random-access-split.py --emit-ingest-records |
Member
There was a problem hiding this comment.
am realizing we probably should specify all the formats here too, wont block
connortsui20
added
action/bench-random-access
github-actions
Bot
removed
the
action/bench-random-access
connortsui20
added
changelog/ci
and removed
changelog/skip
connortsui20
disabled auto-merge
August 25, 2026 21:31
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (2)
Powered by Polar Signals Cloud |
connortsui20
approved these changes
Aug 25, 2026
connortsui20
left a comment
connortsui20
left a comment
Member
There was a problem hiding this comment.
we should wait for the bench runners to finish
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.978x ➖, 2↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.988x ➖, 2↑ 0↓)
vortex / parquet / ns (0.995x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (no group data, 0↑ 0↓)
vortex / arrow-ipc / bytes (no group data, 0↑ 0↓)
|
connortsui20
requested changes
Aug 25, 2026
connortsui20
left a comment
connortsui20
left a comment
Member
There was a problem hiding this comment.
Compression benchmark has an issue:
Details
+ bash /home/runner/_work/vortex/vortex/scripts/bench-taskset.sh target/release_debug/compress-bench --datasets ^taxi$ --formats arrow-ipc,parquet,vortex -d gh-json -o parts/0.gh.json
2026-08-25T21:32:22.987238Z INFO compress_bench: benchmarks/compress-bench/src/main.rs:427: Running taxi benchmark
thread 'main' (6044) panicked at vortex-bench/src/measurements.rs:357:18:
Other error: CompressionTimingMeasurement only supports vortex, lance, and parquet formats
Backtrace:
0: <vortex_bench::measurements::CompressionTimingMeasurement as vortex_bench::measurements::ToJson>::to_json
at ./vortex-bench/src/measurements.rs:357:18
1: vortex_bench::display::print_measurements_json::<vortex_bench::measurements::CompressionTimingMeasurement>
at ./vortex-bench/src/display.rs:121:36
2: compress_bench::run_compress::{closure#0}
at ./benchmarks/compress-bench/src/main.rs:382:13
3: compress_bench::main::{closure#0}
at ./benchmarks/compress-bench/src/main.rs:162:6
4: <tokio::runtime::park::CachedParkThread>::block_on::<compress_bench::main::{closure#0}>::{closure#0}
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/park.rs:284:71
5: tokio::task::coop::with_budget::<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, <tokio::runtime::park::CachedParkThread>::block_on<compress_bench::main::{closure#0}>::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/task/coop/mod.rs:167:5
6: tokio::task::coop::budget::<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, <tokio::runtime::park::CachedParkThread>::block_on<compress_bench::main::{closure#0}>::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/task/coop/mod.rs:133:5
7: <tokio::runtime::park::CachedParkThread>::block_on::<compress_bench::main::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/park.rs:284:31
8: <tokio::runtime::context::blocking::BlockingRegionGuard>::block_on::<compress_bench::main::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/context/blocking.rs:66:14
9: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<compress_bench::main::{closure#0}>::{closure#0}
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/scheduler/multi_thread/mod.rs:92:22
10: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::multi_thread::MultiThread>::block_on<compress_bench::main::{closure#0}>::{closure#0}, core::result::Result<(), anyhow::Error>>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/context/runtime.rs:65:16
11: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<compress_bench::main::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/scheduler/multi_thread/mod.rs:91:9
12: <tokio::runtime::runtime::Runtime>::block_on_inner::<compress_bench::main::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/runtime.rs:376:50
13: <tokio::runtime::runtime::Runtime>::block_on::<compress_bench::main::{closure#0}>
at /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.53.1/src/runtime/runtime.rs:343:18
14: compress_bench::main
at ./benchmarks/compress-bench/src/main.rs:162:6
15: <fn() -> core::result::Result<(), anyhow::Error> as core::ops::function::FnOnce<()>>::call_once
Contributor
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (no group data, 0↑ 0↓)
random-access / vortex-file-compressed / ns (1.096x ➖, 0↑ 1↓)
random-access / parquet / ns (0.999x ➖, 0↑ 0↓)
random-access / lance / ns (0.999x ➖, 0↑ 0↓)
|
Signed-off-by: Will Manning <will@willmanning.io>
lwwmanning
added
the
action/bench-compress
github-actions
Bot
removed
the
action/bench-compress
connortsui20
approved these changes
Aug 25, 2026
connortsui20
left a comment
connortsui20
left a comment
Member
There was a problem hiding this comment.
latest commit should fix the issue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
arrow-ipcas an on-disk format in the compression-size, compression-throughput, and random-access benchmark suites.Compression benchmark runs emit Arrow IPC file sizes, serialization and deserialization timings, and decoded in-memory Arrow bytes as
uncompressed_bytes. Random-access runs read row-addressable Arrow IPC files.The decoded Arrow byte count supplies logical throughput and compression-ratio estimates. The Arrow IPC rows remain file-format results. The develop workflow emits these records across the existing dataset matrix.