Skip to content

Measure numeric arithmetic and comparison benchmarks per CPU feature - #9599

Merged
connortsui20 merged 1 commit into
developfrom
ct/cpu-features-numeric-benches
Aug 25, 2026
Merged

Measure numeric arithmetic and comparison benchmarks per CPU feature#9599
connortsui20 merged 1 commit into
developfrom
ct/cpu-features-numeric-benches

Conversation

@connortsui20

@connortsui20 connortsui20 commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

Sweeps #[cpu_features] across the microbenchmarks that clearly earn it: the binary numeric arithmetic and comparison kernels. Those are portable lane loops — the source is identical on every target and the vector width the compiler picks comes from the build flags — which is the case the attribute exists for. Measuring them in simulation under one fixed +avx2 build hides the only variable that matters.

Middle of a three-PR stack: #9598, then this PR, then #9587. It carries no kernel changes of its own — everything here is a benchmark attribute — so it can be reordered or rebased onto develop without touching the other two.

Changes

Tagged:

  • binary_ops: the primitive arithmetic cases (add_*, subtract_*, multiply_*, mul_*, div_i64_*, sub_i64_constant, and the three *_shapes matrices) and the two primitive comparison cases (eq_i64_constant, lt_i64_nullable).
  • compare: compare_int, compare_int_nullable, compare_int_constant, compare_int_eq, compare_float.
  • scalar_subtract.
  • lane_kernels: lanezip_checked_add_u32 and its arrow_checked_add_u32 baseline. The baseline is tagged too — comparing the two is only meaningful under the same build flags.

Left in simulation, with the reasoning recorded in each file's module docs:

  • Decimal arithmetic and comparison: i128 widening and per-lane rescaling, not something a wider vector register decides.
  • Boolean and/or: already word-at-a-time over a bitmap.
  • String and struct comparison: dominated by view chasing and per-field dispatch.
  • Casts in lane_kernels: vectorization-sensitive, but out of scope here.

Also left alone: the between benchmarks in vortex-fastlanes (new_raw_prim_test_between is a raw-primitive comparison kernel and does qualify) and the bit-packed comparison matrices. Both are types =/consts = parameterized, which #[cpu_features] has no coverage for yet, and both would fan out to dozens of walltime series. Worth a follow-up rather than a guess in this PR.

Note that tagging moves a benchmark out of the sharded simulation job, so these series restart on the walltime legs instead of continuing their simulation history.

@codspeed-hq

codspeed-hq Bot commented Aug 24, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.59%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 1928 untouched benchmarks
🆕 168 new benchmarks
⏩ 106 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation cold_misaligned[(16, 64)] 380.2 µs 430.1 µs -11.59%
🆕 WallTime compare_f32_neon N/A 5.4 µs N/A
🆕 WallTime compare_int_constant_left_neon N/A 4 µs N/A
🆕 WallTime compare_u64_neon N/A 4.4 µs N/A
🆕 WallTime compare_u8_neon N/A 3.7 µs N/A
🆕 WallTime add_i32_nonnull_neon N/A 7.9 µs N/A
🆕 WallTime add_i64_constant_neon N/A 9.9 µs N/A
🆕 WallTime add_i64_nonnull_neon N/A 9.8 µs N/A
🆕 WallTime add_i64_nullable_neon N/A 11.5 µs N/A
🆕 WallTime add_shapes_neon[(128, ConstantPerRow)] N/A 3 µs N/A
🆕 WallTime add_shapes_neon[(128, PerRowConstant)] N/A 3.1 µs N/A
🆕 WallTime add_shapes_neon[(128, PerRowNullableConstant)] N/A 3.7 µs N/A
🆕 WallTime add_shapes_neon[(128, PerRowPerRow)] N/A 1.9 µs N/A
🆕 WallTime add_shapes_neon[(16384, ConstantPerRow)] N/A 9.9 µs N/A
🆕 WallTime add_shapes_neon[(16384, PerRowConstant)] N/A 9.9 µs N/A
🆕 WallTime add_shapes_neon[(16384, PerRowNullableConstant)] N/A 10.5 µs N/A
🆕 WallTime add_shapes_neon[(16384, PerRowPerRow)] N/A 9.9 µs N/A
🆕 WallTime add_u32_nonnull_neon N/A 6.6 µs N/A
🆕 WallTime div_i64_nonnull_neon N/A 52.2 µs N/A
🆕 WallTime div_i64_nullable_neon N/A 33 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/cpu-features-numeric-benches (9688ad0) with develop (e4b3421)2

Open in CodSpeed

Footnotes

  1. 106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (0ef496c) during the generation of this report, so e4b3421 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Base automatically changed from ct/compare-bench-cpu-features to develop August 24, 2026 23:30
connortsui20 added a commit that referenced this pull request Aug 24, 2026
## Summary

The measurement half of #9587, split out so the numbers land before the
kernels do.

Bottom of a three-PR stack: this PR, then #9599, then #9587. Tagging
these with `#[cpu_features]` first means the walltime legs record the
portable lane kernel's throughput on `avx2`, `avx512`, and `neon` metal
as a baseline series. The kernel PR then reports against it rather than
introducing both the benchmark and the thing it measures in one diff.

## Changes

- Adds the primitive comparison cases a hand-written SIMD kernel would
have to beat: constant on the left, `u8`, `u64`, and `f32`.
- Tags those four with `#[cpu_features]`, so each walltime leg measures
them under its own build flags instead of in simulation. The existing
cases are untouched and keep their simulation series.
- `bench_compare` now carries an `ItemsCount`, so the report reads as
throughput rather than a time that only means something next to another
run over the same array length.
- Adds module docs recording why these four are tagged and the rest are
not.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The binary arithmetic and comparison kernels are portable lane loops: the source
is the same on every target and the vector width the compiler picks comes from
the build flags. That is exactly what `#[cpu_features]` is for, so the
microbenchmarks over them move from simulation onto the walltime legs, where they
are measured under each feature set on silicon that implements it.

Tagged: the primitive arithmetic and comparison cases in `binary_ops`, the
integer and float cases in `compare`, `scalar_subtract`, and the checked-add pair
in `lane_kernels` -- the arrow-rs baseline included, since comparing the two is
only meaningful under the same build flags.

Left in simulation: decimal arithmetic (`i128` widening and per-lane rescaling),
the boolean kernels (already word-at-a-time over a bitmap), and the string and
struct comparisons (view chasing and per-field dispatch). A wider vector register
is not what decides any of them.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/cpu-features-numeric-benches branch from fd8e3ec to 9688ad0 Compare August 24, 2026 23:30
@connortsui20
connortsui20 merged commit 17bd9e2 into develop Aug 25, 2026
108 of 110 checks passed
@connortsui20
connortsui20 deleted the ct/cpu-features-numeric-benches branch August 25, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants