Skip to content

Aggregate byte types internally with i64 offsets#23695

Open
maxburke wants to merge 1 commit into
apache:mainfrom
urbanlogiq:fix-offset-overflow
Open

Aggregate byte types internally with i64 offsets#23695
maxburke wants to merge 1 commit into
apache:mainfrom
urbanlogiq:fix-offset-overflow

Conversation

@maxburke

@maxburke maxburke commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

and downcast back to i32-offset-types as necessary.

Which issue does this PR close?

Closes #23694 (and possibly apache/datafusion-comet#4718)

Rationale for this change

Fixes an error when grouping on large string/binary data

What changes are included in this PR?

Inner core groups using i64 offsets and then downcasts to correct type when emitting batch slices

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.09901% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (67947b6) to head (ed48d69).

Files with missing lines Patch % Lines
...hysical-plan/src/aggregates/grouped_hash_stream.rs 89.09% 3 Missing and 3 partials ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 92.00% 0 Missing and 6 partials ⚠️
...plan/src/aggregates/aggregate_hash_table/common.rs 73.33% 0 Missing and 4 partials ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs 57.14% 0 Missing and 3 partials ⚠️
...rc/aggregates/group_values/multi_group_by/bytes.rs 97.67% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #23695   +/-   ##
=======================================
  Coverage   80.67%   80.67%           
=======================================
  Files        1088     1088           
  Lines      367591   367673   +82     
  Branches   367591   367673   +82     
=======================================
+ Hits       296563   296636   +73     
- Misses      53374    53375    +1     
- Partials    17654    17662    +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grouping operations on large datasets can overflow i32 offsets

2 participants