fix(arrow-cast): support unsafe casting with masked null values - #10726
Open
dhruvxvaishnav wants to merge 1 commit into
Open
fix(arrow-cast): support unsafe casting with masked null values#10726dhruvxvaishnav wants to merge 1 commit into
dhruvxvaishnav wants to merge 1 commit into
Conversation
Do not fail safe: false casts on unconvertible values that are masked as nulls in binary, string, list, struct, and map arrays. Fixes apache#10711
Contributor
|
run benchmark cast_kernels |
This comment was marked as duplicate.
This comment was marked as duplicate.
Jefffrey
reviewed
Aug 19, 2026
| return Arc::clone(array.values()); | ||
| } | ||
|
|
||
| let mut builder = BooleanBufferBuilder::new(array.values().len()); |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing fix/cast-unsafe-masked-nulls (df27155) to bb1e6cd (merge-base) diff Run configurationrun benchmark cast_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
safe: falsecasting doesn't account for masked null values #10711Rationale for this change
When casting with
safe: false, errors were raised for invalid values even if those values were masked by nulls in the array's null bitmap or child offsets (e.g. invalid UTF-8 in a null binary slot, invalid castable child items in a null fixed-size list, list, list view, struct, or map). Because masked values are logically null, the cast should succeed and preserve the null mask.What changes are included in this PR?
arrow-cast/src/cast/string.rs: Incast_binary_to_stringandcast_binary_view_to_string_view, if fast-path UTF-8 conversion fails withsafe: falseand the array contains nulls, only validate non-null values for valid UTF-8.arrow-cast/src/cast/list.rs: Propagate parent list nulls to child arrays before casting incast_single_element_fixed_size_list_to_values,cast_list_values,cast_list_view_values,cast_list,cast_list_view, andcast_list_to_list_viewwhen the target field is nullable.arrow-cast/src/cast/mod.rs: Propagate parent nulls forFixedSizeListandStructArraycolumns before callingcast_with_optionswhen the target field is nullable.arrow-cast/src/cast/map.rs: Propagate parent nulls to map key and value child arrays when target fields are nullable.arrow-castcovering all container types and binary-to-string casts with masked null values.Are these changes tested?
Yes, new unit tests added in
arrow-cast/src/cast/mod.rs:test_binary_to_string_unsafe_masked_nullstest_binary_view_to_string_view_unsafe_masked_nullstest_fixed_size_list_unsafe_masked_nullstest_list_unsafe_masked_nullstest_list_view_unsafe_masked_nullstest_struct_unsafe_masked_nullstest_map_unsafe_masked_nullsAll tests and lints run and pass cleanly:
cargo test -p arrow-castcargo +stable fmt --all -- --checkcargo clippy -p arrow-cast --all-targets --all-features -- -D warningsgit diff --checkAre there any user-facing changes?
No breaking changes to public APIs.
safe: falsecasting now correctly succeeds on arrays where unconvertible values are masked by nulls instead of erroneously failing.