Skip to content

Feat Positional Delete by file path - #3024

Open
Kurtiscwright wants to merge 2 commits into
apache:mainfrom
Kurtiscwright:feat-pos-delete-by-file
Open

Feat Positional Delete by file path#3024
Kurtiscwright wants to merge 2 commits into
apache:mainfrom
Kurtiscwright:feat-pos-delete-by-file

Conversation

@Kurtiscwright

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

Adds path-based indexing of position delete files to DeleteFileIndex, mirroring Java's posDeletesByPath.

  • New pos_deletes_by_path map, replacing the commented-out TODO field.
  • New private helper referenced_data_file: uses the explicit referenced_data_file field when set, otherwise infers the path from equal file_path column bounds, matching ContentFileUtil.referencedDataFile. Unequal or missing bounds fall back to partition routing, so truncated bounds stay safe.
  • Routing in PopulatedDeleteFileIndex::new matches on content type first. Each delete lands in exactly one map.
  • get_deletes_for_data_file gains a path-keyed lookup arm with the same greater-than-or-equal sequence filter as the partition arm and no partition spec id check, matching Java's findPathDeletes so path-keyed deletes survive partition evolution.

A file-scoped position delete is now returned only for the data file it references instead of for every file in its partition, so scans load delete files in proportion to the files actually scanned. Deletion vector support (dvByPath in Java) is out of scope.

Are these changes tested?

Yes. Nine new unit tests alongside the two existing ones, all 11 pass: routing via the field and via bounds, fallback on unequal and one-sided bounds, path matching across partition spec ids, greater-than-or-equal
sequence semantics, composition of path-keyed, partition-keyed, and global deletes for one data file, multiple deletes under one path key, and a partitioned delete with a referenced path routing by path. The two pre-existing tests had a hardcoded referenced_data_file removed from their builder so they continue to exercise partition routing. make check passes.

AI Disclosure

The production code was written by hand, guided by an AI-assisted review of the Java reference implementation. Doc comments and unit tests were drafted with AI assistance and reviewed by me. All changes were verified locally with make check and the crate's test suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Index position deletes by referenced data file path in DeleteFileIndex

1 participant