feat(transaction): 207x faster commits by staging a fast-append without a Transaction - #2998
Open
Ignalina wants to merge 1 commit into
Open
feat(transaction): 207x faster commits by staging a fast-append without a Transaction#2998Ignalina wants to merge 1 commit into
Ignalina wants to merge 1 commit into
Conversation
Adds Transaction::stage_fast_append and stage_fast_append_with, which run the FastAppendAction path and return its (updates, requirements) instead of committing them through a Catalog. FastAppendAction::with_check_duplicate already exists but is unreachable without a Transaction, and a Transaction commits through a Catalog. A writer that owns its own commit path cannot get at it, and cannot batch several tables into one atomic commit. validate_duplicate_files loads the manifest list and then load_manifest()s every entry: O(live data files) Avro decode per commit. Measured on an embedded writer, 8.42 s -> 40.6 ms per commit at 4004 data files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Going from 8.4 seconds per commit to 40 milliseconds, by adding one function,
gives 207x speedup.
Found building skade — an embedded Iceberg engine that commits from inside the
writing process: https://codeberg.org/nordisk/skade
Every fast-append runs validate_duplicate_files: loads the manifest list, then
load_manifest() on every entry. O(live data files) Avro decode, per commit. The
cost is the table's own history, re-decoded on every append.
Check off is FLAT across depth. Check on grows superlinearly.
It shows up long before that. The bench is in the repo above:
oden, 32 cores, 200k rows:
with_check_duplicate already exists on FastAppendAction. Unreachable without a
Transaction, and a Transaction commits through a Catalog. An embedded writer with
its own commit path cannot get at it.
This adds:
Returns updates and requirements. Does not commit.
Staged bytes identical either way. The check is a read-only precondition: passes
or aborts. Never touches manifest, manifest list or the returned updates.
Skipping it changes only whether an already-referenced path is rejected. Callers
deriving file names from a per-call unique prefix cannot collide. Callers
appending caller-supplied names must leave it on. stage_fast_append keeps true.
Going from 8.4 seconds per commit to 40 milliseconds, by adding one function,
gives 207x speedup.