Skip to content

feat(transaction): 207x faster commits by staging a fast-append without a Transaction - #2998

Open
Ignalina wants to merge 1 commit into
apache:mainfrom
Ignalina:feat/stage-fast-append
Open

feat(transaction): 207x faster commits by staging a fast-append without a Transaction#2998
Ignalina wants to merge 1 commit into
apache:mainfrom
Ignalina:feat/stage-fast-append

Conversation

@Ignalina

Copy link
Copy Markdown

Going from 8.4 seconds per commit to 40 milliseconds, by adding one function,
gives 207x speedup.

Found building skade — an embedded Iceberg engine that commits from inside the
writing process: https://codeberg.org/nordisk/skade

Every fast-append runs validate_duplicate_files: loads the manifest list, then
load_manifest() on every entry. O(live data files) Avro decode, per commit. The
cost is the table's own history, re-decoded on every append.

files   check on   check off
 1004     1.42 s     61.7 ms
 4004     8.42 s     40.6 ms

Check off is FLAT across depth. Check on grows superlinearly.

It shows up long before that. The bench is in the repo above:

skade/examples/write_commit_bench.rs
cargo run --release --example write_commit_bench

oden, 32 cores, 200k rows:

lone-commit   0.71 ms/commit
              first 5 commits  0.52 ms
              last 5 commits   0.86 ms      1.65x worse after 40 commits

per-file commit    8 899 775 rows/s
batched commit    37 770 955 rows/s          4.24x

with_check_duplicate already exists on FastAppendAction. Unreachable without a
Transaction, and a Transaction commits through a Catalog. An embedded writer with
its own commit path cannot get at it.

This adds:

pub async fn stage_fast_append(
    table: &Table,
    data_files: Vec<DataFile>,
) -> Result<(Vec<TableUpdate>, Vec<TableRequirement>)>

pub async fn stage_fast_append_with(
    table: &Table,
    data_files: Vec<DataFile>,
    check_duplicate: bool,
) -> Result<(Vec<TableUpdate>, Vec<TableRequirement>)>

Returns updates and requirements. Does not commit.

Staged bytes identical either way. The check is a read-only precondition: passes
or aborts. Never touches manifest, manifest list or the returned updates.
Skipping it changes only whether an already-referenced path is rejected. Callers
deriving file names from a per-call unique prefix cannot collide. Callers
appending caller-supplied names must leave it on. stage_fast_append keeps true.

Going from 8.4 seconds per commit to 40 milliseconds, by adding one function,
gives 207x speedup.

Adds Transaction::stage_fast_append and stage_fast_append_with, which run the
FastAppendAction path and return its (updates, requirements) instead of committing
them through a Catalog.

FastAppendAction::with_check_duplicate already exists but is unreachable without a
Transaction, and a Transaction commits through a Catalog. A writer that owns its own
commit path cannot get at it, and cannot batch several tables into one atomic commit.

validate_duplicate_files loads the manifest list and then load_manifest()s every
entry: O(live data files) Avro decode per commit. Measured on an embedded writer,
8.42 s -> 40.6 ms per commit at 4004 data files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant