Skip to content

feat(async): chunk-level traversal (achunk-next) + AsyncReverseSeq termination-contract fix#20

Open
whilo wants to merge 1 commit into
mainfrom
feat/chunked-async-seq
Open

feat(async): chunk-level traversal (achunk-next) + AsyncReverseSeq termination-contract fix#20
whilo wants to merge 1 commit into
mainfrom
feat/chunked-async-seq

Conversation

@whilo

@whilo whilo commented Jul 17, 2026

Copy link
Copy Markdown
Member

First PR of the datahike cljs-async arc (PR A of the grounded workplan).

Chunk-level async traversal

PAsyncChunkedSeq/achunk-next on AsyncSeq: resolves [keys-array start end next] — the consumer runs a synchronous inner loop over the leaf's key array and awaits once per leaf, not per element. Element-wise anext allocates a continuation per element (measured ~4x slower on warm data in the datahike engine experiments); the chunk shape is what a query engine's scan kernels consume, and awaiting per 512-element chunk measured within ~8% of a fully synchronous loop.

Boundary semantics mirror the sync Iter's IChunkedSeq exactly: same-leaf right bound via path-get, next leaf via -next-path with the path-lt guard. Parity is pinned chunk-for-chunk against Iter (same boundaries, same elements) over full/mid/same-leaf/single/empty ranges plus a leaf-boundary sweep — both in-memory and through a restored async storage (cold node loads).

AsyncReverseSeq termination-contract fix

anext's body was a literal two-slot vector of when forms, so an exhausted reverse seq resolved a truthy [nil nil] instead of nil — one spurious nil element for any consumer written against AsyncSeq's contract (aseq/transduce's if-let, chunk walkers). All internal constructors happen to guard emptiness, so this was latent — but external consumers constructing or re-driving reverse seqs would hit it. The guard now encloses the whole tuple; the AsyncRange test helper had the same shape and is fixed alike; a regression test pins anext → nil on exhausted seqs of both orientations.

Also: the lookup facade gains the opts arity the other facades already have (datahike's IIndex/-lookup opts threading needs it).

Tests

node-tests (advanced): 137 tests / 1826 assertions green — includes the two new namespaces (async-exhaustion, async-chunk). cljs-only change; JVM untouched.

PAsyncChunkedSeq/achunk-next on AsyncSeq resolves [keys-array start end
next]: consumers run a synchronous inner loop over the leaf's key array
and await once per leaf hop. Element-wise anext allocates a continuation
per element (~4x slower warm, measured); the chunk shape is the one a
query engine's scan kernels want, and its boundary semantics mirror the
sync Iter's IChunkedSeq exactly (same-leaf right bound via path-get,
next leaf via -next-path + path-lt guard). Parity is pinned chunk-for-
chunk against Iter, in-memory and through restored async storage.

Also fixes the PAsyncSeq termination contract in AsyncReverseSeq: anext
was a literal two-slot vector of `when` forms, so an exhausted reverse
seq yielded a truthy [nil nil] instead of nil — one spurious nil element
for any consumer written against AsyncSeq's contract (aseq/transduce's
if-let, chunk walkers). The guard now encloses the whole tuple; the
AsyncRange test helper had the same shape and is fixed alike. The
lookup facade gains the opts arity the other facades already have.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant