Data: cursor pagination for stable iteration over large collections#78
Merged
Conversation
Add opt-in cursor paging alongside the existing limit/offset. A list response
now carries a nextCursor; pass it back as ?after=<cursor> to continue in stable
(sort, id) order — no skips or repeats when rows are inserted mid-iteration,
unlike offset. Pairs with declared indexes to stay fast at any size.
- The cursor encodes { sortField, dir, value, id } (base64url). The next page
adds a row-value predicate `(sortField, id) > (value, id)` (flipped for DESC),
reusing the dialect's typed JSON comparison so it's correct on SQLite and
Postgres. id is always the unique tiebreaker, and is appended to ORDER BY for a
deterministic total order.
- A malformed cursor, or one whose sort doesn't match the request, is a clean 400.
- list() now returns { documents, nextCursor }; offset paging is unchanged when
no cursor is given. SDK: page({ after }) + buildQuery emits after; Query.after
and Page.nextCursor typed.
Verification: 64/64 SQLite (+ a test that walks a collection by cursor while
inserting a row mid-iteration — every original row seen once, no dupes; plus
bad-cursor and sort-mismatch 400s), 46/46 Postgres 16 (exercises the jsonb
comparison path via a JSON sort field). Docs updated (README, /docs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds opt-in cursor pagination alongside the existing
limit/offset. A list response now carries anextCursor; pass it back as?after=<cursor>to continue in stable(sort, id)order — no skips or repeats when rows are inserted mid-iteration, the way offset drifts. Built to pair with the declared indexes so it stays fast at any size.How
{ sortField, dir, value, id }(base64url). The next page adds a row-value predicate(sortField, id) > (value, id)(flipped to<forDESC), reusing the dialect's typed JSON comparison so it's correct on SQLite (json_extract) and Postgres ((data->'f')vsto_jsonb(?::type)).idis the unique tiebreaker and is appended toORDER BYfor a deterministic total order.sort— is a clean400.list()now returns{ documents, nextCursor }; offset paging is unchanged when no cursor is supplied. SDK:page({ after }),buildQueryemitsafter, andQuery.after/Page.nextCursorare typed.Verification
400s.to_jsonbcomparison path./docs(REST table + queries).This completes both items from your steer — per-collection schemas (#77) and now cursor pagination.
https://claude.ai/code/session_018efxvWw3MRjdtvE5xgBqya
Generated by Claude Code