Skip to content

Data: cursor pagination for stable iteration over large collections#78

Merged
jwicks31 merged 1 commit into
mainfrom
claude/cursor-pagination
Jun 13, 2026
Merged

Data: cursor pagination for stable iteration over large collections#78
jwicks31 merged 1 commit into
mainfrom
claude/cursor-pagination

Conversation

@jwicks31

Copy link
Copy Markdown
Owner

What

Adds opt-in cursor pagination alongside the existing limit/offset. A list response now carries a nextCursor; pass it back as ?after=<cursor> to continue in stable (sort, id) order — no skips or repeats when rows are inserted mid-iteration, the way offset drifts. Built to pair with the declared indexes so it stays fast at any size.

How

  • The cursor encodes { sortField, dir, value, id } (base64url). The next page adds a row-value predicate (sortField, id) > (value, id) (flipped to < for DESC), reusing the dialect's typed JSON comparison so it's correct on SQLite (json_extract) and Postgres ((data->'f') vs to_jsonb(?::type)). id is the unique tiebreaker and is appended to ORDER BY for a deterministic total order.
  • A malformed cursor — or one whose encoded sort doesn't match the request's sort — is a clean 400.
  • list() now returns { documents, nextCursor }; offset paging is unchanged when no cursor is supplied. SDK: page({ after }), buildQuery emits after, and Query.after / Page.nextCursor are typed.

Verification

  • SQLite 64/64 including a test that walks a collection page-by-page via the cursor while inserting a new row mid-iteration — every original row is seen exactly once with no duplicates, and the late insert sorts into place; plus malformed-cursor and sort-mismatch 400s.
  • Postgres 46/46 (Postgres 16) — the test sorts by a JSON field, exercising the to_jsonb comparison path.
  • Docs updated: README (data section) and /docs (REST table + queries).

This completes both items from your steer — per-collection schemas (#77) and now cursor pagination.

https://claude.ai/code/session_018efxvWw3MRjdtvE5xgBqya


Generated by Claude Code

Add opt-in cursor paging alongside the existing limit/offset. A list response
now carries a nextCursor; pass it back as ?after=<cursor> to continue in stable
(sort, id) order — no skips or repeats when rows are inserted mid-iteration,
unlike offset. Pairs with declared indexes to stay fast at any size.

- The cursor encodes { sortField, dir, value, id } (base64url). The next page
  adds a row-value predicate `(sortField, id) > (value, id)` (flipped for DESC),
  reusing the dialect's typed JSON comparison so it's correct on SQLite and
  Postgres. id is always the unique tiebreaker, and is appended to ORDER BY for a
  deterministic total order.
- A malformed cursor, or one whose sort doesn't match the request, is a clean 400.
- list() now returns { documents, nextCursor }; offset paging is unchanged when
  no cursor is given. SDK: page({ after }) + buildQuery emits after; Query.after
  and Page.nextCursor typed.

Verification: 64/64 SQLite (+ a test that walks a collection by cursor while
inserting a row mid-iteration — every original row seen once, no dupes; plus
bad-cursor and sort-mismatch 400s), 46/46 Postgres 16 (exercises the jsonb
comparison path via a JSON sort field). Docs updated (README, /docs).
@jwicks31 jwicks31 merged commit 4b01126 into main Jun 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants