feat: add the zarr-indexing package (TensorStore-style index transforms, ndsel wire format) - #4196
Open
d-v-b wants to merge 10 commits into
Open
feat: add the zarr-indexing package (TensorStore-style index transforms, ndsel wire format)#4196d-v-b wants to merge 10 commits into
d-v-b wants to merge 10 commits into
Conversation
github-actions
Bot
added
the
needs release notes
Contributor
Author
|
ping @jbms in case you have comments or suggestions |
d-v-b
changed the title
d-v-b
marked this pull request as ready for review
July 28, 2026 20:03
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4196 +/- ##
=======================================
Coverage 93.95% 93.95%
=======================================
Files 91 91
Lines 12747 12747
=======================================
Hits 11977 11977
Misses 770 770 🚀 New features to boost your workflow:
|
* fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes zarr-developers#4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5
Standalone workspace package extracted from the lazy-indexing branch (zarr-developers#3906): composable, lazy coordinate transforms (IndexTransform / IndexDomain / output maps), dependency-aware chunk resolution against a DimensionGridLike protocol, and an ndsel-conformant JSON wire format validated against the vendored conformance corpus. zarr itself does not depend on zarr-indexing yet — the runtime wiring lands separately once 0.1.0 is published. The package is numpy-only; its tests exercise chunk resolution against zarr's concrete ChunkGrid, so they run from the workspace root (uv sync --all-packages). Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Candidate-chunk enumeration took the cartesian product of each correlated ArrayMap's per-dimension distinct chunk ids and relied on intersect() to filter untouched combinations. For a diagonal selection of P scattered points that is P**2 intersect calls — quadratic in the number of selected points, the same workload shape as zarr-developers#4174 (400 points: ~2.6s; 10k points: ~30min). Group correlated maps jointly instead: broadcast their per-point chunk ids, take the distinct rows (np.unique(axis=0), O(P log P)), and enumerate exactly the touched combinations. Candidate slots now carry chunk-coordinate tuples covering one or more output dimensions; orthogonal/constant/slice dimensions keep their existing per-dimension candidates. 400-point diagonal resolution drops from 2628ms to 14ms and scales linearly. Assisted-by: ClaudeCode:claude-fable-5
Mirror the treatment zarr-metadata received in zarr-developers#4208/zarr-developers#4210 onto zarr-indexing: a self-contained mkdocs site under the package (own mkdocs.yml, landing page, ndsel wire-format guide, mkdocstrings page per module, and .readthedocs.yaml for a dedicated RTD project), so the package presents as a separate project with docs versioned by its own zarr_indexing-v* release tags rather than zarr-python's. The zarr-python site's API Reference nav links out to it, and each RTD project now skips PR builds that do not touch its half of the repo. The package gains a pinned docs dependency group, a docs build job in its CI workflow, and a justfile with package-scoped dev recipes. Two recipes deviate from the zarr-metadata original by design: - `test` runs against the workspace-root environment (`uv run --project ../.. --all-packages --group test`), because the chunk-resolution tests exercise this package against zarr's chunk grids and `zarr` is deliberately not a dependency of this package. - `typecheck` uses plain `pyright`, unpinned and on the default interpreter, mirroring this package's own CI invocation. The zarr-metadata pin exists for a PEP 661 sentinel regression that zarr-indexing's sources do not hit. composition.py gains the module docstring the other modules already have, since mkdocstrings renders it as the page introduction. Assisted-by: ClaudeCode:claude-fable-5
The bytes-codec byte-order fix this fragment describes shipped upstream and its entry is already in docs/release-notes.md; the fragment survived on this branch only as a rebase remnant, and would emit a duplicate entry in the next release. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
feat/zarr-indexing-package
branch
from
July 30, 2026 11:41
d1e931f to
733b3c3
Compare
…/ndsel Also aligns the zarr-indexing workflow's setup-uv pin (v8.3.2) with the rest of the repo. The vendored-corpus sha is present upstream; the historical d-v-b/ndsel#1 PR reference stays as provenance. Assisted-by: ClaudeCode:claude-fable-5
Per review: the root pyproject.toml should not change in this PR. The package now operates fully standalone (like zarr-metadata); the test invocations layer the package into the repo-root environment as an editable overlay instead (python -m pytest, since a base-env console script would not see the overlay). Assisted-by: ClaudeCode:claude-fable-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR defines
zarr-indexing, a new sub-package in the repository for lazy chunked array indexing. This package currently contains an implementation of tensorstore-style explicit-coordinate indexing, which complies with the spec here: https://github.com/zarr-developers/ndsel.If we merge this, I would move forward on defining zarr-python's current indexing datastructures in zarr-indexing, and then making zarr-python depend on zarr-indexing. At that point we can start releasing new zarr-indexing features and bumping the minimum zarr-indexing version in zarr-python.
For reviewers
This is a big AI-written PR. That might make it boring. But it also represents progress towards a major improvement in how zarr-python models array indexing, and it's safe -- it has no impact on zarr-python itself. If you are interested in lazy indexing, please have a look at the data structures in this new package.
Author attestation
TODO
docs/user-guide/*.mdchanges/This was written by claude. See claude's PR here: d-v-b#249