fix: reject malformed chunk keys in DefaultChunkKeyEncoding - #250
Open
d-v-b wants to merge 5 commits into
Open
Conversation
d-v-b
force-pushed
the
main
branch
2 times, most recently
from
July 29, 2026 13:24
6121499 to
123268a
Compare
…zarr-developers#4205) - Codec construction warnings (e.g. sharding's "disables partial reads") fired twice per array open, and on every decode/encode through the fused pipeline's async fallback. Re-constructions of an already-validated codec chain now go through codecs_from_list_unchecked, which validates structure without repeating first-construction advisory warnings; each warning fires exactly once per open under both pipelines. - concurrent_iter returned a lazy generator while its docstring promised eagerly scheduled tasks; it now materializes the task list so awaiting one at a time cannot serialize the batch. - A garbage codec_pipeline.max_workers value (e.g. from the environment) raised ValueError mid-read; it now warns and falls back to the default, consistent with tolerant handling of config input. - The as-completed pipeline helpers abandoned in-flight tasks when one failed, leaving stray background writes and "Task exception was never retrieved" warnings; failures now cancel and drain outstanding tasks. - Benchmarks: seed the data generator for reproducibility; fix a copy-pasted docstring. - Remove dead commented-out test blocks referencing the removed set_range API. Assisted-by: ClaudeCode:claude-sonnet-5
…th 11 updates (zarr-developers#4216) * chore(deps): bump the python-dependencies group across 1 directory with 11 updates Bumps the python-dependencies group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [numpy](https://github.com/numpy/numpy) | `2.5.0` | `2.5.1` | | [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.0` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.14.3` | `7.15.2` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.155.7` | `6.160.0` | | [tomlkit](https://github.com/python-poetry/tomlkit) | `0.15.0` | `0.15.1` | | [uv](https://github.com/astral-sh/uv) | `0.11.26` | `0.11.31` | | [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.6` | `9.7.7` | | [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) | `1.0.4` | `1.0.6` | | [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.1` | `1.12.3` | | [ruff](https://github.com/astral-sh/ruff) | `0.15.20` | `0.15.22` | | [mypy](https://github.com/python/mypy) | `2.1.0` | `2.3.0` | Updates `numpy` from 2.5.0 to 2.5.1 - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.5.0...v2.5.1) Updates `typer` from 0.26.8 to 0.27.0 - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.26.8...0.27.0) Updates `coverage` from 7.14.3 to 7.15.2 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.14.3...7.15.2) Updates `hypothesis` from 6.155.7 to 6.160.0 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.155.7...v6.160.0) Updates `tomlkit` from 0.15.0 to 0.15.1 - [Release notes](https://github.com/python-poetry/tomlkit/releases) - [Changelog](https://github.com/python-poetry/tomlkit/blob/master/CHANGELOG.md) - [Commits](python-poetry/tomlkit@0.15.0...0.15.1) Updates `uv` from 0.11.26 to 0.11.31 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/0.11.31/CHANGELOG.md) - [Commits](astral-sh/uv@0.11.26...0.11.31) Updates `mkdocs-material[imaging]` from 9.7.6 to 9.7.7 - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](squidfunk/mkdocs-material@9.7.6...9.7.7) Updates `mkdocstrings` from 1.0.4 to 1.0.6 - [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases) - [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md) - [Commits](mkdocstrings/mkdocstrings@1.0.4...1.0.6) Updates `markdown-exec[ansi]` from 1.12.1 to 1.12.3 - [Release notes](https://github.com/pawamoy/markdown-exec/releases) - [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md) - [Commits](pawamoy/markdown-exec@1.12.1...1.12.3) Updates `ruff` from 0.15.20 to 0.15.22 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/0.15.22/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.20...0.15.22) Updates `mypy` from 2.1.0 to 2.3.0 - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v2.1.0...v2.3.0) --- updated-dependencies: - dependency-name: coverage dependency-version: 7.15.2 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.160.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: markdown-exec[ansi] dependency-version: 1.12.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocs-material[imaging] dependency-version: 9.7.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocstrings dependency-version: 1.0.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mypy dependency-version: 2.3.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: numpy dependency-version: 2.5.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.15.22 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: tomlkit dependency-version: 0.15.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: typer dependency-version: 0.27.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.11.31 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * fix: satisfy numpy 2.5.1 type stubs in indexing selection normalization numpy 2.5.1 stubs infer np.asarray(<list>) as a float64 array, so mypy now rejects the untyped asarray calls in replace_lists and CoordinateIndexer. Pass dtype=np.intp where the integer dtype is guaranteed, and cast to ArrayOfIntOrBool where the list contents are only known at runtime. Assisted-by: ClaudeCode:claude-fable-5 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
decode_chunk_key stripped a single leading character and split the rest, so any key at all decoded to something. "0/1" silently became (1,) -- the "0" was eaten as if it were the "c" prefix -- and a key written with one separator decoded wrongly under an encoding configured with the other. Validate the "c<separator>" prefix and raise ValueError when it is absent, so a key that is not a chunk key for this encoding is reported rather than silently misread. Adds the tests this method never had, covering the round trip for both separators and each way a key can fail to carry the prefix. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
fix/decode-chunk-key-validation
branch
from
July 30, 2026 12:02
9be2ba2 to
d08f630
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
What
DefaultChunkKeyEncoding.decode_chunk_keystripped a single leading character and split the rest, so any string decoded to something rather than being rejected:The same applies to a key written with the other separator: under
separator="/", the keyc.0.1decoded via a path that never checked the prefix it was documented to require.This validates the
c<separator>prefix and raisesValueErrorwhen it is absent, so a string that is not a chunk key for this encoding is reported instead of silently misread.Why it matters
The failure was silent and produced plausible-looking coordinates, so a caller validating a key by "does it decode?" got the wrong answer rather than an error. Anything mapping untrusted or foreign keys onto a chunk grid — a server, a translating store, a validator — inherits that.
Compatibility
This is a behavior change: input that previously returned garbage now raises. Nothing in the test suite depended on the old behavior — the full suite passes unchanged (6647 passed) — but it is worth a reviewer's eye in case a downstream caller was relying on the lax parse.
Tests
decode_chunk_keyhad no direct test coverage. This addstests/test_chunk_key_encodings.py: the round trip for both separators and several chunk shapes, one test per way a key can fail to carry the prefix, and the v2 encoding's documented 0-d asymmetry (encode(()) == "0"whiledecode("0") == (0,), which is inherent —"0"is ambiguous between a 0-d array's sole chunk and a 1-d array's chunk 0, and only the array's rank disambiguates it).🤖 Generated with Claude Code