Skip to content

fix: reject malformed chunk keys in DefaultChunkKeyEncoding - #250

Open
d-v-b wants to merge 5 commits into
mainfrom
fix/decode-chunk-key-validation
Open

fix: reject malformed chunk keys in DefaultChunkKeyEncoding#250
d-v-b wants to merge 5 commits into
mainfrom
fix/decode-chunk-key-validation

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

What

DefaultChunkKeyEncoding.decode_chunk_key stripped a single leading character and split the rest, so any string decoded to something rather than being rejected:

>>> enc = DefaultChunkKeyEncoding(separator="/")
>>> enc.decode_chunk_key("0/1")   # not a chunk key for this encoding
(1,)                              # the "0" was eaten as if it were the "c" prefix

The same applies to a key written with the other separator: under separator="/", the key c.0.1 decoded via a path that never checked the prefix it was documented to require.

This validates the c<separator> prefix and raises ValueError when it is absent, so a string that is not a chunk key for this encoding is reported instead of silently misread.

Why it matters

The failure was silent and produced plausible-looking coordinates, so a caller validating a key by "does it decode?" got the wrong answer rather than an error. Anything mapping untrusted or foreign keys onto a chunk grid — a server, a translating store, a validator — inherits that.

Compatibility

This is a behavior change: input that previously returned garbage now raises. Nothing in the test suite depended on the old behavior — the full suite passes unchanged (6647 passed) — but it is worth a reviewer's eye in case a downstream caller was relying on the lax parse.

Tests

decode_chunk_key had no direct test coverage. This adds tests/test_chunk_key_encodings.py: the round trip for both separators and several chunk shapes, one test per way a key can fail to carry the prefix, and the v2 encoding's documented 0-d asymmetry (encode(()) == "0" while decode("0") == (0,), which is inherent — "0" is ambiguous between a 0-d array's sole chunk and a 1-d array's chunk 0, and only the array's rank disambiguates it).

🤖 Generated with Claude Code

@d-v-b
d-v-b force-pushed the main branch 2 times, most recently from 6121499 to 123268a Compare July 29, 2026 13:24
d-v-b and others added 3 commits July 30, 2026 11:08
…zarr-developers#4205)

- Codec construction warnings (e.g. sharding's "disables partial reads")
  fired twice per array open, and on every decode/encode through the fused
  pipeline's async fallback. Re-constructions of an already-validated codec
  chain now go through codecs_from_list_unchecked, which validates
  structure without repeating first-construction advisory warnings; each
  warning fires exactly once per open under both pipelines.
- concurrent_iter returned a lazy generator while its docstring promised
  eagerly scheduled tasks; it now materializes the task list so awaiting
  one at a time cannot serialize the batch.
- A garbage codec_pipeline.max_workers value (e.g. from the environment)
  raised ValueError mid-read; it now warns and falls back to the default,
  consistent with tolerant handling of config input.
- The as-completed pipeline helpers abandoned in-flight tasks when one
  failed, leaving stray background writes and "Task exception was never
  retrieved" warnings; failures now cancel and drain outstanding tasks.
- Benchmarks: seed the data generator for reproducibility; fix a
  copy-pasted docstring.
- Remove dead commented-out test blocks referencing the removed set_range
  API.

Assisted-by: ClaudeCode:claude-sonnet-5
…th 11 updates (zarr-developers#4216)

* chore(deps): bump the python-dependencies group across 1 directory with 11 updates

Bumps the python-dependencies group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [numpy](https://github.com/numpy/numpy) | `2.5.0` | `2.5.1` |
| [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.0` |
| [coverage](https://github.com/coveragepy/coveragepy) | `7.14.3` | `7.15.2` |
| [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.155.7` | `6.160.0` |
| [tomlkit](https://github.com/python-poetry/tomlkit) | `0.15.0` | `0.15.1` |
| [uv](https://github.com/astral-sh/uv) | `0.11.26` | `0.11.31` |
| [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.6` | `9.7.7` |
| [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) | `1.0.4` | `1.0.6` |
| [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.1` | `1.12.3` |
| [ruff](https://github.com/astral-sh/ruff) | `0.15.20` | `0.15.22` |
| [mypy](https://github.com/python/mypy) | `2.1.0` | `2.3.0` |



Updates `numpy` from 2.5.0 to 2.5.1
- [Release notes](https://github.com/numpy/numpy/releases)
- [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst)
- [Commits](numpy/numpy@v2.5.0...v2.5.1)

Updates `typer` from 0.26.8 to 0.27.0
- [Release notes](https://github.com/fastapi/typer/releases)
- [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md)
- [Commits](fastapi/typer@0.26.8...0.27.0)

Updates `coverage` from 7.14.3 to 7.15.2
- [Release notes](https://github.com/coveragepy/coveragepy/releases)
- [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst)
- [Commits](coveragepy/coveragepy@7.14.3...7.15.2)

Updates `hypothesis` from 6.155.7 to 6.160.0
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@v6.155.7...v6.160.0)

Updates `tomlkit` from 0.15.0 to 0.15.1
- [Release notes](https://github.com/python-poetry/tomlkit/releases)
- [Changelog](https://github.com/python-poetry/tomlkit/blob/master/CHANGELOG.md)
- [Commits](python-poetry/tomlkit@0.15.0...0.15.1)

Updates `uv` from 0.11.26 to 0.11.31
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/0.11.31/CHANGELOG.md)
- [Commits](astral-sh/uv@0.11.26...0.11.31)

Updates `mkdocs-material[imaging]` from 9.7.6 to 9.7.7
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@9.7.6...9.7.7)

Updates `mkdocstrings` from 1.0.4 to 1.0.6
- [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases)
- [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md)
- [Commits](mkdocstrings/mkdocstrings@1.0.4...1.0.6)

Updates `markdown-exec[ansi]` from 1.12.1 to 1.12.3
- [Release notes](https://github.com/pawamoy/markdown-exec/releases)
- [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md)
- [Commits](pawamoy/markdown-exec@1.12.1...1.12.3)

Updates `ruff` from 0.15.20 to 0.15.22
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/0.15.22/CHANGELOG.md)
- [Commits](astral-sh/ruff@0.15.20...0.15.22)

Updates `mypy` from 2.1.0 to 2.3.0
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](python/mypy@v2.1.0...v2.3.0)

---
updated-dependencies:
- dependency-name: coverage
  dependency-version: 7.15.2
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: hypothesis
  dependency-version: 6.160.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: markdown-exec[ansi]
  dependency-version: 1.12.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocs-material[imaging]
  dependency-version: 9.7.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mkdocstrings
  dependency-version: 1.0.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: mypy
  dependency-version: 2.3.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: numpy
  dependency-version: 2.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: ruff
  dependency-version: 0.15.22
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: tomlkit
  dependency-version: 0.15.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
- dependency-name: typer
  dependency-version: 0.27.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: python-dependencies
- dependency-name: uv
  dependency-version: 0.11.31
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: python-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix: satisfy numpy 2.5.1 type stubs in indexing selection normalization

numpy 2.5.1 stubs infer np.asarray(<list>) as a float64 array, so mypy
now rejects the untyped asarray calls in replace_lists and
CoordinateIndexer. Pass dtype=np.intp where the integer dtype is
guaranteed, and cast to ArrayOfIntOrBool where the list contents are
only known at runtime.

Assisted-by: ClaudeCode:claude-fable-5

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
decode_chunk_key stripped a single leading character and split the rest,
so any key at all decoded to something. "0/1" silently became (1,) -- the
"0" was eaten as if it were the "c" prefix -- and a key written with one
separator decoded wrongly under an encoding configured with the other.

Validate the "c<separator>" prefix and raise ValueError when it is absent,
so a key that is not a chunk key for this encoding is reported rather than
silently misread.

Adds the tests this method never had, covering the round trip for both
separators and each way a key can fail to carry the prefix.

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
d-v-b force-pushed the fix/decode-chunk-key-validation branch from 9be2ba2 to d08f630 Compare July 30, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant