Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 34 additions & 11 deletions .github/workflows/zarr-metadata.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
name: zarr-metadata

# Job steps delegate to packages/zarr-metadata/justfile, the single source of
# truth for this package's verbs; CI owns only the python matrix and caching.

on:
push:
branches: [main]
Expand Down Expand Up @@ -39,12 +42,14 @@ jobs:
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
with:
enable-cache: true
- name: Install just
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}
- name: Sync test dependency group
run: uv sync --group test --python ${{ matrix.python-version }}
- name: Run pytest
run: uv run --group test pytest tests
run: just test

ruff:
name: ruff
Expand All @@ -59,8 +64,10 @@ jobs:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
- name: Install just
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: Run ruff
run: uvx ruff check .
run: just lint

pyright:
name: pyright
Expand All @@ -77,19 +84,35 @@ jobs:
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
with:
enable-cache: true
- name: Set up Python
run: uv python install 3.11
- name: Sync test dependency group
run: uv sync --group test --python 3.11
- name: Install just
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: Run pyright
# Pinned to the last version that types PEP 661 sentinels in class
# attributes correctly; 1.1.405+ regressed (microsoft/pyright#11115).
# Unpin when the fix lands.
run: uv run --group test --with 'pyright==1.1.404' pyright src
# The pyright version and interpreter pins live in the justfile.
run: just typecheck

docs:
name: docs
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: packages/zarr-metadata
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
with:
enable-cache: true
- name: Install just
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: Build docs
run: just docs-check

zarr-metadata-complete:
name: zarr-metadata complete
needs: [test, ruff, pyright]
needs: [test, ruff, pyright, docs]
if: always()
runs-on: ubuntu-latest
steps:
Expand Down
1 change: 1 addition & 0 deletions changes/4201.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed the opt-in `FusedCodecPipeline` for serializers that advertise the partial-decode/encode mixins with only the documented async partial methods: the partial dispatch previously asserted on the private `_decode_partial_sync`/`_encode_partial_sync` hooks (an `AssertionError`, or an `AttributeError` mid-IO under `python -O`); such codecs now take the full-chunk sync path.
10 changes: 10 additions & 0 deletions changes/4202.bugfix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Fixed `FusedCodecPipeline` (the opt-in synchronous pipeline) silently skipping
array-array/bytes-bytes codecs placed outside a sharding serializer on its
partial-decode/partial-encode fast paths. With an outer compressor (e.g.
`compressors=[GzipCodec()]` around a `ShardingCodec` serializer), the fused
pipeline wrote non-conforming stored bytes that `BatchedCodecPipeline` (and any
other conforming reader) could not read, and could fail to read data that
`BatchedCodecPipeline` had written. With an outer array-array codec (e.g.
`TransposeCodec`), it silently returned wrong data in both directions with no
error. Only the opt-in `FusedCodecPipeline` was affected; the default
`BatchedCodecPipeline` was never impacted.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ nav:
- '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code>Β <code>zarr.testing.utils</code>': api/zarr/testing/utils.md
- '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code>Β <code>zarr.zeros</code>': api/zarr/functions/zeros.md
- '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code>Β <code>zarr.zeros_like</code>': api/zarr/functions/zeros_like.md
- 'zarr-metadata β†ͺ': https://zarr-metadata.readthedocs.io/
- release-notes.md
- contributing.md
hooks:
Expand Down
20 changes: 20 additions & 0 deletions packages/zarr-metadata/.readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Read the Docs configuration for the zarr-metadata docs site, separate from
# the zarr-python site configured by the repo-root .readthedocs.yaml. The RTD
# project for zarr-metadata must set its configuration-file path to
# packages/zarr-metadata/.readthedocs.yaml.
version: 2

build:
os: ubuntu-22.04
tools:
python: "3.12"
jobs:
install:
- pip install --upgrade pip
- pip install ./packages/zarr-metadata --group packages/zarr-metadata/pyproject.toml:docs
build:
html:
- mkdocs build --strict -f packages/zarr-metadata/mkdocs.yml --site-dir $READTHEDOCS_OUTPUT/html

mkdocs:
configuration: packages/zarr-metadata/mkdocs.yml
19 changes: 19 additions & 0 deletions packages/zarr-metadata/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Python types, models, and validators for Zarr v2 and v3 metadata.

Documentation: <https://zarr-metadata.readthedocs.io/>

## What this is

Two layers and an optional integration:
Expand Down Expand Up @@ -82,6 +84,23 @@ store I/O. The models begin and end at the metadata documents themselves β€”
`from_key_value` / `to_key_value` map documents to store keys and bytes,
and everything past that belongs to consumer libraries.

## Developing

Package-scoped development commands live in the [`justfile`](./justfile)
(requires [just](https://github.com/casey/just)):

```
just test # run the test suite (extra args go to pytest)
just lint # ruff, same invocation as CI
just typecheck # pyright, pinned to the version CI uses
just docs-check # strict build of the docs site
just check # all of the above
just docs-serve # serve the docs site locally
```

Run them from this directory, or from anywhere in the repository as
`just packages/zarr-metadata/<recipe>`.

## Releasing

The package version is derived from git tags by `hatch-vcs`. Tags must
Expand Down
31 changes: 31 additions & 0 deletions packages/zarr-metadata/docs/api/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: API reference
---

# API reference

The package is organized to mirror the structure of the Zarr specifications:

- [`zarr_metadata.model`](model.md) β€” frozen-dataclass document models,
structural validators, loc-aware parsers, and the `UNSET` sentinel
- [`zarr_metadata.pydantic`](pydantic.md) β€” optional Pydantic field types
over the models
- [`zarr_metadata.v2`](v2.md) β€” `TypedDict` shapes for Zarr v2 documents
(`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`)
- [`zarr_metadata.v3`](v3/index.md) β€” `TypedDict` shapes for Zarr v3
documents, with subpackages for [chunk grids](v3/chunk_grid.md),
[chunk key encodings](v3/chunk_key_encoding.md), [codecs](v3/codec.md),
and [data types](v3/data_type.md)

Every public name is also re-exported at the top level, so
`from zarr_metadata import ZarrV3ArrayMetadataJSON` and
`from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON` are equivalent.

## Common types

A few cross-cutting aliases are exported only from the top-level
`zarr_metadata` namespace:

::: zarr_metadata.JSONValue

::: zarr_metadata.ZarrV3NamedConfigJSON
5 changes: 5 additions & 0 deletions packages/zarr-metadata/docs/api/model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
title: model
---

::: zarr_metadata.model
5 changes: 5 additions & 0 deletions packages/zarr-metadata/docs/api/pydantic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
title: pydantic
---

::: zarr_metadata.pydantic
17 changes: 17 additions & 0 deletions packages/zarr-metadata/docs/api/v2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: v2
---

::: zarr_metadata.v2
options:
members: false

::: zarr_metadata.v2.array

::: zarr_metadata.v2.group

::: zarr_metadata.v2.attributes

::: zarr_metadata.v2.codec

::: zarr_metadata.v2.consolidated
11 changes: 11 additions & 0 deletions packages/zarr-metadata/docs/api/v3/chunk_grid.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: chunk_grid
---

::: zarr_metadata.v3.chunk_grid
options:
members: false

::: zarr_metadata.v3.chunk_grid.regular

::: zarr_metadata.v3.chunk_grid.rectilinear
11 changes: 11 additions & 0 deletions packages/zarr-metadata/docs/api/v3/chunk_key_encoding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: chunk_key_encoding
---

::: zarr_metadata.v3.chunk_key_encoding
options:
members: false

::: zarr_metadata.v3.chunk_key_encoding.default

::: zarr_metadata.v3.chunk_key_encoding.v2
25 changes: 25 additions & 0 deletions packages/zarr-metadata/docs/api/v3/codec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
title: codec
---

::: zarr_metadata.v3.codec
options:
members: false

::: zarr_metadata.v3.codec.blosc

::: zarr_metadata.v3.codec.bytes

::: zarr_metadata.v3.codec.cast_value

::: zarr_metadata.v3.codec.crc32c

::: zarr_metadata.v3.codec.gzip

::: zarr_metadata.v3.codec.scale_offset

::: zarr_metadata.v3.codec.sharding_indexed

::: zarr_metadata.v3.codec.transpose

::: zarr_metadata.v3.codec.zstd
45 changes: 45 additions & 0 deletions packages/zarr-metadata/docs/api/v3/data_type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: data_type
---

::: zarr_metadata.v3.data_type
options:
members: false

::: zarr_metadata.v3.data_type.bool

::: zarr_metadata.v3.data_type.int8

::: zarr_metadata.v3.data_type.int16

::: zarr_metadata.v3.data_type.int32

::: zarr_metadata.v3.data_type.int64

::: zarr_metadata.v3.data_type.uint8

::: zarr_metadata.v3.data_type.uint16

::: zarr_metadata.v3.data_type.uint32

::: zarr_metadata.v3.data_type.uint64

::: zarr_metadata.v3.data_type.float16

::: zarr_metadata.v3.data_type.float32

::: zarr_metadata.v3.data_type.float64

::: zarr_metadata.v3.data_type.complex64

::: zarr_metadata.v3.data_type.complex128

::: zarr_metadata.v3.data_type.raw

::: zarr_metadata.v3.data_type.bytes

::: zarr_metadata.v3.data_type.string

::: zarr_metadata.v3.data_type.numpy_datetime64

::: zarr_metadata.v3.data_type.numpy_timedelta64
15 changes: 15 additions & 0 deletions packages/zarr-metadata/docs/api/v3/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: v3
---

::: zarr_metadata.v3
options:
members: false

::: zarr_metadata.v3.ZarrV3MetadataFieldJSON

::: zarr_metadata.v3.array

::: zarr_metadata.v3.group

::: zarr_metadata.v3.consolidated
Loading
Loading