Skip to content

fix: minor correctness and hygiene fixes from the sync-pipeline audit - #259

Closed
d-v-b wants to merge 1 commit into
mainfrom
fix/perf-pr-minor-cleanups
Closed

fix: minor correctness and hygiene fixes from the sync-pipeline audit#259
d-v-b wants to merge 1 commit into
mainfrom
fix/perf-pr-minor-cleanups

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Bundles the minor findings from the PR zarr-developers#3885 audit — six independent items, one commit:

  1. Duplicated codec warnings: construction warnings (e.g. sharding's "disables partial reads") fired twice per array open — and on every decode/encode through the fused pipeline's async fallback. Re-constructions of an already-validated chain now use a quiet codecs_from_list_unchecked variant (named function per the two-semantics-two-names convention, no global warning-filter mutation). Regression test asserts exactly one warning per open under both pipelines; direct from_codecs construction still warns once.
  2. concurrent_iter laziness: it returned a generator of yet-to-be-created tasks while its docstring promised eager scheduling — a serialize-on-await trap for future callers. Now materializes the task list, with a flag-based eagerness test.
  3. max_workers env parsing: a garbage ZARR_CODEC_PIPELINE__MAX_WORKERS raised ValueError mid-read; now warns and falls back to the default (tolerant handling for config/env input).
  4. Abandoned in-flight tasks: the as-completed pipeline helpers left already-spawned fetch/write tasks running when one failed — stray background writes plus "Task exception was never retrieved" noise. Failures now cancel and drain outstanding tasks (single drain over the concatenated task sets, so outer cancellation can't skip half).
  5. Benchmark hygiene: seeded data generator (reproducible runs), fixed a copy-pasted docstring.
  6. Removed dead commented-out set_range_sync test blocks left behind by the pre-merge removal of that API.

One review note worth knowing: BatchedCodecPipeline.evolve_from_array_spec now propagates self.batch_size instead of re-resolving config — verified safe (all production callers construct via from_codecs, which resolves config at that point; the sharding inner-pipeline cache already keys on the config value).

Verification: warning test passes for both pipelines (fused failed pre-fix, TDD); 97 + 2042 targeted tests green; full repo suite 6589 passed pre-fix-round with the same items; mypy/ruff clean. Two-round task review (initial review caught the fused double-warn and a thread-safety hazard in the first approach; re-review verified both closed with no new issues).

🤖 Generated with Claude Code

@d-v-b
d-v-b force-pushed the fix/perf-pr-minor-cleanups branch from 15224a7 to 457c85c Compare July 29, 2026 12:04
@d-v-b

d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Merge-order note: this branch has small textual conflicts with #254 in src/zarr/core/chunk_utils.py and tests/test_fused_pipeline.py (adjacent edits). Whichever merges second needs a trivial rebase — ask and I'll do it.

@d-v-b

d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

Update: after #254 was narrowed (its async-only-inner-codec fix deferred to upstream zarr-developers#4179), the chunk_utils.py conflict is gone; only the trivial tests/test_fused_pipeline.py adjacent-append conflict with #254 remains.

- Codec construction warnings (e.g. sharding's "disables partial reads")
  fired twice per array open, and on every decode/encode through the fused
  pipeline's async fallback. Re-constructions of an already-validated codec
  chain now go through codecs_from_list_unchecked, which validates
  structure without repeating first-construction advisory warnings; each
  warning fires exactly once per open under both pipelines.
- concurrent_iter returned a lazy generator while its docstring promised
  eagerly scheduled tasks; it now materializes the task list so awaiting
  one at a time cannot serialize the batch.
- A garbage codec_pipeline.max_workers value (e.g. from the environment)
  raised ValueError mid-read; it now warns and falls back to the default,
  consistent with tolerant handling of config input.
- The as-completed pipeline helpers abandoned in-flight tasks when one
  failed, leaving stray background writes and "Task exception was never
  retrieved" warnings; failures now cancel and drain outstanding tasks.
- Benchmarks: seed the data generator for reproducibility; fix a
  copy-pasted docstring.
- Remove dead commented-out test blocks referencing the removed set_range
  API.

Assisted-by: ClaudeCode:claude-sonnet-5
@d-v-b
d-v-b force-pushed the fix/perf-pr-minor-cleanups branch from 6d30b53 to cdb0e00 Compare July 30, 2026 07:24
@d-v-b d-v-b closed this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant