Skip to content

fix: ManagedMemoryStore sync methods must respect the path prefix - #4204

Merged
d-v-b merged 5 commits into
zarr-developers:mainfrom
d-v-b:fix/store-sync-async-parity
Jul 30, 2026
Merged

fix: ManagedMemoryStore sync methods must respect the path prefix#4204
d-v-b merged 5 commits into
zarr-developers:mainfrom
d-v-b:fix/store-sync-async-parity

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fixes bugged behavior in the ManagedMemoryStore caused by failure to override x_sync methods inherited from MemoryStore, leading to the fused codec pipeline writing chunks to the wrong place in the store.

The tests required to fix this revealed a few other bugs in the managed memory store, which this PR fixes.

Assisted-by: ClaudeCode:claude-sonnet-5

based on a claude-authored PR: d-v-b#258

Summary

[Describe what this PR changes and why, in your own words.]

For reviewers

[What would you most value a second look at? What are you already confident in? For a refactor, say whether behavior is meant to be unchanged.]

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.91%. Comparing base (020e5c3) to head (bbf7af0).

Files with missing lines Patch % Lines
src/zarr/testing/store.py 94.28% 2 Missing ⚠️
src/zarr/storage/_memory.py 93.33% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4204   +/-   ##
=======================================
  Coverage   93.91%   93.91%           
=======================================
  Files          91       91           
  Lines       12678    12723   +45     
=======================================
+ Hits        11906    11949   +43     
- Misses        772      774    +2     
Files with missing lines Coverage Δ
src/zarr/storage/_memory.py 96.28% <93.33%> (-0.28%) ⬇️
src/zarr/testing/store.py 98.50% <94.28%> (-0.41%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@d-v-b
d-v-b force-pushed the fix/store-sync-async-parity branch from 4ee9723 to 78228b1 Compare July 29, 2026 14:32
ManagedMemoryStore inherited get_sync/set_sync/delete_sync from
MemoryStore, which use the raw key, while every async method prefixed
keys with self.path. Any code taking the sync fast path (e.g.
FusedCodecPipeline) wrote/read chunks outside the store's path prefix,
so a fresh handle re-reading through the prefix silently got fill
values. Override the three sync methods to prefix like their async
counterparts.

GpuMemoryStore.set_sync gets the same treatment: it now converts its
value to a gpu.Buffer like set does, preserving the store's
all-values-are-gpu invariant for the sync API.

Also fix ManagedMemoryStore.get_partial_values, which applied self.path
twice whenever path was non-empty (it pre-prefixed keys, then
delegated to MemoryStore.get_partial_values, which itself dispatches
through the already-overridden self.get) -- this made it return None
for every key. Discovered via the strengthened test fixture below.

Add sync/async parity laws to the shared StoreTests suite so every
store subclass exercises this invariant: set through one API and read
through the other (including byte_range variants), and confirm
delete_sync is visible to async get. These are the tests that would
have caught the ManagedMemoryStore bug.

TestManagedMemoryStore's raw set/get test helpers now respect self.path,
and store_kwargs uses a non-empty path, so prefix handling is actually
exercised instead of passing vacuously. Add an end-to-end regression
with FusedCodecPipeline writing to a ManagedMemoryStore(path=...)
sharing a dict with a fresh handle. Add a LocalStore.delete_sync
directory-branch test.

Assisted-by: ClaudeCode:claude-sonnet-5
@d-v-b
d-v-b force-pushed the fix/store-sync-async-parity branch from 78228b1 to 1a20c4c Compare July 29, 2026 14:35
@d-v-b d-v-b changed the title fix: ManagedMemoryStore sync methods must respect the path prefix - #258 fix: ManagedMemoryStore sync methods must respect the path prefix Jul 29, 2026
@d-v-b
d-v-b marked this pull request as ready for review July 29, 2026 14:38
@d-v-b

d-v-b commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

I will be self-merging this as it resolves a data corruption risk in the context of the fused codec pipeline

@d-v-b
d-v-b merged commit a88f889 into zarr-developers:main Jul 30, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant