Warn about Zarr layouts that make viewing awkward - #430
Conversation
Check the dataset layout client-side from metadata that is already parsed, and surface the result through the MetadataHint notifications used for broken metadata. All three checks are advisory - nothing is withheld, and viewer links are untouched. Resolution levels: a multiscales group declaring a single dataset provides no downsampled data, so every zoom reads full-resolution chunks. This was the root cause of the production incident behind these checks. Keyed on the dataset count rather than the number of shapes, because a plain zarr array also has exactly one shape while claiming nothing about being multiscale. Chunk size: chunks past the browser's cache entry limit are re-fetched on every access. Sizes are computed from shape and dtype, so they are logical: exact for a raw array, an upper bound for a compressed one. The limit is 10 MB when no compressor is present and 32 MB when one is, since the compression ratio is a property of the pixels and cannot be derived from metadata. Unknown or unfetched codec info counts as compressed, so uncertainty costs a missed warning rather than a false one. Codec classification reuses capability-manifest's classifyCodec, plus a walk over the pipeline it does not do itself: a sharded v3 array lists only the structural sharding_indexed codec at the top level and carries the real compressor in its configuration. Plain arrays now carry codec info too - the v3 branch reads it from the zarr.json it already parsed, the v2 branch fetches .zarray as the group path does. Axis order: OME-Zarr requires T, C, Z, Y, X order, and many tools take the last two axes to be the image plane. A dataset ordered C, X, Y, Z therefore renders as a Y-Z cross-section in those tools while Neuroglancer, which reads the axis names, looks fine - a confusing failure worth naming explicitly. Datasets using custom axis names are left alone, since their intent cannot be judged. No thumbnail check: ome-zarr.js already refuses when the lowest level exceeds its own limit, and modelling its behaviour a second time proved unreliable - it renders from the level nearest the requested size rather than the smallest, and reads a single plane rather than the whole level. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
At some point, we may want to provide virtualization for Zarr datasets like this as well. |
mkitti
left a comment
mkitti
left a comment
There was a problem hiding this comment.
I am not sure how well these warnings are grounded in the spec and perhaps more specifically in Neuroglancer's implementation. I'm not sure where the 10 MiB cache limit comes from. Could we check the cache limits and add citations when possible to clearly state whether these are browser or middleware issues.
I suspect that the dimension order could be relaxed here especially in light of RFC-3. The main relevant ordering is probably the clustering of the spatial dimensions at the end but the order of the spatial dimensions is not really a problem for Neuroglancer.
Another warning I would add here is the dimensionality of the chunks versus the dimensionality of the array. If we have 2D chunks and a 3D spatial array, we will have issues for viewing in some of the orthogonal views. If the array is nD, then we want the dimension of the chunks to have non-singleton dimensions for at leastmin(n, 3) dimensions.
Also chunks that are too small might be an issue as well. In the extreme a chunk that is a single voxel is highly problematic. I would suggest a minimum element count of 256 elements if there are more elements than this in the array.
| return { | ||
| kind: 'warning', | ||
| title: 'Axes are not in the order OME-Zarr specifies', | ||
| description: `The axes are ordered ${variant.axisOrder}, but the spec requires T, C, Z, Y, X order - here that would be ${variant.expectedOrder}. Many tools take the last two axes to be the image plane, so they will show a cross-section rather than the expected view. Rewriting the dataset with the axes in spec order avoids this.` |
There was a problem hiding this comment.
RFC 3 will likely make this advice obsolete with respect to the spec statement:
There was a problem hiding this comment.
I know that, but in the meantime it is creating issues, specifically with ome-zarr.js (and I would bet there are other cases).
For example: https://github.com/BioNGFF/ome-zarr.js/blob/main/src/image.ts#L336-L337
| return { | ||
| kind: 'warning', | ||
| title: 'Chunks may be too large for efficient viewing', | ||
| description: `This dataset uses ${variant.size} chunks ${variant.compressed ? '(before compression)' : '(without compression)'}. Chunk files larger than the browser's cache limit are re-downloaded on every access, which makes viewing slow. A final chunk size of 1-10 MB works best.` |
There was a problem hiding this comment.
We may need to qualify this for shards. Shards are also large top-level chunks.
There was a problem hiding this comment.
I'm not sure I follow. Shards are on the server side, the client side deals with chunks. Why would shard size matter to the client?
There was a problem hiding this comment.
A shard is also a chunk. It is just a specific kind of chunk that contains other "inner" chunks. To specify the size of a "shard" in Zarr I say chunk_shape: [1024, 1024, 1024].
Your warning shoud be qualified to say "inner" chunk perhaps.
| /** | ||
| * Chunks above this defeat the browser cache. Applies when the array is stored | ||
| * without compression, so the size we compute is the size that transfers. | ||
| */ | ||
| export const MAX_CHUNK_BYTES = 10 * 1024 ** 2; | ||
| /** |
There was a problem hiding this comment.
I'm not sure if this limit is the one that would apply to Zarr chunks.
https://claude.ai/share/2c1c2b4f-c9a3-43d3-a24d-9de809b04062
There was a problem hiding this comment.
Some browsers do have a limit on the size of a file they will cache. For example, Firefox will only cache up to 5 MB files in memory (browser.cache.memory.max_entry_size) and 50 MB on disk (browser.cache.disk.max_entry_size). This 10 MB threshold is uncompressed, and assumes 2-3x compression will put the chunk under the 5 MB threshold. It's just a heuristic.
Of course, there are other reasons to favor smaller chunk sizes, but not too small. Maybe there are better heuristics or rules we can use here.
There was a problem hiding this comment.
I would suggest either softening the warning copy to something like "may exceed some browsers' in-memory HTTP cache limits, causing slower (disk-cache or re-fetch) access" rather than stating a re-fetch as the consequence, or dropping the specific causal mechanism and just framing the threshold as a general chunk-size-for-viewing-performance heuristic (which is closer to where the 64 MB / 32 MB revision in this thread is already headed, and is more defensible than pinning it to one browser's memory-tier default).
Browsers have limits at 5 MB, as explained in the code comment. This thread suggests that reasonable chunk file sizes are between 1-10MB. What we want to do here is to provide a soft warning to people when they use chunk sizes that are clearly not ideal. Where would you set that threshold without creating too many false positives? I don't expect that this threshold can be a perfect classifier for correct/wrong, since there are many different (and often opposing) forces acting upon it.
But RFC-3 is not a part of any official specification version, and I'm not sure when it would be merged. Even after it is released, it will take time for libraries (such as ome-zarr.js) to catch up with it. In the meantime, it creates issues when the spec is not followed. Although there may be good reasons for deviating from the spec (and maybe some of those inspired RFC-3), I don't see the problem with alerting people that they are deviating.
Yes, but that seems out of scope for this PR, which is trying to address concrete issues that happened in production.
Yes, but also out of scope here. |
|
Some minor suggestions for the wording of the warnings:
|
Amazon's "Best Practices Design
https://d1.awsstatic.com/whitepapers/AmazonS3BestPractices.pdf 32 MiB is the upper limit I usually recommend, especially in the absence of sharding. I typically recommend 256³ for uint16 datasets on microscope systems running Windows when I anticipate relatively high file overhead from the Windows file system and the pending remote transfer of the entire dataset. 1 MB chunk sizes is the target I recommend when on systems where the number of files is not an acute constraint. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-best-practices.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-best-practices.html My main recommendation for Zarr v3 is 1 GiB shards (hard upper limit is 2 GiB to support 32-bit systems that address memory or arrays via signed 32-bit integers, such as Java and JavaScript). Scalable Minds recommends 32³ to 128³ voxel chunks within the shard, preferring smaller chunks for smaller datasets. https://docs.webknossos.org/webknossos/data/zarr.html The Keller Lab, Fernando Amat, specifically, found optimal chunk sizes around 96 x 96 x 8 in shape for KLB. https://www.nature.com/articles/nprot.2015.111/figures/9 So depending on the underlying systems and intended uses we recommendations spanning 32³ to 256³ voxels and 32 KiB to 32 MiB in terms of bytes. It would be worthwhile to do some study on this with systematic benchmarks to make sure we understand the impact of chunk size on our current systems and the attribution of this across various layers and applications. |
Conversion and data copies can get expensive. If our main target application is Neuroglancer or other viewers intended for volumetric display (BDV, BigStitcher, Imaris), I do not see the cost of conversion or copies as being worthwhile here for permutations of the spatial dimensions within the lowest strided dimensions. The original OME recommendations are based on viewing applications such as OMERO which focus on 2D display and are carried over from OME-TIFF. My main concern here is that the chunks are 3D and isotropic. In that case, the permutation of XYZ should not strongly matter. It seems that the main concern is for thumbnail generation. However, I find the assumption of which two dimensions to preview to be conceptually flawed. If we have a true isotropic volume, then a preview should be either 3D or allow for multiple 2D slices / projections (XY, YZ, ,XZ). |
|
My question here is whether these issues are actually intrinsic to the array. In many cases, I think we could adapt the display to the array properties. Do we only have one scale level or large chunks? Zoom-in to display fewer chunks at a time. Are the chunks not isotropic? Orient the initial display for the optimal two dimensions. Perhaps do not show all three orthogonal dimensions if this would result in loading multiple chunks. If we have XY tiles with no Z depth, then only show the XY plane perspective. |
I will increase the chunk size threshold for warnings from 10 MB to 64 MB, which should account for compression down to ~32 MB. That would still catch the 128 MB chunk issue we saw, while accommodating the larger chunks you use in practice. I'd also be interested in understanding the cases where those larger chunk sizes are beneficial, since that could help inform the guidance we give.
I agree that there are limitations in this implementation, but it does follow the existing specification, so for now I think it makes sense to keep it as-is. When the spec is changed (e.g. when RFC-3 is merged) then we will need to revisit how thumbnails are implemented.
For this PR I'm trying to keep the scope focused on providing practical advice that can be useful to users today. It would be helpful to capture some of the adaptive-display ideas separately, so that we can consider them independent of this PR. |
|
Here's the OME-Zarr to roadmap to v1.0: 
RFC-3 related comments:
As shown in the roadmap, this is the focus of primary activity in the community right now. The objective is to finalize RFC-3 this month.
This is what is confusing me. I do not see the axis order advice as practical. It does not affect Neuroglancer. I do not think people should waste time or resources on axis order issues. Even worse, I think the axis order advice is suboptimal for many applications. For example, if you have a true RGB image or RGBA image often these are expected to be the last dimension as exemplified by these libraries:
While it is true that many images coming our way are from fluoresence microscopy are not in RGB format, fluorescence microscopy is the exception. Most histology images (e.g. H&E sections) for example are in RGB format. Regarding ome-zarr.js, I do see a lot of code that does not assume the order of the dimensions:
I submitted a pull request to address the parts of the code where the order of the dimensions is assumed: I would particularly focus on the chunk issues for the warnings since there really is not any solution that we can implement short of rewriting the entire array. The single resolution issue can be resolved by adding the additional scales by materializing them to disk or calculating them on the fly in some cases, which is a service we might be able to provide. The dimension order issue is at best superfluous and at worse deleterious to potential applications. The dimension order issue should just be left to a OME-NGFF validator and hopefully removed from the spec as soon as possible. If a tool assumes the last two dimensions are Y and X despite clear metadata to contrary, it is a bug that we can and should fix. |
3 participants
Adds three advisory warnings to the Zarr preview panel, flagging layout choices that make a dataset expensive to view. They run client-side on metadata fileglancer already parses, so there are no extra requests on the common path.
Motivation is the production incident that occurred on Friday, August 14th. The IO thread pools were starved by a handful of viewing sessions on a 91 GB image published with a single resolution level. Nothing here changes server behaviour — the aim is to tell data creators before they publish.
The checks
Only one resolution level. A multiscales group declaring a single dataset provides no downsampled data, so every zoom reads full resolution. Keyed on the dataset count rather than the number of shapes, because a plain zarr array also has exactly one shape while claiming nothing about being multiscale — an early version keyed on shapes and fired on healthy pyramid levels that users had browsed into. Gated at 1 GB.
Chunks too large. Chunks past the browser's cache entry limit are re-fetched on every access. Sizes come from shape and dtype, so they are logical: exact for a raw array, an upper bound for a compressed one. Limit is 10 MB with no compressor and 32 MB with one, since the compression ratio is a property of the pixels and is not in the metadata. Unknown or unfetched codec info counts as compressed, so uncertainty costs a missed warning rather than a false one.
Codec classification reuses
capability-manifest'sclassifyCodec, plus a walk over the pipeline it does not do itself: a sharded v3 array lists only the structuralsharding_indexedcodec at the top level and carries the real compressor in its configuration, so a flat scan calls every sharded array uncompressed.Axis order. OME-Zarr requires
t, c, z, y, x, and many tools take the last two axes to be the image plane. A dataset orderedx, y, z, crenders as a cross-section in those tools while Neuroglancer, which reads axis names, looks fine — a confusing failure worth naming. Datasets with custom axis names are left alone.Screenshot
Notes for review
ome-zarr.jsalready refuses when the lowest level exceeds its own limit. Modelling its behaviour a second time proved unreliable: it renders from the level nearest the requested size rather than the smallest, and reads a single plane rather than a whole level.zarrQueries.tsnow populates codec info for plain arrays too — the v3 branch reads it from thezarr.jsonit already parsed, the v2 branch fetches.zarrayas the group path does.__tests__/mocks/omezarrHelper.tsis a whole-module mock, so it needs every new export listed or the component tests fail.Testing
21 unit tests covering both compression paths, nested sharding codecs, unknown codecs, each axis-order case, and the plain-array cases that earlier versions got wrong. Full frontend suite passes (321). Verified end to end against a pair of synthetic fixtures that reproduce the original dataset's defects and the corrected version — the broken one returns all three warnings, the fixed one returns none.
@StephanPreibisch @JaneliaSciComp/fileglancer @dchen116