Add full X-chromosome example dataset as VCF Zarr - #165
Merged
Conversation
examples/scikit_allel_comparison.py defaults to the full Anopheles X-chromosome store, but that store was never tracked, so a fresh clone could only run the example with --small. Ship it re-encoded with bio2zarr rather than in the original scikit-allel layout. Dropping the ten fields pg_gpu never reads and raising the variant chunk size from 1000 to 100000 takes the store from 55 MB in 90,126 files down to 15 MB in 411, shrinks the checkout footprint from 356 MB to 16 MB, and cuts load time from 20.3s to 7.9s. pi, theta_w and Tajima's D are bit-identical to the old store. Also correct the missing-file error text, which pointed at `git lfs pull` in a repo that does not use LFS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #161.
examples/scikit_allel_comparison.pydefaults toexamples/data/gamb.X.phased.n100.zarr, which was never tracked in the repo..gitignoreexcludesdata/, and only the 4 Mb subset store was ever force-added. A fresh clone could run the example with--smalland no other way.Rather than commit the original store as-is, this ships it re-encoded with bio2zarr. The original was a scikit-allel-era Zarr v2 store carrying all 26 VCF fields at full 5.3M-variant length with 1000-variant chunks, which is what produced 90,126 files for 55 MB of data.
Two changes: drop the ten fields
pg_gpu/zarr_io.pynever reads (call_PQ,call_PS,variant_AA,variant_AAProb,variant_AAcond,variant_AC,variant_AF,variant_AN,variant_CM,variant_quality), and raise the variant chunk size from 1000 to 100000.For scale, the 4 Mb subset store already in the repo is 17,400 files; this full chromosome is 411.
Verification
Loaded both stores and compared:
The example itself runs end-to-end against the new store, with all of its internal cross-checks against scikit-allel passing:
Notes
.zarrto.vczto match the bio2zarr convention; the example's path constant is updated to match.vcf2zarrcommands that regenerate the store are recorded in a comment at the top of the example, so it is reproducible from the VCF already inexamples/data/.git lfs pull, but this repo does not use LFS. Corrected to point at the regeneration commands.