Skip to content

Add full X-chromosome example dataset as VCF Zarr - #165

Merged
andrewkern merged 1 commit into
mainfrom
data/reencode-example-x-vcz
Jul 29, 2026
Merged

Add full X-chromosome example dataset as VCF Zarr#165
andrewkern merged 1 commit into
mainfrom
data/reencode-example-x-vcz

Conversation

@andrewkern

Copy link
Copy Markdown
Member

Closes #161.

examples/scikit_allel_comparison.py defaults to examples/data/gamb.X.phased.n100.zarr, which was never tracked in the repo. .gitignore excludes data/, and only the 4 Mb subset store was ever force-added. A fresh clone could run the example with --small and no other way.

Rather than commit the original store as-is, this ships it re-encoded with bio2zarr. The original was a scikit-allel-era Zarr v2 store carrying all 26 VCF fields at full 5.3M-variant length with 1000-variant chunks, which is what produced 90,126 files for 55 MB of data.

Two changes: drop the ten fields pg_gpu/zarr_io.py never reads (call_PQ, call_PS, variant_AA, variant_AAProb, variant_AAcond, variant_AC, variant_AF, variant_AN, variant_CM, variant_quality), and raise the variant chunk size from 1000 to 100000.

original store this PR
Data 55 MB 15 MB
Checkout footprint 356 MB 16 MB
Files 90,126 411
Git objects 47,878 220
Added to pack 52.8 MiB 14.0 MiB
Load time 20.3 s 7.9 s

For scale, the 4 Mb subset store already in the repo is 17,400 files; this full chromosome is 411.

Verification

Loaded both stores and compared:

shape     identical=True   (200, 5297731)
samples   identical=True   ['AA0045-C', 'AA0046-C']
pos0/posN identical=True   3 / 24392613
pi        identical=True   0.0021073932663811298
theta_w   identical=True   0.008740797101718526
tajimasD  identical=True   -2.4556936206779456

The example itself runs end-to-end against the new store, with all of its internal cross-checks against scikit-allel passing:

Loading gamb.X.phased.n100.vcz ...
  loaded in 8.50s; 200 haplotypes, 5,297,731 variants, chrom range 3-24,392,613
  pi:        max abs diff = 3.339e-17
  theta_w:   max abs diff = 1.492e-16
  tajimas_d: max abs diff = 3.286e-14
  median r^2: max abs diff = 1.514e-08

Notes

  • The file is renamed .zarr to .vcz to match the bio2zarr convention; the example's path constant is updated to match.
  • The exact vcf2zarr commands that regenerate the store are recorded in a comment at the top of the example, so it is reproducible from the VCF already in examples/data/.
  • The missing-file error text told users to run git lfs pull, but this repo does not use LFS. Corrected to point at the regeneration commands.
  • The same treatment would shrink the 4 Mb subset store considerably as well; left out of scope here.

examples/scikit_allel_comparison.py defaults to the full Anopheles
X-chromosome store, but that store was never tracked, so a fresh clone
could only run the example with --small.

Ship it re-encoded with bio2zarr rather than in the original
scikit-allel layout. Dropping the ten fields pg_gpu never reads and
raising the variant chunk size from 1000 to 100000 takes the store from
55 MB in 90,126 files down to 15 MB in 411, shrinks the checkout
footprint from 356 MB to 16 MB, and cuts load time from 20.3s to 7.9s.
pi, theta_w and Tajima's D are bit-identical to the old store.

Also correct the missing-file error text, which pointed at `git lfs
pull` in a repo that does not use LFS.
@andrewkern
andrewkern merged commit bf47f21 into main Jul 29, 2026
1 check passed
@andrewkern
andrewkern deleted the data/reencode-example-x-vcz branch July 29, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing file in examples/data

1 participant