Skip to content

Pin statistics across implementations in a parity test suite - #137

Merged
nspope merged 1 commit into
nsp-multiallelic-trunkfrom
test/implementation-parity
Jul 13, 2026
Merged

Pin statistics across implementations in a parity test suite#137
nspope merged 1 commit into
nsp-multiallelic-trunkfrom
test/implementation-parity

Conversation

@andrewkern

@andrewkern andrewkern commented Jul 13, 2026

Copy link
Copy Markdown
Member

Summary

Addresses #135. Adds tests/test_implementation_parity.py, a suite that asserts the same population-genetic statistic agrees across the code paths that compute it: scalar functions, FrequencySpectrum, and the windowed scatter, fused CUDA, and python-loop engines.

Targets nsp-multiallelic-trunk (this suite validates the multiallelic work), not main.

Each statistic is described once in a declarative _STATS registry; the parametrized test drives every supported path across four data conditions — clean, missing/include, missing/exclude, and multiallelic — and asserts it matches the scalar reference. 16 statistics, ~180 cases: 149 passed, 12 skipped, 19 xfailed. GPU-only (skipped without CUDA).

This is pure test infrastructure — no library code changes. Divergences between paths are recorded as strict=True xfails in _XFAILS with a mechanism-level reason; a (stat, path) may carry several condition-scoped rules. Fixing any divergence removes its rule and flips the cell to a pass.

Divergences this suite currently pins

  1. fst_wc (fused; all conditions) — the fused kernel was not converted to the per-allele one-vs-rest Weir-Cockerham ANOVA the scalar/python-loop path now uses.
  2. theta_w, tajimas_d (fused; missing/include) — the fused kernel's per-site sample-size handling under missing data still diverges from the scalar.
  3. tajimas_d, normalized_fay_wu_h, zeng_e (fs and scatter; missing/include) — the Achaz neutrality-test variance uses a modal / full-sample size rather than per-site valid counts.
  4. pi, theta_w, tajimas_d, segregating_sites, fay_wu_h, normalized_fay_wu_h, zeng_e (fs; multiallelic)FrequencySpectrum builds a per-allele SFS whose segregating-class count and pi differ from the scalar's per-allele counts.
  5. da (scatter; missing/exclude) — the within-population pi terms use a different site set than the scalar (dxy agrees).

Notes

  • The fused engine only runs under missing_data='include', so it is skipped (not failed) under the exclude condition.
  • Two-population fixtures use an even haplotype count per population, which fst_weir_cockerham requires for its diploid pairing.

@nspope

nspope commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

did you base off of #132 [that is, merge target is nsp-multiallelic-trunk]? I think that fixed the singleton and da and fst issues, at least

@andrewkern

Copy link
Copy Markdown
Member Author

shit, I did not. not me pull in those changes to this branch

@nspope

nspope commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

i'd just rebase on nsp-multiallelic-trunk, that's where i'm doing all the multiallelic stuff so I don't break main in the meantime

@andrewkern
andrewkern marked this pull request as draft July 13, 2026 20:37
Add tests/test_implementation_parity.py, which asserts that the same
population-genetic statistic agrees across the code paths that compute it:
scalar functions, FrequencySpectrum, and the windowed scatter, fused-CUDA, and
python-loop engines. Each statistic is described once in a declarative _STATS
registry; the parametrized test drives every supported path across four data
conditions (clean, missing/include, missing/exclude, multiallelic) and asserts
it matches the scalar reference. GPU-only, skipped without CUDA.

16 statistics, ~180 cases. Divergences between paths are recorded as strict
xfails in _XFAILS with a mechanism-level reason; a (stat, path) may carry
several condition-scoped rules. The remaining divergences are:

- fst_wc (fused, all conditions): the fused kernel was not converted to the
  per-allele one-vs-rest Weir-Cockerham ANOVA the scalar/python-loop path uses
- theta_w, tajimas_d (fused, missing/include): the fused kernel's per-site
  sample-size handling under missing data still diverges from the scalar
- tajimas_d, normalized_fay_wu_h, zeng_e (fs and scatter, missing/include): the
  Achaz neutrality-test variance uses a modal / full-sample size rather than
  per-site valid counts
- pi, theta_w, tajimas_d, segregating_sites, fay_wu_h, normalized_fay_wu_h,
  zeng_e (fs, multiallelic): FrequencySpectrum builds a per-allele SFS whose
  segregating-class count and pi differ from the scalar per-allele counts
- da (scatter, missing/exclude): the within-population pi terms use a different
  site set than the scalar (dxy agrees)

Refs #135
@andrewkern
andrewkern force-pushed the test/implementation-parity branch from 39b4996 to 2fe2fca Compare July 13, 2026 20:45
@andrewkern
andrewkern changed the base branch from main to nsp-multiallelic-trunk July 13, 2026 20:46
@andrewkern
andrewkern marked this pull request as ready for review July 13, 2026 21:52
@andrewkern

Copy link
Copy Markdown
Member Author

okay this should be ready to go now @nspope

@nspope
nspope merged commit 63572c6 into nsp-multiallelic-trunk Jul 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants