multitrait changes

[petrelharp]

Update to SLiM v6. The docs build unit test is not expected to pass until SLiM v6 is released, since setting up the github docs build action to work with development head slim is more than I am up for at the moment.

Besides the updates to metadata, etcetera, a big thing here is trying to get across/make clear that accessing top-level metadata is now costly.

Summary of changes:

• some methods now take a ts_metdata argument, so if you've "pre-computed" this you can pass it in: individual_ages, individuals_alive_at, slim_time
• other methods take a private _ts_metadata argument for the same reason but just for internal use; these are methods that are involved in a workflow that rewrites the tree sequence anyhow, so cacheing and re-using the metadata wouldn't save much and could easily lead to errors; these are: has_vacant_samples, remove_vacant, restore_vacant,
• similarly, nucleotide_at takes an optional mut_metadata argument, which can be pre-computed
• deprecate node_is_vacant and add nodes_vacant; I haven't added a ts_metadata argument to this one since we expect people to call it only once
• added mutation_metadata to pull out a version of the mutation metadata indexed by SLiM id
• added add_mutation_metadata to add this to metadata, which is necessary after sim_mutations

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.20635% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.57%. Comparing base (4c228c9) to head (1003fb7).

Files with missing lines	Patch %	Lines
pyslim/slim_metadata.py	98.41%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #431      +/-   ##
==========================================
+ Coverage   96.50%   96.57%   +0.06%     
==========================================
  Files           8        8              
  Lines         773      904     +131     
  Branches      146      181      +35     
==========================================
+ Hits          746      873     +127     
- Misses         17       18       +1     
- Partials       10       13       +3     

Flag	Coverage Δ
python-tests	96.57% <99.20%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[petrelharp]

Note: docs are not building on CI because the docs require multitrait SLiM, and we're using the tskit action to build docs.

[petrelharp]

Back at this. After the (non-)resolution of cacheing metadata in tskit-dev/tskit#3475, I'm trying to minimize calls to ts.metadata. Here's the flowchart of which methods use top-level metadata:

[petrelharp]

The top-level metadata sure does affect runtime: for instance, without cacheing the test suite runtimes are:

========================================================================================== slowest 20 durations ===========================================================================================
22.77s call     tests/test_tree_sequence.py::TestReferenceSequence::test_nucleotide_at[recipe_chromosomes_adds_muts.slim]
13.30s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_chromosomes_adds_muts.slim]
11.33s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_recap[recipe_record_everyone_WF_late.slim]
10.48s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_everyone[recipe_record_everyone_WF_late.slim]
9.41s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_recap[recipe_record_everyone_WF_early.slim]
9.12s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_recap[recipe_record_everyone_WF_first.slim]
8.60s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_everyone[recipe_record_everyone_WF_first.slim]
8.57s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_everyone[recipe_record_everyone_WF_early.slim]
5.67s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_recap[recipe_record_everyone_nonWF_early.slim]
5.53s call     tests/test_tree_sequence.py::TestRecapitate::test_recapitation[recipe_many_chromosomes.slim]
5.07s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_everyone[recipe_record_everyone_nonWF_early.slim]
4.16s call     tests/test_tree_sequence.py::TestIndividualAges::test_ages[recipe_roots.slim]
3.95s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_simplify[recipe_record_everyone_WF_early.slim]
3.91s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_simplify[recipe_record_everyone_WF_late.slim]
3.63s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_simplify[recipe_record_everyone_WF_first.slim]
3.60s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_all_the_chromosome_types.slim]
3.53s call     tests/test_tree_sequence.py::TestReferenceSequence::test_nucleotide_spectrum[recipe_mutation_spectrum.slim]
2.53s call     tests/test_tree_sequence.py::TestHasIndividualParents::test_post_simplify[recipe_record_everyone_nonWF_early.slim]
2.28s call     tests/test_tree_sequence.py::TestVacancy::test_recapitate_keeps_vacant[recipe_chromosomes_adds_muts.slim]
1.94s call     tests/test_tree_sequence.py::TestConvertNucleotides::test_generate_nucleotides[recipe_chromosomes_adds_muts.slim]
=============================================================================== 1888 passed, 8 skipped in 217.24s (0:03:37) ===============================================================================
217.29user 2.11system 3:37.73elapsed 100%CPU (0avgtext+0avgdata 359656maxresident)k
2552inputs+132040outputs (0major+828171minor)pagefaults 0swaps

and with:

========================================================================================== slowest 20 durations ===========================================================================================
13.32s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_chromosomes_adds_muts.slim]
5.35s call     tests/test_tree_sequence.py::TestRecapitate::test_recapitation[recipe_many_chromosomes.slim]
4.16s call     tests/test_tree_sequence.py::TestIndividualAges::test_ages[recipe_roots.slim]
4.05s call     tests/test_tree_sequence.py::TestVacancy::test_recapitate_keeps_vacant[recipe_chromosomes_adds_muts.slim]
3.53s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_all_the_chromosome_types.slim]
2.40s call     tests/test_tree_sequence.py::TestVacancy::test_remove_restore_vacant_tables[recipe_chromosomes_adds_muts.slim]
2.37s call     tests/test_tree_sequence.py::TestVacancy::test_remove_restore_vacant[recipe_chromosomes_adds_muts.slim]
1.81s call     tests/test_tree_sequence.py::TestIndividualAges::test_ages[recipe_many_chromosomes.slim]
1.51s call     tests/test_tree_sequence.py::TestConvertNucleotides::test_generate_nucleotides[recipe_chromosomes_adds_muts.slim]
1.37s call     tests/test_tree_sequence.py::TestRecapitate::test_with_recomb_map[recipe_chromosomes_adds_muts.slim]
1.31s call     tests/test_tree_sequence.py::TestRecapitate::test_recapitation[recipe_chromosomes_adds_muts.slim]
1.15s call     tests/test_tree_sequence.py::TestConvertNucleotides::test_generate_nucleotides[recipe_with_traits.slim]
1.11s setup    tests/test_metadata.py::TestMultichrom::test_chromosome_types[recipe_chromosomes_adds_muts.slim]
1.09s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_with_traits.slim]
0.95s setup    tests/test_metadata.py::TestTreeSequenceMetadata::test_WF_model_type[recipe_roots.slim]
0.95s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_WF_Z.slim]
0.95s call     tests/test_metadata.py::TestMetadataSchemas::test_slim_metadata[recipe_many_chromosomes.slim]
0.93s call     tests/test_tree_sequence.py::TestVacancy::test_recapitate_keeps_vacant[recipe_many_chromosomes.slim]
0.86s setup    tests/test_tree_sequence.py::TestConvertNucleotides::test_convert_alleles[recipe_long_nucleotides.slim]
0.83s call     tests/test_tree_sequence.py::TestReferenceSequence::test_mutation_at[recipe_WF_X.slim]
=============================================================================== 1888 passed, 8 skipped in 98.32s (0:01:38) ================================================================================
97.83user 2.68system 1:38.78elapsed 101%CPU (0avgtext+0avgdata 457820maxresident)

So, that's 217s to 97s in total: not a big deal. But, some of those differ by 10x, and this is after going through and trying to minimize amount of multiple metadata accesses. It's all fine, but people will have to pay attention.

[gregorgorjanc]

I have read through the docs changes and suggested some minor edits or gave comments. Have not looked at the code.

[mufernando]

@petrelharp I gave a careful read to all the docs changes as well as the docstrings to the new functions. It all looks pretty clear to me. Thanks for the big push!!

[petrelharp]

This is big and unwieldy, I'm going to merge this and deal with subsequent issues separately.