Regarding HapNe-LD: Integrating sample ages for custom non-AADR datasets

[NauseaCantWrite]

Hi HapNe team,

First off, thanks for the fantastic tool and congrats on the 2023 Nature Communications paper! I'm particularly thankful for the features that facilitate the handling of low coverage, genotyping errors, and pseudohaploid data, which are huge for the ancient DNA community.

I have a quick question regarding the best way to prep human and utilize HapNe's time-heterogeneity feature (if I understand correctly this only applies to HapNe-LD but I may be mistaken) using aDNA data not derived from the AADR database (e.g. one's own data).

Importantly, starting from VCF already (or PLINK files), we think of course about bypassing the eigenstrat2vcf step found in the pipeline.py script provided in your documentation. However, we want to ensure that HapNe-LD correctly incorporates the radiocarbon dates of our ancient samples to account for temporal distance during the LD calculation and we thought of creating a custom ".anno" text file that mimics the AADR format, but then we'd like to make sure if there are any other specific columns in the .anno file that get_age_from_anno absolutely requires to run successfully beyond the ID and Date columns? Then we would need to manually call get_age_from_anno just before the inference engine.

We also wonder if this could instead be achieved in a more direct way, passing an array of sample dates to the hapne_ld function without needing to mock an AADR .anno file?

Thanks for your time and for maintaining such a great resource!

[rmnfournier]

Hi!

The current time-correction indeed only applies to HapNe-LD, and should prevent spurious expansions in recent effective population size that is due to individuals being sampled at different time points.

To answer your question more specifically:

1. In src/backend/IO.py : the code tries to locate a file that has processed the age from the annofile
   default_loc = config["CONFIG"].get("output_folder")
   popname = config["CONFIG"].get("population_name")
   default_loc += f"/DATA/{popname}.age"
   self.time_heterogeneity_file = config['CONFIG'].get('age_samples', fallback=default_loc)
   if not exists(self.time_heterogeneity_file):
   self.time_heterogeneity_file = None
   else:
   self.time_heterogeneity_bin = config.getint("CONFIG", "age_samples_bin_width", fallback=1)

This file, "{popname}.age", is simply a file that contains two columns: FROM and TO. In utils.py, there is an example about how this file can be created. Note that the vales in this file are in generations;

age_from = (age_from_bp / years_per_gen).reshape((-1, 1)).astype(int)
age_to = (age_to_bp / years_per_gen).reshape((-1, 1)).astype(int)
to_save = pd.DataFrame(np.concatenate((age_from, age_to), axis=1), columns=["FROM", "TO"])
save_as = config["CONFIG"].get("output_folder")
popname = config["CONFIG"].get("population_name")
save_as += "/DATA"
# Create folder if it does not exist
if not exists(save_as):
    makedirs(save_as)
to_save.to_csv(save_as + f"/{popname}.age", index=False)

To make sure the data is loaded correctly, you can replace the last line of the code in the readme by what the function actually does:

hapne_ld(config)
becomes:
config["CONFIG"]["method"] = "ld"
hapne = HapNe(config)
print(hapne.io.time_heterogeneity) # -> should not be None
hapne.fit()

Feel free to reach out by email if you need more specific help !

Best,
Romain