Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
456d2fe
fix: force multi-line scan path for records with blank lines
Paururo Jul 18, 2026
505fd4b
fix(vcf): render gap reference as '*' instead of a bare '-'
Paururo Jul 18, 2026
37bbf0c
fix: count all-gap columns as ambiguous under --include-gaps
Paururo Jul 18, 2026
554f2b9
fix: accept bare output filenames without a directory component
Paururo Jul 18, 2026
b1e17a8
perf(vcf): assemble each data row in a reused byte buffer
Paururo Jul 18, 2026
f8fe607
feat(cli): add --threads to pin the Rayon thread pool
Paururo Jul 18, 2026
01723cf
feat(vcf): add --chrom to set the CHROM / contig name
Paururo Jul 18, 2026
470ed99
feat: emit a header-only VCF when there are no variable sites
Paururo Jul 18, 2026
152f3f5
feat(cli): add --quiet to silence progress logs
Paururo Jul 18, 2026
64e7e93
test: cover parallel scan, N-ref fallback, lowercase, CRLF, length mi…
Paururo Jul 18, 2026
7f780ae
ci: gate the release build on the tag-exists check
Paururo Jul 18, 2026
baa4a26
ci: run CI on the 1.0.2 release branch
Paururo Jul 18, 2026
27ac5a2
docs: fix pre-built binary download links and note gap encoding
Paururo Jul 18, 2026
ae60a94
docs: add CHANGELOG and link it from the README
Paururo Jul 18, 2026
5d0c6b0
chore: release 1.0.2
Paururo Jul 18, 2026
0347e9e
docs: correct snpick description in CONTRIBUTING
Paururo Jul 18, 2026
835a826
docs: add MkDocs Material documentation site
Paururo Jul 18, 2026
c8345b4
docs: enrich the site with Material feature set
Paururo Jul 18, 2026
d49d453
docs: enlarge the header logo
Paururo Jul 18, 2026
d3bacb3
docs: add an executable Jupyter tutorial
Paururo Jul 18, 2026
4dfe081
docs: fix tutorial cross-links to resolve on the built site
Paururo Jul 18, 2026
92aaa3f
docs: link to the documentation site from the README
Paururo Jul 18, 2026
b4d5c8b
chore: add package metadata, richer --help, and a CI test matrix
Paururo Jul 18, 2026
bcbdc96
feat: add --reference for polarity and guard against bad sequence IDs
Paururo Jul 18, 2026
e10da6d
feat(cli): add --stats-json and --dry-run
Paururo Jul 18, 2026
48f84fd
feat: add --keep-samples / --exclude-samples
Paururo Jul 18, 2026
9786a5c
feat: per-site filtering (--max-missing, --mac, --maf, --min-samples,…
Paururo Jul 18, 2026
0ac202a
feat: add --mask to exclude BED regions from the analysis
Paururo Jul 18, 2026
3aad9a7
feat: reference-anchored coordinates (--ref-coords, --sites-output)
Paururo Jul 18, 2026
f7c4589
feat: add --on-invalid and --check composition audit
Paururo Jul 18, 2026
1a53e38
feat: add --iupac-mode to optionally resolve ambiguity codes
Paururo Jul 18, 2026
6606b5a
feat: transparent gzip input and stdin/stdout streaming
Paururo Jul 18, 2026
22c8db9
feat: add --format for PHYLIP and NEXUS output
Paururo Jul 18, 2026
4111d33
perf: parallel prefault and an auto-vectorizable scan kernel
Paururo Jul 18, 2026
1c4c2e5
feat(cli): add -v/-vv verbosity and distinct exit codes
Paururo Jul 18, 2026
9f0ab0d
feat(cli): show an extraction progress line on interactive terminals
Paururo Jul 18, 2026
5a06d1b
refactor: expose the core as a library crate (lib.rs)
Paururo Jul 18, 2026
0c3f90e
test: add proptest, a criterion benchmark, and a cargo-fuzz target
Paururo Jul 18, 2026
426c6c2
build: add Docker and Apptainer container recipes
Paururo Jul 18, 2026
41a4943
feat(python): add pyo3 bindings built on the library crate
Paururo Jul 18, 2026
1eb4d03
feat(wasm): add WebAssembly bindings and make rayon/mmap optional
Paururo Jul 18, 2026
ed0cbfb
docs: document the full 1.0.2 feature set in the README and site
Paururo Jul 18, 2026
74a18fd
fix: guard --stats-json/--sites-output against clobbering the input
Paururo Jul 18, 2026
81cb858
fix(input): tolerate short-read gzip magic and clean up temp on failure
Paururo Jul 18, 2026
8dc5e5e
fix(vcf): valid gap REF, consistent reference coordinates, u32-safe POS
Paururo Jul 18, 2026
ad0103f
fix(cli): emit --sites-output under --dry-run
Paururo Jul 18, 2026
d5eed96
fix(filter): count the resolved allele set under --iupac-mode resolve
Paururo Jul 18, 2026
77ec6cf
fix(nexus): quote taxon labels containing NEXUS metacharacters
Paururo Jul 18, 2026
38792bc
fix: sidecar collision, fraction validation, BED chrom, sites stdout
Paururo Jul 18, 2026
92ffa24
docs: note duplicate VCF POS for insertion columns under --ref-coords
Paururo Jul 18, 2026
cc7f223
fix(vcf): stream --vcf-output - to stdout like the other outputs
Paururo Jul 19, 2026
41dcb5b
fix(cli): skip output-path checks under --check and guard stdout coll…
Paururo Jul 19, 2026
d1d6266
fix(samples): treat every @file line as a literal sample ID
Paururo Jul 19, 2026
3a66e57
fix(input): keep gzip detection rewind-safe for non-seekable paths
Paururo Jul 19, 2026
e4bad85
fix(coords): compare reference BED intervals without truncating to u32
Paururo Jul 19, 2026
44c378a
fix(vcf): keep the contig length consistent with the clamped POS
Paururo Jul 19, 2026
730f70c
docs(vcf): correct POS default, ##reference header, and gap-REF encoding
Paururo Jul 19, 2026
43dd575
docs: document stdout ('-') support for every output sink
Paururo Jul 19, 2026
e719212
docs(benchmarks): reconcile the figures with benchmarks/results.tsv
Paururo Jul 19, 2026
b1b7dbc
docs(architecture): qualify the zero-copy and O(L)-memory claims
Paururo Jul 19, 2026
68a86a5
docs: state MSRV 1.74, fix README byte count, date changelog releases
Paururo Jul 19, 2026
72f68d7
docs(tutorial): note that a gap in REF is written N, not *
Paururo Jul 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
target/
fuzz/target/
site/
docs/
benchmarks/
logo/
.git/
.github/
.cache/
bindings/*/target/
8 changes: 5 additions & 3 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@ Thank you for your interest in contributing to **snpick**! We welcome contributi
- [Code of Conduct](#code-of-conduct)

## Project Description
`snpick` is a tool designed to identify **Multi-Nucleotide Variants (MNVs)** within the same codon in genomic sequences. MNVs occur when multiple Single Nucleotide Variants (SNVs) are present within the same codon, leading to the translation of a different amino acid. This tool addresses limitations in current annotation programs like **ANNOVAR** or **SnpEff**, which are primarily designed to work with individual SNVs and might overlook the actual amino acid changes resulting from MNVs.
`snpick` is a fast, memory-efficient tool for extracting variable (SNP) sites from whole-genome **FASTA alignments**. It produces reduced alignments ready for phylogenetic inference with ascertainment-bias correction (ASC) in **IQ-TREE** and **RAxML**, and can optionally emit a VCF file. Its zero-copy, memory-mapped architecture scales to thousands of genomes in seconds with minimal RAM, where matrix-in-memory tools such as **snp-sites** struggle.

### Current Limitations
**IMPORTANT**: This script currently works only with **SNVs** against a reference genome. Insertions and deletions that modify the reading frame are not supported yet.
### Scope
- Input is a **FASTA alignment**: every sequence must have the same length. The first sequence is used as the reference for `REF`/`ALT` polarization.
- Standard bases `A/C/G/T` (case-insensitive) define variability; IUPAC ambiguous bases (N, R, Y, …) are treated as missing data rather than alleles.
- Gaps (`-`) are ignored by default and can be included as a 5th character with `-g`.

## How to Contribute
We appreciate all contributions, whether it’s fixing bugs, proposing new features, improving the documentation, or suggesting a new direction for the tool.
Expand Down
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Pull Request Template for get_MNV
# Pull Request Template for SNPick

## Description
Please include a summary of the changes made in this pull request. Mention any related issues or features that are being addressed.
Expand Down
40 changes: 40 additions & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Container

on:
push:
tags: [ "[0-9]+.[0-9]+.[0-9]+" ]
workflow_dispatch:

permissions:
contents: read
packages: write

jobs:
docker:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository_owner }}/snpick
tags: |
type=ref,event=tag
type=raw,value=latest

- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
43 changes: 43 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: Docs

on:
push:
branches: [ main ]
paths:
- 'docs/**'
- 'overrides/**'
- 'mkdocs.yml'
- 'CHANGELOG.md'
- '.github/CONTRIBUTING.md'
- '.github/workflows/docs.yml'
workflow_dispatch:

permissions:
contents: write

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# Full history so git-revision-date-localized can read commit dates.
fetch-depth: 0

- uses: actions/setup-python@v5
with:
python-version: '3.x'
cache: pip

- name: Install Cairo/Pango (social cards)
run: |
sudo apt-get update
sudo apt-get install -y libcairo2-dev libfreetype6-dev libffi-dev \
libjpeg-dev libpng-dev libz-dev pngquant

- name: Install MkDocs Material and plugins
run: pip install -r docs/requirements.txt

- name: Build and deploy to gh-pages
# GitHub Actions sets CI=true, which enables the social-cards plugin.
run: mkdocs gh-deploy --force
7 changes: 5 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ jobs:
outputs:
version: ${{ steps.version.outputs.version }}
tag: ${{ steps.version.outputs.tag }}
skip: ${{ steps.tagcheck.outputs.skip }}
steps:
- uses: actions/checkout@v4

Expand All @@ -32,16 +33,17 @@ jobs:
echo "📦 Version: $VERSION"

- name: Check if tag already exists
id: tagcheck
run: |
if git ls-remote --tags origin | grep -q "refs/tags/${{ steps.version.outputs.tag }}$"; then
echo "⚠️ Tag ${{ steps.version.outputs.tag }} already exists — skipping release."
echo "SKIP=true" >> "$GITHUB_ENV"
echo "skip=true" >> "$GITHUB_OUTPUT"
fi

# Build matrix for cross-platform binaries
build:
needs: release
if: needs.release.outputs.version != ''
if: needs.release.outputs.version != '' && needs.release.outputs.skip != 'true'
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -94,6 +96,7 @@ jobs:
# Create GitHub release with all binaries
publish:
needs: [release, build]
if: needs.release.outputs.skip != 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down
39 changes: 27 additions & 12 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,42 @@ name: Rust

on:
push:
branches: [ "main", "1.0.1" ]
branches: [ "main", "1.0.2" ]
pull_request:
branches: [ "main" ]

env:
CARGO_TERM_COLOR: always

jobs:
build:
runs-on: ubuntu-latest
test:
name: Test (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ ubuntu-latest, ubuntu-24.04-arm, macos-14, macos-15-intel ]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v4

- uses: dtolnay/rust-toolchain@stable
with:
components: clippy

- name: Clippy
run: cargo clippy -- -D warnings
- name: Clippy
run: cargo clippy -- -D warnings

- name: Build
run: cargo build --verbose
- name: Test
run: cargo test --verbose

- name: Test
run: cargo test --verbose
- name: Build (release)
run: cargo build --release

- name: Build (release)
run: cargo build --release
msrv:
name: MSRV 1.74
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@1.74.0
- name: Build on the minimum supported Rust version
run: cargo build --verbose
17 changes: 17 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,20 @@ test_*.fasta
test_*.vcf
*.vcf
!src/

# MkDocs build output
site/

# MkDocs Material social-cards / plugin cache
.cache/

# cargo-fuzz
fuzz/target/
fuzz/corpus/
fuzz/artifacts/
fuzz/Cargo.lock

# binding crates
bindings/*/target/
bindings/*/Cargo.lock
bindings/wasm/pkg/
82 changes: 82 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Changelog

All notable changes to SNPick are documented here. The format is based on
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.2] - 2026-07-18

### Added

**Filtering & selection**
- Per-site filtering: `--max-missing`, `--mac`, `--maf`, `--min-samples`,
`--max-alleles`. Filtered sites stay variable (never re-enter `fconst`), so ASC
correction remains valid.
- `--keep-samples` / `--exclude-samples` (comma-separated IDs or `@file`) subset
the panel before the scan, so `fconst` is recomputed for the retained samples.
- `--mask <BED>` (and `--mask-ref`) excludes regions from both the output and `fconst`.

**Coordinates & references**
- `--reference <ID>` chooses the REF/polarity sequence and the VCF `##reference`.
- `--ref-coords` writes VCF `POS` as ungapped reference positions.
- `--sites-output <TSV>` maps each site's alignment column to its reference position.

**I/O & formats**
- Transparent gzip/bgzip input, plus stdin (`-f -`) and stdout (`-o -`) streaming.
- `--format {fasta,phylip,nexus}` for the reduced alignment.
- `--stats-json <FILE>` (or `-`) writes a typed JSON run summary with the `fconst` array.

**Robustness & QC**
- `--dry-run` (stats only) and `--check` (composition audit) exit without writing.
- `--on-invalid {ignore,warn,error}` guards against non-nucleotide input.
- `--iupac-mode resolve` optionally resolves IUPAC codes to their bases.
- Empty/duplicate sequence IDs are rejected (`--allow-dup-ids` to permit).
- `-v`/`-vv` diagnostics; distinct exit codes (`2` bad input, `1` I/O).

**Threading & VCF**
- `-t, --threads <N>` pins the Rayon thread pool (deterministic wall-clock; output
is unaffected by thread count).
- `--chrom <NAME>` sets the VCF `CHROM` / `##contig` name (default `1`).
- `-q, --quiet` silences the `[snpick]` progress logs (errors still print).
- A valid header-only VCF is written when `--vcf` is requested but there are no
variable sites, so pipelines declaring the `.vcf` output no longer break.

**Packaging**
- Published as a reusable Rust library crate, with Python (pyo3/maturin) and
WebAssembly bindings, and Docker/Apptainer container recipes.

### Fixed
- Silent SNP loss when a single-line FASTA record contained a blank line: such
records were read with misaligned byte offsets. They now use the
newline-skipping scanner.
- Gap reference bases produced an invalid VCF `REF` of `-` under `--include-gaps`;
the `REF` is now rendered as `N` (a valid VCF base). ALT gaps stay `*` (the snp-sites
convention), so `REF` and `ALT` gap encodings differ by design.
- All-gap columns were tallied in no category under `--include-gaps`, breaking the
reported `variable + constant + ambiguous == length` invariant.
- Bare output filenames (`-o snps.fasta`, no directory) were rejected during path
resolution, breaking the documented quickstart.
- The release workflow's "tag already exists" guard was never read, so merges
without a version bump re-published over existing tags.
- The README's pre-built binary download command pointed at an asset name the
release never publishes (404).

### Performance
- VCF data rows are assembled in a reused byte buffer instead of one formatted
write per genotype field — a large speedup on many-sample inputs. Output is
byte-identical.

## [1.0.1] - 2026-03-31

- First Bioconda release.
- Cross-platform Build & Release CI workflow (Linux/macOS, x86_64/aarch64).
- README overhaul and benchmarks.

## [1.0.0] - 2024-11-16

- Initial release: zero-copy memory-mapped extraction of variable sites from
FASTA alignments, optional VCF v4.2 output, and ASC `fconst` reporting.

[1.0.2]: https://github.com/PathoGenOmics-Lab/snpick/compare/1.0.1...1.0.2
[1.0.1]: https://github.com/PathoGenOmics-Lab/snpick/compare/1.0.0...1.0.1
[1.0.0]: https://github.com/PathoGenOmics-Lab/snpick/releases/tag/1.0.0
Loading
Loading