feat: add aria2c build-fullmap downloads

CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Added
+- **`tablassert build-fullmap --aria2c` / `-a`** opt-in downloader acceleration. When requested, the BABEL download stage uses the installed `aria2c` executable with segmented HTTP downloads plus resume/retry flags (`--continue=true`, `--max-tries`, `--retry-wait`) while keeping the existing Python downloader as the default. Missing or failing `aria2c` fails loud instead of silently falling back, and aria2 `.aria2` control files are preserved so interrupted downloads can resume on rerun.
+
 ## 8.2.0 - 2026-08-10
 
 ### Breaking Changes

docs/cli.md

@@ -95,9 +95,12 @@ tablassert build-fullmap [ARGS]
 | `--cache`, `-c` | Path | No | `./fullmap/downloads` | Directory for downloaded BABEL files (`classes/`, `synonyms/`) |
 | `--version`, `-v` | str | No | `2026jul22` | BABEL snapshot date to fetch (a RENCI stamp, **not** Tablassert's version) |
 | `--threads`, `-t` | int | No | `None` (auto) | Worker threads; auto-capped by memory on Linux (`/proc/meminfo`), else ~90% of CPUs |
+| `--aria2c`, `-a` | Flag | No | `False` | Opt into the installed `aria2c` executable for resumable segmented BABEL downloads; fails loud if `aria2c` is missing or exits non-zero |
 
 ```bash
 tablassert build-fullmap --output /data/fullmap/fullmap.redb
+# Optional: use aria2c for faster/resumable BABEL downloads when installed
+tablassert build-fullmap --aria2c --output /data/fullmap/fullmap.redb
 ```
 
 See [Fullmap](fullmap.md) for the data pipeline, output schema, and graph-config usage.

docs/fullmap.md

@@ -10,18 +10,21 @@ files (`fullmap.s0.redb` … `fullmap.s15.redb` by default) holding the term→p
 biological synonyms, CURIEs, Biolink categories, taxon IDs, and source provenance, built from NCATS
 Translator BABEL export files.
 
-Fullmap is built entirely in-process by Tablassert's own Rust extension — no external tool or install step required (this is an in-process redb shard scheme, not the older external DuckDB shards).
+Fullmap is built entirely in-process by Tablassert's own Rust extension — no external tool or install step required by default (this is an in-process redb shard scheme, not the older external DuckDB shards). If you opt into `build-fullmap --aria2c` / `-a`, only the download stage uses an installed external `aria2c` executable.
 
 ## Build Command
 
 ```bash
 # Build a fullmap database (downloads BABEL data automatically)
 tablassert build-fullmap
+
+# Optional: use installed aria2c for resumable segmented BABEL downloads
+tablassert build-fullmap --aria2c
 ```
 
 See the [CLI Reference → build-fullmap](cli.md#build-fullmap) for the complete flag table (output path,
-cache directory, BABEL snapshot version, worker threads), their defaults, and more examples. Two facts
-matter most when planning a build:
+cache directory, BABEL snapshot version, worker threads, and the optional `--aria2c` / `-a` downloader),
+their defaults, and more examples. Two facts matter most when planning a build:
 
 - The BABEL **version** flag selects a RENCI BABEL snapshot date (default `2026jul22`) — *not*
   Tablassert's package version. Bumping it fetches a different snapshot and requires rebuilding; the
@@ -34,7 +37,7 @@ matter most when planning a build:
 
 The build is a parallel, **memory-bounded** pipeline executed by the Rust extension:
 
-1. **Download** — fetch BABEL class and synonym files from RENCI into the cache (resumable, reused).
+1. **Download** — fetch BABEL class and synonym files from RENCI into the cache (resumable, reused). By default this uses Tablassert's Python downloader; `--aria2c` / `-a` opts into the installed `aria2c` executable, preserving aria2 resume control files across dropped downloads and failing loud if the executable is missing or the download fails.
 2. **Equivalents index** — parse class files into sorted on-disk runs, then k-way merge them into a
    memory-mapped index mapping each primary CURIE to its equivalents.
 3. **Synonym pass** — a producer/consumer pool streams byte-bounded line-chunks; workers dedup CURIEs,
@@ -50,7 +53,10 @@ The build is a parallel, **memory-bounded** pipeline executed by the Rust extens
     multi-threaded allocation from bloating resident memory.
 
     - **Download** — files come from `https://stars.renci.org/var/babel_outputs` via resumable,
-      range-request downloads; cached files are reused.
+      range-request downloads; cached files are reused. Passing `--aria2c` / `-a` switches only this
+      stage to the installed `aria2c` executable, using aria2's segmented HTTP downloads and retry/resume
+      control files while suppressing aria2's own progress UI so Tablassert's progress bar stays clean.
+      The progress detail remains file-level (`aria2c downloading`) rather than byte-level in this mode.
     - **Equivalents index** — class files parse in parallel into sorted on-disk runs, k-way merged into a
       single memory-mapped CURIE→equivalents index; only a compact `(hash, offset)` index lives in RAM,
       the string data is mmap'd.

src/tablassert/cli.py

@@ -1,6 +1,8 @@
 from __future__ import annotations
 
 import re
+import shutil
+import subprocess
 import sys
 import time
 from collections.abc import Callable
@@ -420,6 +422,93 @@ def download_babel_file(filename: str, url: str, destination: Path, retries: int
     raise BabelDownloadError(url, retries, last_error or RuntimeError("no attempts made")) from last_error
 
 
+def download_babel_file_aria2c(filename: str, url: str, destination: Path, retries: int = 5) -> Path:
+    """Download one BABEL file with the optional external ``aria2c`` executable.
+
+    The helper mirrors ``download_babel_file``'s final-file cache contract but
+    delegates resume/retry behavior to aria2. Incomplete aria2 downloads leave a
+    ``<filename>.aria2`` control file next to the target; when that control file
+    exists we do NOT treat the target as a cache hit, and failures never remove
+    either file so a later run can continue.
+
+    Args:
+        filename: Output basename under ``destination``.
+        url: Source URL.
+        destination: Directory to download into (created if missing).
+        retries: Maximum aria2 tries (forwarded to ``--max-tries``).
+
+    Returns:
+        Path to the downloaded file.
+
+    Raises:
+        BabelDownloadError: If ``aria2c`` is missing, fails, or does not leave a
+            complete final file.
+    """
+    destination.mkdir(parents=True, exist_ok=True)
+    final_path: Path = destination / filename
+    control_path: Path = destination / f"{filename}.aria2"
+    if final_path.is_file() and not control_path.exists():
+        download_logger.info("Reusing cached BABEL file: {path}", path=final_path)
+        return final_path
+    if retries < 1:
+        error = ValueError("aria2c retries must be a positive integer")
+        raise BabelDownloadError(url, retries, error) from error
+
+    binary: str | None = shutil.which("aria2c")
+    if binary is None:
+        error = FileNotFoundError("aria2c executable not found; install aria2 or omit --aria2c")
+        raise BabelDownloadError(url, 0, error) from error
+
+    command: list[str] = [
+        binary,
+        "--continue=true",
+        "--max-tries",
+        str(retries),
+        "--retry-wait",
+        "5",
+        "--allow-overwrite=true",
+        "--auto-file-renaming=false",
+        "--max-connection-per-server=8",
+        "--split=8",
+        "--min-split-size=1M",
+        "--summary-interval=0",
+        "--console-log-level=warn",
+        "--show-console-readout=false",
+        "--dir",
+        str(destination),
+        "--out",
+        filename,
+        url,
+    ]
+    try:
+        completed: subprocess.CompletedProcess[str] = subprocess.run(
+            command, shell=False, check=False, capture_output=True, text=True, errors="replace"
+        )
+    except OSError as e:
+        raise BabelDownloadError(url, retries, e) from e
+
+    if completed.returncode != 0:
+        output: str = (completed.stderr or completed.stdout or "").strip()
+        detail: str = f"aria2c exited with status {completed.returncode}"
+        if output:
+            detail = f"{detail}: {output[-2000:]}"
+        error = RuntimeError(detail)
+        raise BabelDownloadError(url, retries, error) from error
+
+    if not final_path.is_file() or control_path.exists():
+        suffix: str = ""
+        if control_path.exists():
+            suffix = f"; resume control file still present: {control_path}"
+        output = (completed.stderr or completed.stdout or "").strip()
+        if output:
+            suffix = f"{suffix}; aria2c output: {output[-2000:]}"
+        error = FileNotFoundError(f"aria2c completed but did not create a complete file at {final_path}{suffix}")
+        raise BabelDownloadError(url, retries, error) from error
+
+    download_logger.info("Downloaded {url} -> {path} with aria2c", url=url, path=final_path)
+    return final_path
+
+
 def stream_copy(source: BinaryIO, destination: BinaryIO, on_bytes: Callable[[int], None] | None = None) -> None:
     """Copy ``source`` to ``destination`` in 1 MiB chunks.
 
@@ -780,7 +869,12 @@ def rebuild_agent_graph(
 
 
 def build_fullmap_pipeline(
-    output: Path, progress: PipelineProgress, cache: Path = Path("./fullmap/downloads"), version: str = BABEL_VERSION, threads: int | None = None
+    output: Path,
+    progress: PipelineProgress,
+    cache: Path = Path("./fullmap/downloads"),
+    version: str = BABEL_VERSION,
+    threads: int | None = None,
+    aria2c: bool = False,
 ) -> None:
     """Build an embedded fullmap redb database from BABEL outputs.
 
@@ -793,6 +887,7 @@ def build_fullmap_pipeline(
         cache: Directory for downloaded BABEL files.
         version: BABEL version label.
         threads: Optional thread count forwarded to Rust.
+        aria2c: Use the optional aria2c executable for downloads when true.
     """
     from tablassert import rs
 
@@ -816,18 +911,19 @@ def build_fullmap_pipeline(
     def report_progress(downloaded: int, total: int) -> None:
         sub_step(_download_detail(downloaded, total))
 
-    class_files: list[Path] = []
-    for filename, url in class_urls:
+    def download_one(filename: str, url: str, destination: Path) -> Path:
         start(filename)
-        sub_step("downloading")
-        class_files.append(download_babel_file(filename, url, cache / "classes", on_progress=report_progress))
-        advance()
-    synonym_files: list[Path] = []
-    for filename, url in synonym_urls:
-        start(filename)
-        sub_step("downloading")
-        synonym_files.append(download_babel_file(filename, url, cache / "synonyms", on_progress=report_progress))
+        if aria2c:
+            sub_step("aria2c downloading")
+            path: Path = download_babel_file_aria2c(filename, url, destination)
+        else:
+            sub_step("downloading")
+            path = download_babel_file(filename, url, destination, on_progress=report_progress)
         advance()
+        return path
+
+    class_files: list[Path] = [download_one(filename, url, cache / "classes") for filename, url in class_urls]
+    synonym_files: list[Path] = [download_one(filename, url, cache / "synonyms") for filename, url in synonym_urls]
 
     # Stage 3/3: build fullmap database.
     progress.stage("Building Fullmap Database")
@@ -852,6 +948,7 @@ def build_fullmap(
     cache: Annotated[Path, cyclopts.Parameter(name=["--cache", "-c"])] = Path("./fullmap/downloads"),
     version: Annotated[str, cyclopts.Parameter(name=["--version", "-v"])] = BABEL_VERSION,
     threads: Annotated[int | None, cyclopts.Parameter(name=["--threads", "-t"])] = None,
+    aria2c: Annotated[bool, cyclopts.Parameter(name=["--aria2c", "-a"], negative="")] = False,
 ) -> None:
     """Build an embedded fullmap redb database from hardcoded BABEL outputs."""
-    run(3, build_fullmap_pipeline, output, cache=cache, version=version, threads=threads)
+    run(3, build_fullmap_pipeline, output, cache=cache, version=version, threads=threads, aria2c=aria2c)