fix(biolink): make KGX output validate against the Biolink Model

CHANGELOG.md

@@ -5,6 +5,7 @@ All notable changes to this project are documented in this file.
 ## Unreleased
 
 ### Breaking Changes
+- **`dspy` moved out of the `[agent]` extra into a new `[optimize]` extra.** `dspy` is used ONLY by the GEPA prompt-optimization path (`agent --optimize`); ordinary agent runs never import it. Installs that use `--optimize` must now install `pip install "tablassert[agent,optimize]"` (or add `tablassert[optimize]`); the missing-package error now points at `tablassert[optimize]` accordingly. Installs that never run `--optimize` get a lighter `[agent]` install (no `dspy`).
 - **Fullmap databases built by older releases must be rebuilt.** The Rust extension upgraded its embedded database engine from redb 2.6 to redb 4.1, and redb ≥ 3 dropped the old v2 file format. Existing `fullmap.redb` (and sibling `fullmap.s*.redb`) files fail to open with `fullmap DB is outdated or needs repair; rebuild with 'tablassert build-fullmap'`. Run `tablassert build-fullmap` once after upgrading. BABEL downloads stay cached, but the command rebuilds the fullmap files. The on-disk fullmap schema is now `tablassert.fullmap.v5` (the table layout is unchanged; the bump makes the redb-4 rebuild explicit and lets an older extension reject new files loudly).
 
 ### Changed

README.md

@@ -100,7 +100,8 @@ CSV/TSV/Excel sources; optional extras add runtime and pipeline capabilities:
 | ----- | ---- | ------- |
 | `rt` | CPU-compatible Polars runtime | `pip install "tablassert[rt]"` |
 | `qc` | three-stage QC audit (exact → fuzzy → BioBERT embeddings) | `pip install "tablassert[qc]"` |
-| `agent` | autonomous agent (smolagents, litellm, dspy, PDF context) | `pip install "tablassert[agent]"` |
+| `agent` | autonomous agent (smolagents, litellm, PDF context) | `pip install "tablassert[agent]"` |
+| `optimize` | GEPA prompt optimization for `agent --optimize` (dspy) | `pip install "tablassert[optimize]"` |
 
 QC is opt-in at build time (`build-kg --qc`). See the
 [Installation guide](https://skyeav.github.io/Tablassert/installation/) for the full matrix and the

docs/agent.md

@@ -12,23 +12,32 @@ ReAct loop) and [DSPy](https://dspy.ai) GEPA for prompt optimization.
 !!! warning "Optional extra"
     The base `tablassert` package does **not** require any of this. `smolagents` and `dspy` are imported
     **lazily** in `tablassert.agent`, so the base install and its test suite are unaffected. Install the
-    extra with `pip install tablassert[agent]`.
+    extra with `pip install tablassert[agent]`. GEPA prompt optimization (`--optimize`) additionally needs
+    the `[optimize]` extra (`dspy`): `pip install tablassert[optimize]`.
 
 ## Installation
 
 ```bash
 pip install tablassert[agent]
+
+# GEPA prompt optimization (--optimize) additionally needs dspy:
+pip install "tablassert[agent,optimize]"
 ```
 
 The extra pins:
 
 | Package | Version | Role |
 | --- | --- | --- |
 | `smolagents` | `==1.26.0` | `CodeAgent` ReAct loop, `OpenAIModel`/`LiteLLMModel`, tools |
-| `dspy` | `==3.2.1` | `dspy.GEPA` black-box prompt optimization |
 | `litellm` | (any) | optional fallback / rate-limiting model backend |
 | `pdfminer.six` | (any) | extract a `.pdf` main text into data-fenced context (`pmc_article_context`) |
 
+The `[optimize]` extra (only needed for `agent --optimize`) pins:
+
+| Package | Version | Role |
+| --- | --- | --- |
+| `dspy` | `==3.2.1` | `dspy.GEPA` black-box prompt optimization |
+
 ## PMC-AWS data source
 
 Tables are fetched from the **new** PMC open-access S3 bucket — the sanctioned bulk path.

docs/cli.md

@@ -12,6 +12,7 @@ exposes **four subcommands** — `agent`, `build-fullmap`, `build-kg`, `validate
 | [`build-fullmap`](#build-fullmap) | Build the embedded fullmap redb used for entity resolution |
 | [`build-kg`](#build-kg) | Build a KGX NDJSON knowledge graph from a YAML configuration |
 | [`validate`](#validate) | Validate a graph or table configuration without executing it |
+| [`validate-kgx`](#validate-kgx) | Validate built KGX NDJSON against the Biolink Model |
 
 ## App flags
 
@@ -32,7 +33,8 @@ These are flags on the root `tablassert` command, **not** subcommands.
 
 Use this to autonomously turn one or more PMC articles into audited, improved KG configs and graphs
 (fetch → derive config → build + audit → improve until coverage maps). Requires the `[agent]` extra
-(`pip install tablassert[agent]`).
+(`pip install tablassert[agent]`); `--optimize` additionally needs the `[optimize]` extra
+(`pip install tablassert[optimize]`, pulls `dspy`).
 
 ```bash
 tablassert agent --fullmap PATH [OPTIONS] PMC-IDS...
@@ -163,11 +165,45 @@ tablassert validate graph.yaml --schema graph
 
 ---
 
+## validate-kgx
+
+Use this to check that a completed build is actually Biolink-compliant. Where
+[`validate`](#validate) checks your *configuration*, `validate-kgx` checks the *output*: every node
+and edge is constructed as the Biolink Pydantic class named by its own `category` — the same classes
+[`NCATSTranslator/translator-ingests`](https://github.com/NCATSTranslator/translator-ingests) builds
+when it ingests your files.
+
+```bash
+tablassert validate-kgx --nodes MY_KG_1.0.0.nodes.ndjson --edges MY_KG_1.0.0.edges.ndjson
+```
+
+| Option | Type | Required | Default | Description |
+| --- | --- | --- | --- | --- |
+| `--nodes`, `-n` | Path | Yes | — | Built `*.nodes.ndjson` file to validate |
+| `--edges`, `-e` | Path | Yes | — | Built `*.edges.ndjson` file to validate |
+| `--limit` | int | No | `20` | Maximum example failures to retain per file |
+
+Failures are grouped by field and error type, so a systematic modelling problem shows up as one line
+rather than a million:
+
+```text
+biolink-model 4.4.3
+nodes: 424141/424141 valid (0 failures)
+edges: 2000085/2000085 valid (0 failures)
+KGX output is Biolink-compliant.
+```
+
+Exits non-zero when any record fails, so it can gate a release in CI.
+
+---
+
 ## Typical workflow
 
 1. Author a table config, then a graph config that references it.
 2. `tablassert validate graph.yaml --schema graph` — fail fast on schema errors.
 3. `tablassert build-kg graph.yaml` — produce KGX NDJSON + RIG (add `--qc` to audit mappings).
+4. `tablassert validate-kgx -n MY_KG_1.0.0.nodes.ndjson -e MY_KG_1.0.0.edges.ndjson` — confirm the
+   output validates against the Biolink Model before shipping it downstream.
 
 ## Next Steps
 

docs/installation.md

@@ -62,7 +62,8 @@ pip install tablassert
 |---|---|---|
 | `rt` | Runtime-compatible Polars build | `polars[rtcompat]` |
 | `qc` | QC runtime (exact → fuzzy → BioBERT audit) | `scikit-learn`, `sentence-transformers` (`torch` + `numpy` arrive transitively; `rapidfuzz` is a core dependency) |
-| `agent` | Autonomous PMC → KG agent (`tablassert agent`) | `smolagents`, `dspy`, `litellm` |
+| `agent` | Autonomous PMC → KG agent (`tablassert agent`) | `smolagents`, `litellm` |
+| `optimize` | GEPA prompt optimization (`tablassert agent --optimize`) | `dspy` |
 
 ```bash
 # Install with runtime-compatible Polars

pyproject.toml

@@ -84,10 +84,12 @@ qc = [
 ]
 agent = [
     "smolagents>=1.26.0",
-    "dspy>=3.2.1",
     "litellm>=1.93.0",
     "pdfminer.six>=20221105",
 ]
+optimize = [
+    "dspy>=3.2.1",
+]
 
 [dependency-groups]
 dev = [

rust/src/json.rs

@@ -1,22 +1,17 @@
-use serde_json::{Map, Number, Value};
+use serde_json::{Map, Value};
 
 fn is_bad_token(text: &str) -> bool {
     let lowered: String = text.trim().to_ascii_lowercase();
     matches!(lowered.as_str(), "" | "na" | "nan" | "null" | "none")
 }
 
-fn is_zero_number(number: &Number) -> bool {
-    number.as_i64().is_some_and(|x| x == 0)
-        || number.as_u64().is_some_and(|x| x == 0)
-        || number.as_f64().is_some_and(|x| x == 0.0)
-}
-
-// ? Mirrors Python truthiness (`if v`) for JSON values
-fn is_truthy(value: &Value) -> bool {
+// ? Drops absent values only. Deliberately NOT Python truthiness: `0` and `false` are
+// ? meaningful Biolink values (a p_value of 0, `number_of_cases: 0`, `negated: false`),
+// ? and treating them as absent silently deletes the key from the emitted record.
+fn is_present(value: &Value) -> bool {
     match value {
         Value::Null => false,
-        Value::Bool(flag) => *flag,
-        Value::Number(number) => !is_zero_number(number),
+        Value::Bool(_) | Value::Number(_) => true,
         Value::String(text) => !text.is_empty(),
         Value::Array(items) => !items.is_empty(),
         Value::Object(entries) => !entries.is_empty(),
@@ -32,7 +27,7 @@ fn passes_bad_check(value: &Value) -> bool {
 }
 
 fn keep(value: &Value) -> bool {
-    is_truthy(value) && passes_bad_check(value)
+    is_present(value) && passes_bad_check(value)
 }
 
 // ? Python value transform: lists recurse only into dict items and keep scalars verbatim;
@@ -80,14 +75,13 @@ mod tests {
     use serde_json::json;
 
     #[test]
-    fn strip_nulls_removes_falsey_and_null_like_values() {
+    fn strip_nulls_removes_absent_and_null_like_values() {
         let value = json!({
             "keep": "BRCA1",
             "empty": "",
             "blank": "  ",
             "na": "NA",
-            "zero": 0,
-            "false": false,
+            "null": null,
             "empty_array": [],
             "empty_object": {},
             "nested": {"drop": "null", "keep": true}
@@ -97,6 +91,20 @@ mod tests {
         assert_eq!(result, json!({"keep": "BRCA1", "nested": {"keep": true}}));
     }
 
+    #[test]
+    fn strip_nulls_keeps_zero_and_false() {
+        // ! `0` and `false` are meaningful Biolink values (a p_value of 0,
+        // ! `number_of_cases: 0`, `negated: false`). Treating them as absent - as
+        // ! Python truthiness would - silently deletes the key from the record.
+        let value = json!({"p_value": 0, "number_of_cases": 0, "negated": false, "rate": 0.0});
+
+        let result = strip_nulls(&value);
+        assert_eq!(
+            result,
+            json!({"p_value": 0, "number_of_cases": 0, "negated": false, "rate": 0.0})
+        );
+    }
+
     #[test]
     fn strip_nulls_keeps_emptied_nested_dict_and_list_scalars() {
         // ! Faithful Python semantics: a nested dict that empties stays as {}, and list

src/tablassert/agent.py

@@ -2,9 +2,14 @@
 
 This module hosts a smolagents ``CodeAgent`` pipeline that autonomously builds
 and audits KGX knowledge graphs from PubMed Central articles. It is part of the
-OPTIONAL ``[agent]`` extra, so ``smolagents`` and ``dspy`` are imported LAZILY
-(via :class:`tablassert._lazy.LazyModule`) and the base package never requires
-them at import time. Install the extra with ``pip install tablassert[agent]``.
+OPTIONAL ``[agent]`` extra, so ``smolagents`` is imported LAZILY (via
+:class:`tablassert._lazy.LazyModule`) and the base package never requires it at
+import time. Install the extra with ``pip install tablassert[agent]``.
+
+``dspy`` powers ONLY the GEPA prompt-optimization path (``agent --optimize``)
+and lives in its own OPTIONAL ``[optimize]`` extra
+(``pip install tablassert[optimize]``); it is likewise lazy-imported and never
+required by ordinary agent runs.
 """
 
 from __future__ import annotations
@@ -49,6 +54,10 @@
     smolagents = LazyModule("smolagents")
 
 AGENT_EXTRA: str = "pip install tablassert[agent]"
+OPTIMIZE_EXTRA: str = "pip install tablassert[optimize]"
+
+# Package -> install hint for the extra that actually ships it (default: [agent]).
+_EXTRA_HINT: dict[str, str] = {"dspy": OPTIMIZE_EXTRA}
 
 logger = cat("AGENT")
 
@@ -58,7 +67,7 @@ def _require(name: str) -> None:
     try:
         import_module(name)
     except ImportError as exc:
-        raise ImportError(f"tablassert agent features require the '{name}' package. Install with {AGENT_EXTRA}.") from exc
+        raise ImportError(f"tablassert agent features require the '{name}' package. Install with {_EXTRA_HINT.get(name, AGENT_EXTRA)}.") from exc
 
 
 def is_lazy() -> bool: