feat: incremental brain/add without rebuild (#14)

AGENTS.md

@@ -40,7 +40,7 @@ PLAN.md       decisions + execution + open questions
 docs/         published docs
 skills/       in-project agent skills (vendored, no external links)
 bin/          self-describing tools bin/{subject}/{method}.go (shebang)
-bin/brain/    search.go serve.go index.go get.go stats.go eval.go watch.go
+bin/brain/    search.go serve.go index.go add.go get.go stats.go eval.go watch.go
 bin/chats/    sync.go import.go facts.go apply.go; libs in internal/chats
 bin/mail/     sync.go import.go (index_mail → brain/index.go)
 bin/markdown/ import.go (H2 leaf split; Python bin/md/import fallback)
@@ -74,9 +74,9 @@ bin/brain/index.go --rebuild                                            # rebuil
   `pdftotext -layout` fast path (~15ms); textless/scanned PDFs fall back to
   docling (isolated subprocess — its native onnx can segfault the parent).
   Conversion never touches the brain DB (crash safety).
-- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Ladybug
-  corrupts its WAL when brand-new leafs are bulk-inserted while FTS/vector
-  indexes exist; a fresh DB with indexes created last is the only safe path.
+- `index_mail` is a deprecation shim for `bin/brain/index.go --rebuild`. Bulk
+  rebuild still deletes `var/kb.lbug` and creates FTS/HNSW last. Single-leaf
+  write is `bin/brain/add.go` (safe while indexes exist; do not DROP INDEX).
   Keep conversion + indexing separate so a conversion crash can't leave the
   DB mid-transaction.
 
@@ -89,6 +89,8 @@ bin/kb/search "query" [--repo X]                  # deprecated wrapper → bin/b
 bin/brain/search.go "query" [--root facts|info]   # deduction search → YAML
 bin/brain/search.go "query" --no-web              # local graph only
 eval "$(bin/cgo/zig env)"                         # Zig cc + liblbug (not gcc)
+bin/brain/add.go --text T --root facts --source "a.md x b.md"  # incremental write
+bin/brain/add.go --json                                      # stdin leaf or {leafs:[...]}
 bin/brain/get.go <id> [--body] [--json]          # Go read; Python bin/kb/get CI fallback
 bin/brain/stats.go [--json]
 bin/brain/eval.go [--json]                       # recall@5; questions in internal/brain/rank

PLAN.md

@@ -31,9 +31,9 @@ detective method: **a fact needs ≥2 independent sources or it is
 | D3 | web search | Go client `bin/web/search.go` (`internal/websearch`). SearXNG URL is config (`BRAIN_SEARCH_URL`). Optional Compose profile `searxng` (sanitized settings). Do not run a second copy on a host that already has one. Empty/`throttled` ≠ “nothing exists”. |
 | D4 | embeddings | **model2vec** `minishlab/potion-multilingual-128M` instead of embeddinggemma. |
 | D5 | parser | **mistune** for MD → leaf extraction (duckdb-md documented as future optional SQL/export layer, not v1). |
-| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Index/write stays Python (`compose --profile index`) until the Go write path is safe. |
+| D6 | graph engine | **LadybugDB**. Go is the service (`bin/brain/search.go`, `bin/brain/serve.go` in-process, `internal/brain`). Read path is Go + Zig CGO (D21). Python `bin/kb/{get,stats,eval}` is the CI fallback when Zig/libs are not fetched. Incremental write is Python `bin/kb/add` (`bin/brain/add.go`). Bulk rebuild stays `compose --profile index` until the Go write path is safe. |
 | D7 | db access | `db-yaml`/`psql-yq`-style, read-only, YAML out. OnlyOffice Postgres via SSH tunnel (`127.0.0.1:5433`). |
-| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. Auto-pair docker ps × compose × ssh-config × docs. |
+| D8 | evidence | detective method: ≥2 independent sources or `(not confirmed)`. 2-source auto-pair docker ps × compose × ssh-config × docs. |
 | D9 | facts/goal model | Who / What / How / Where / When + evidence + confidence on every edge. |
 | D10 | versioning | everything is a leaf with `sha256 + observed_at + source_rev`; `File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`. Stale = `source_rev` < git HEAD. |
 | D11 | strong/weak | `root` column: `facts` (strong) vs `info` (weak). Answer is `confirmed` only from facts root. |
@@ -45,7 +45,7 @@ detective method: **a fact needs ≥2 independent sources or it is
 | D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. |
 | D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). |
 | D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. |
-| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). |
+| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`/`ingest`). |
 | D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc` → `zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. |
 
 ## Architecture
@@ -57,8 +57,10 @@ detective method: **a fact needs ≥2 independent sources or it is
   skills/                   in-project skills (web-search, postgres, brain, picoclaw, diataxis-docs)
   bin/
     facts/extract.go audit.go crm.go  # D14 shebang; Python implementation
-    kb/index                Python write path (called by bin/brain/index.go)
+    kb/index                Python bulk write (called by bin/brain/index.go)
+    kb/add                  Python incremental write (called by bin/brain/add.go)
     brain/index.go          rebuild FTS + HNSW (incl. --with-mail)
+    brain/add.go            incremental leaf write (no rebuild)
     brain/get.go stats.go eval.go  # Go read (cgo); Python bin/kb/* CI fallback
     brain/watch.go
     brain/search.go         deduction: facts → info → web-search
@@ -160,18 +162,19 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
 
 ## Gap to v1 (epic #16)
 
-Read path + MCP are in. The detective brain is not closed until the graph is
-**writable incrementally** and search can **walk** it. Board:
+Read path + MCP are in. Incremental `brain/add` is in. The detective brain is
+not closed until search can **walk** the graph and the facts+chats corpus
+lands on rebuild. Board:
 [epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
 milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
 Narrative: [docs/roadmap.md](docs/roadmap.md).
 
 | Order | Issue | Gap |
 |-------|-------|-----|
-| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | Write stays Python rebuild; `brain/add` / `POST /ingest` are hints. Ladybug 0.19 WAL corrupts if new leafs land while FTS/HNSW exist. |
+| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | **in** — `bin/brain/add.go` / `POST /ingest` write facts+info without deleting `kb.lbug`. Bulk corpus still `--rebuild`. Leftover Python (mail/facts) is not the living-graph blocker. |
 | 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | `--hop` errors. `FROM_FILE` / `HAS_VERSION` are in schema; search does not walk them. |
 | 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. |
-| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | Lever = 2dph fact-check. Loop = PicoClaw/MCP `search` → `get` → `audit`. Specify in-repo, not only live config. |
+| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | **in** — lever/loop documented (`search` → `get` → `audit`). |
 | 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. |
 
 Does **not** block epic close: [#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR, OQ1, OQ3, OQ4.

README.md

@@ -120,6 +120,7 @@ Mail is a first-class corpus (retrievable through the same search):
 ```bash
 bin/mail/sync.go --source onlyoffice,gmail --workers 8 --out var/mail  # raw sync (Go)
 bin/mail/import.go --from-raw var/mail                                  # JSON → markdown
+bin/brain/add.go --text T --root facts --source "a.md x b.md"
 bin/brain/index.go --rebuild                                            # rebuild brain (incl. mail)
 bin/brain/search.go "invoice from last week"                            # same search over mail leafs
 ```
@@ -128,8 +129,9 @@ bin/brain/search.go "invoice from last week"                            # same s
 
 - **LadybugDB** — single `var/kb.lbug`, Cypher + HNSW + BM25, embedded.
   Read tools (`get` / `stats` / `eval`) are Go + Zig CGO (`bin/cgo/zcc`).
-  Python fallbacks stay for CI until the runner fetches Zig. Write is
-  Compose profile `index` (`bin/brain/index.go`).
+  Python fallbacks stay for CI until the runner fetches Zig. Incremental
+  write is `bin/brain/add.go` (Python `kblib.add_leafs`). Bulk rebuild is
+  Compose profile `index` (`bin/brain/index.go --rebuild`).
 - **model2vec** — `potion-multilingual-128M` (256-dim), CPU, no Ollama
   runtime dependency.
 - facts and info split by `root` but written in the same transaction.

bin/brain/add.go

@@ -0,0 +1,21 @@
+//usr/bin/env go run -tags=brain_add "$0" "$@"; exit
+//go:build brain_add
+//
+// bin/brain/add.go - incremental leaf write (Python kblib, no rebuild).
+//
+//	./bin/brain/add.go --text T --root facts --source "a.md x b.md"
+//	./bin/brain/add.go --json
+//
+// D6: write stays Python. Does not delete var/kb.lbug.
+// NOTE: never run `gofmt -w` on this file — it breaks the shebang.
+package main
+
+import (
+	"os"
+
+	"github.com/eSlider/2dph/internal/cmdbin"
+)
+
+func main() {
+	os.Exit(cmdbin.ExecFile("bin/kb/add", os.Args[1:]))
+}

bin/brain/index.go

@@ -7,8 +7,8 @@
 //	./bin/brain/index.go --rebuild --with-mail
 //	./bin/brain/index.go --dry-run --with-mail
 //
-// v1 write is always a rebuild when mail is included (live FTS/HNSW + bulk
-// insert corrupts Ladybug 0.19 WAL). `add` is v2.
+// v1 write: bin/brain/add.go for one/few leafs (indexes may already exist).
+// Bulk mail/corpus still --rebuild (fresh file, indexes last).
 // NOTE: never run `gofmt -w` on this file — it breaks the shebang.
 package main
 

bin/kb/add

@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+"""kb/add - incremental leaf write (no rebuild).
+
+  bin/kb/add --text T --root facts|info --source S
+  bin/kb/add --json                 # stdin: one object or {"leafs":[...]}
+  bin/kb/add --db PATH --json
+
+Writes facts+info in one Ladybug transaction. Does not delete kb.lbug.
+Embedding is used when provided; otherwise model2vec encodes the text.
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "bin" / "tools"))
+
+from kblib import (  # noqa: E402
+    EMBED_DIM,
+    add_leafs,
+    connect,
+    ensure_indexes,
+    init_schema,
+)
+
+
+def _as_leafs(payload: object) -> list[dict]:
+    if isinstance(payload, list):
+        return [dict(x) for x in payload]
+    if isinstance(payload, dict):
+        if "leafs" in payload:
+            return [dict(x) for x in payload["leafs"]]
+        return [dict(payload)]
+    raise ValueError("json must be an object, a list, or {leafs:[...]}")
+
+
+def _embed_missing(leafs: list[dict]) -> None:
+    missing = [lf for lf in leafs if not lf.get("embedding")]
+    if not missing:
+        return
+    from model2vec import StaticModel
+
+    model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
+    for lf in missing:
+        text = str(lf.get("text") or "")
+        vec = model.encode([text])[0].astype(float).tolist()
+        if len(vec) != EMBED_DIM:
+            vec = (vec + [0.0] * EMBED_DIM)[:EMBED_DIM]
+        lf["embedding"] = vec
+
+
+def main(argv: list[str]) -> int:
+    import argparse
+
+    p = argparse.ArgumentParser(description="add leafs without rebuilding the brain")
+    p.add_argument("--db", default="", help="path to kb.lbug (default var/kb.lbug)")
+    p.add_argument("--json", action="store_true", help="read leaf JSON from stdin")
+    p.add_argument("--text", default="", help="leaf text")
+    p.add_argument("--root", default="info", choices=("facts", "info"))
+    p.add_argument("--source", default="")
+    p.add_argument("--confidence", default="confirmed")
+    p.add_argument("--source-rev", default="working-tree")
+    p.add_argument("--how", default="brain/add")
+    p.add_argument("--loc", default="")
+    p.add_argument("--type", default="reference", dest="type_")
+    args = p.parse_args(argv)
+
+    if args.json:
+        raw = sys.stdin.read()
+        if not raw.strip():
+            print("kb/add: empty stdin", file=sys.stderr)
+            return 2
+        leafs = _as_leafs(json.loads(raw))
+    else:
+        if not args.text or not args.source:
+            print("kb/add: --text and --source are required (or --json)", file=sys.stderr)
+            return 2
+        leafs = [{
+            "text": args.text,
+            "root": args.root,
+            "source": args.source,
+            "confidence": args.confidence,
+            "source_rev": args.source_rev,
+            "how": args.how,
+            "loc": args.loc or args.source,
+            "type": args.type_,
+        }]
+
+    for lf in leafs:
+        if not lf.get("text") or not lf.get("source"):
+            print("kb/add: each leaf needs text and source", file=sys.stderr)
+            return 2
+
+    _embed_missing(leafs)
+
+    from kblib import DB_PATH, VAR
+
+    dbpath = Path(args.db) if args.db else DB_PATH
+    dbpath.parent.mkdir(parents=True, exist_ok=True)
+    VAR.mkdir(exist_ok=True)
+    db, conn = connect(dbpath, read_only=False)
+    init_schema(conn)
+    ids = add_leafs(conn, leafs)
+    ensure_indexes(conn)
+    conn.close()
+    db.close()
+    print(json.dumps({"mode": "add", "ids": ids, "db": str(dbpath)}))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

bin/tools/kblib.py

@@ -4,7 +4,7 @@
 >=2 independent sources) and info (narrative leafs). Hybrid retrieval: BM25
 (FTS extension) + HNSW cosine (VECTOR extension) + Cypher graph hops.
 
-All access is read-only unless `--rebuild` is passed to kb/index.
+All access is read-only unless `--rebuild` (kb/index) or `kb/add`.
 """
 from __future__ import annotations
 
@@ -118,6 +118,50 @@ def upsert_leaf(conn: ladybug.Connection, *, text: str, root: str, confidence: s
     return lid
 
 
+def add_leafs(conn: ladybug.Connection, leafs: list[dict]) -> list[str]:
+    """Write facts+info leafs in one transaction. Safe while FTS/HNSW exist.
+
+    Each leaf dict: text, source, optional root/confidence/source_rev/how/loc/type/embedding.
+    Does not delete the database file. Measured on Ladybug 0.19: MERGE of new
+    ids (and updates) stays FTS+HNSW queryable; DROP INDEX is the fatal path.
+    """
+    if not leafs:
+        return []
+    started = False
+    try:
+        conn.execute("BEGIN TRANSACTION")
+        started = True
+    except Exception:
+        started = False
+    ids: list[str] = []
+    try:
+        for lf in leafs:
+            ids.append(
+                upsert_leaf(
+                    conn,
+                    text=str(lf["text"]),
+                    root=str(lf.get("root") or ROOT_INFO),
+                    confidence=str(lf.get("confidence") or CONF_CONFIRMED),
+                    source=str(lf["source"]),
+                    source_rev=str(lf.get("source_rev") or "working-tree"),
+                    how=str(lf.get("how") or "brain/add"),
+                    loc=str(lf.get("loc") or lf.get("source") or ""),
+                    type_=str(lf.get("type") or lf.get("type_") or "reference"),
+                    embedding=lf.get("embedding"),
+                )
+            )
+        if started:
+            conn.execute("COMMIT")
+    except Exception:
+        if started:
+            try:
+                conn.execute("ROLLBACK")
+            except Exception:
+                pass
+        raise
+    return ids
+
+
 def leaf_index_names(conn: ladybug.Connection) -> set[str]:
     """Return index names on the Leaf table (e.g. {'id', 'Leaf_vec', '_PK'})."""
     rows = conn.execute("CALL SHOW_INDEXES() RETURN *").get_all()

bin/tools/test_bin_layout.py

@@ -75,9 +75,20 @@ def _assert_shebang(self, rel: str) -> None:
         )
 
     def test_brain_methods_are_shebangs(self) -> None:
-        for method in ("index.go", "get.go", "stats.go", "eval.go", "watch.go"):
+        for method in ("index.go", "add.go", "get.go", "stats.go", "eval.go", "watch.go"):
             self._assert_shebang(f"bin/brain/{method}")
 
+    def test_brain_add_is_python_write_not_rebuild(self) -> None:
+        self._assert_shebang("bin/brain/add.go")
+        text = (ROOT / "bin" / "brain" / "add.go").read_text()
+        self.assertIn("cmdbin.ExecFile", text)
+        self.assertIn("bin/kb/add", text)
+        self.assertNotIn("--rebuild", text)
+        py = (ROOT / "bin" / "kb" / "add").read_text()
+        self.assertIn("add_leafs", py)
+        self.assertIn("--json", py)
+        self.assertNotIn("unlink", py.lower())
+
     def test_brain_get_stats_eval_are_not_python_exec(self) -> None:
         for method in ("get.go", "stats.go", "eval.go"):
             text = (ROOT / "bin" / "brain" / method).read_text()