Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Evidence-first brain over the ops/eSlider stack. Facts need proof or they are
`(not confirmed)`.

Read first: [PLAN](PLAN.md) → [docs](docs/).
Read first: [PLAN](PLAN.md) → [docs](docs/) → [roadmap](docs/roadmap.md)
(epic [#16](https://git.produktor.io/eSlider/2dph/issues/16)).

## Method (detective, no fork)

Expand Down
38 changes: 30 additions & 8 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ A brain that loves facts and deduction. Evidence-first knowledge graph + hybrid
RAG over the operational Brain/ops/eSlider stack. Built like Sherlock
Holmes: nothing is asserted unless it has proof.

Status: **in progress** — this file is the plan and the record of decisions.
Status: **in progress** — read path + MCP work; v1 goal is [epic #16](https://git.produktor.io/eSlider/2dph/issues/16)
(milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12)).
Gap: [docs/roadmap.md](docs/roadmap.md).

## What

Expand Down Expand Up @@ -41,7 +43,7 @@ detective method: **a fact needs ≥2 independent sources or it is
| D15 | repo | Gitea [`eSlider/2dph`](https://git.produktor.io/eSlider/2dph) is origin + [issues](https://git.produktor.io/eSlider/2dph/issues). GitHub `eSlider/2dph` is the public clone (PRs + Actions CI). No direct `main` pushes. TDD → PR → CI green → merge. |
| D16 | contradictions | ≥2 yes vs ≥2 no → unrelated sources conflict → hypothesis → `(not confirmed)`. Resolution (authority, staleness adjudication) = **v2**, tracked as open question. |
| D17 | assertion gate | Fact-check every *claim* (facts → info → live → web), not every edit. `bin/brain/search.go` adds a `web` block when there is no facts hit (`throttled`/`skipped`/`refused` ≠ absence). `--root` and `--no-web` stay local. Missing graph ≠ “does not exist”. |
| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is not shipped; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. |
| D18 | reasoner | Pluggable OpenAI-compatible URL (`REASONER_BASE_URL`). RAM: `Qwen/Qwen3.5-9B`. Quality: `prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B. CPU bake-off: `bin/reasoner/bakeoff.go` + compose profile `reasoner` (`OLLAMA_NUM_GPU=0`, `:11435`). PicoClaw is compose profile `picoclaw`; tools are `search`/`get`/`audit`. Weights are not copied into the 2dph image. Agent lever/loop: [#15](https://git.produktor.io/eSlider/2dph/issues/15). |
| D19 | git history | [go-git](https://github.com/go-git/go-git) via `bin/git/import.go`. No subprocess of the git binary. Conversion prints commit leafs; brain write is `bin/brain/index.go`. |
| D20 | agent API | OpenAPI + MCP are generated from the same `internal/httpapi.Ops` table as `bin/brain/serve.go` handlers. `GET /openapi.json`, `POST /mcp` (JSON-RPC tools/list + tools/call). Tool names match OpenAPI paths (`search`/`get`/`stats`/`audit`). |
| D21 | CGO | Ladybug/tokenizers CGO is compiled with **Zig** (`bin/cgo/zcc` → `zig cc -target …-linux-gnu`), not gcc. `bin/cgo/zig` pins Zig 0.14.1 + liblbug 0.19.1 + libtokenizers 1.27.0. Compose `target: api` has no CPython; write/rebuild is profile `index`. |
Expand Down Expand Up @@ -83,7 +85,8 @@ detective method: **a fact needs ≥2 independent sources or it is

Node tables: `Person, Service, Host, Container, Repo, File, Commit, Leaf`.
`Leaf(embedding FLOAT[N])` — FTS on `text`, HNSW vector index on `embedding`.
Edges: `RUNS / USES / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
Edges: `RUNS / USES / FROM_FILE / HAS_VERSION / AUTHORED / ABOUT / ASSOCIATED / SIMILAR_0.85`.
`FROM_FILE` / `HAS_VERSION` exist in schema; search `--hop` does not walk them yet ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).

Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
`where`, `when`, `source_rev`.
Expand All @@ -108,9 +111,9 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
## Open questions (v2)

- OQ1: mutually-contradicting evidence — how to resolve (authority weighting,
temporal freshness, audit adjudication).
- OQ2: OCR pipeline for pdfs/images/docs — mostly solved: poppler pdftotext
fast-path for born-digital PDFs, docling fallback for the ~5% textless ones.
temporal freshness, audit adjudication). **v2**; does not block epic #16.
- OQ2: OCR — poppler `pdftotext` fast-path exists; scans still docling.
[#6](https://git.produktor.io/eSlider/2dph/issues/6) (v2, does not block #16).
- OQ3: optional duckdb-md layer for `SELECT … FORMAT MARKDOWN` export/write-back.
- OQ4: YAML-first storage for leafs — deferred: JSON is ~10x faster to
serialize and unambiguous; YAML only where humans edit files.
Expand Down Expand Up @@ -138,6 +141,7 @@ Common props on every node/edge: `root`, `confidence`, `evidence[]`, `how`,
3. python -m unittest discover -s bin/tools (includes published-docs SoT)
4. `bin/facts/audit self` (lexicon internal consistency; `bin/facts/audit.go` is the D14 wrapper)
5. `bin/kb/eval` (recall@5 ≥ 0.95). Local SoT is `bin/brain/eval.go` via Zig CGO.
CI SoT switch: [#19](https://git.produktor.io/eSlider/2dph/issues/19).
6. `bin/cgo/zig go build -tags system_ladybug` (compile search with zig cc; fetches pinned zig+libs).

Feedback loop: every commit → PR → CI → green/gate → merge. Same discipline as
Expand All @@ -151,5 +155,23 @@ Feedback loop: every commit → PR → CI → green/gate → merge. Same discipl
4. .venv: ladybug + model2vec + mistune
5. schema + tools with TDD (kb + md + facts + brain)
6. ~/.config/brain config
7. corpus extraction (facts/info)
8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit
7. corpus extraction (facts/info) — **open**: [#18](https://git.produktor.io/eSlider/2dph/issues/18)
8. verify: web-search smoke, onlyoffice pg, md-db round-trip, eval, audit

## Gap to v1 (epic #16)

Read path + MCP are in. The detective brain is not closed until the graph is
**writable incrementally** and search can **walk** it. Board:
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16),
milestone [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Narrative: [docs/roadmap.md](docs/roadmap.md).

| Order | Issue | Gap |
|-------|-------|-----|
| 1 | [#14](https://git.produktor.io/eSlider/2dph/issues/14) | Write stays Python rebuild; `brain/add` / `POST /ingest` are hints. Ladybug 0.19 WAL corrupts if new leafs land while FTS/HNSW exist. |
| 2 | [#17](https://git.produktor.io/eSlider/2dph/issues/17) | `--hop` errors. `FROM_FILE` / `HAS_VERSION` are in schema; search does not walk them. |
| 3 | [#18](https://git.produktor.io/eSlider/2dph/issues/18) | Rebuild is mostly `info` (repo md + mail). `facts/extract` and chats are not a first-class index input. WhatsApp sync is a stub. |
| 4 | [#15](https://git.produktor.io/eSlider/2dph/issues/15) | Lever = 2dph fact-check. Loop = PicoClaw/MCP `search` → `get` → `audit`. Specify in-repo, not only live config. |
| 5 | [#19](https://git.produktor.io/eSlider/2dph/issues/19) | GitHub CI recall still runs Python `bin/kb/eval`. |

Does **not** block epic close: [#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR, OQ1, OQ3, OQ4.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,9 @@ docker compose up brain-watch # auto re-index on change
skills (`web-search`, `postgres`, …) that 2dph integrates
- detective method — the two-source method

Work board (issues): [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
Work board (issues): [epic #16](https://git.produktor.io/eSlider/2dph/issues/16)
on [git.produktor.io/eSlider/2dph/issues](https://git.produktor.io/eSlider/2dph/issues).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).

See [PLAN.md](PLAN.md) for decisions, execution status, and v2 open questions.
See [PLAN.md](PLAN.md) for decisions, [docs/roadmap.md](docs/roadmap.md) for
the gap to v1, and v2 open questions.
17 changes: 17 additions & 0 deletions bin/tools/test_published_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ def test_docs_do_not_claim_hop_walks(self) -> None:
ROOT / "skills" / "diataxis-docs" / "SKILL.md",
ROOT / "docs" / "runbook.md",
ROOT / "docs" / "README.md",
ROOT / "docs" / "roadmap.md",
]
# Command-style `--hop 1` / `--hop N` plus follow/walk = the old lie.
# Honest "not implemented" notes must not match.
Expand All @@ -173,3 +174,19 @@ def test_docs_are_portable_diataxis(self) -> None:
readme = (ROOT / "README.md").read_text()
self.assertIn("docs/runbook.md", readme)
self.assertNotIn("search.ops.io", readme)

def test_v1_epic_is_named_in_docs(self) -> None:
plan = (ROOT / "PLAN.md").read_text()
self.assertIn("Gap to v1", plan)
self.assertIn("eSlider/2dph/issues/16", plan)
self.assertIn("eSlider/2dph/issues/17", plan)
self.assertIn("eSlider/2dph/milestone/12", plan)
road = (ROOT / "docs" / "roadmap.md").read_text()
self.assertIn("type: explanation", road)
self.assertIn("issues/16", road)
self.assertIn("issues/14", road)
index = (ROOT / "docs" / "README.md").read_text()
self.assertIn("roadmap.md", index)
self.assertIn("epic #16", index)
agents = (ROOT / "AGENTS.md").read_text()
self.assertIn("roadmap.md", agents)
8 changes: 6 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ related:
- docs/runbook.md
- docs/design.md
- PLAN.md
- docs/roadmap.md
---

# 2dph docs (Diataxis)
Expand All @@ -16,6 +17,7 @@ Evidence-first knowledge graph. Facts need proof or they are
|------|-----|
| tutorial / howto | [runbook](runbook.md) — run anywhere (uv, Go, Docker) |
| explanation | [design](design.md) — two roots, deduction, D17/D20/D18 |
| explanation | [roadmap](roadmap.md) — gap to v1 (epic #16) |
| howto | [picoclaw](picoclaw.md) — MCP agent profile |
| howto | [reasoner](reasoner.md) — CPU bake-off (D18) |
| reference | [PLAN.md](../PLAN.md) — decisions D1–D21 |
Expand All @@ -26,9 +28,11 @@ Python write sidecar, **D14** `bin/{subject}/{method}.go`, **D15** Gitea origin,

Search: `bin/brain/search.go "query"` (HTTP: `bin/brain/serve.go` —
`/health` `/search` `/get` `/stats` `/audit` `/ingest`). `--hop` is
not a walk; the flag errors until File/FROM_FILE edges exist.
not a walk; the flag errors. Schema has `FROM_FILE`; search does not
use it ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).

Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues).
Work board: [Gitea issues](https://git.produktor.io/eSlider/2dph/issues)
([epic #16](https://git.produktor.io/eSlider/2dph/issues/16)).
PRs and CI: GitHub [`eSlider/2dph`](https://github.com/eSlider/2dph).

Published docs live here and match live commands.
11 changes: 8 additions & 3 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ status: current
related:
- docs/README.md
- docs/runbook.md
- docs/roadmap.md
---

# Design — facts, info, deduction
Expand Down Expand Up @@ -33,8 +34,9 @@ bin/brain/search.go "question"
is not evidence of absence; `--no-web` / `--root` skip it)
```

`--hop` is not implemented yet (needs File/FROM_FILE edges). The flag is an
error; it is not a graph walk.
`--hop` is not implemented. `FROM_FILE` / `HAS_VERSION` exist in schema;
search does not walk them ([#17](https://git.produktor.io/eSlider/2dph/issues/17)).
The flag is an error; it is not a graph walk.

## Who / What / How / Where / When + evidence

Expand Down Expand Up @@ -94,4 +96,7 @@ Pluggable OpenAI-compatible URL. RAM: `Qwen/Qwen3.5-9B`. Quality:
`prism-ml/Bonsai-27B-gguf` or `Qwen/Qwen3.6-27B`. No official Qwen3.6-9B.
CPU sidecar: compose profile `reasoner` (`OLLAMA_NUM_GPU=0`,
`127.0.0.1:11435`). Bake-off: `bin/reasoner/bakeoff.go`. Weights stay out
of the 2dph image. See [docs/reasoner.md](reasoner.md).
of the 2dph image. See [docs/reasoner.md](reasoner.md).

Gap to v1 (write, hops, corpus, agent loop): [roadmap](roadmap.md),
[epic #16](https://git.produktor.io/eSlider/2dph/issues/16).
67 changes: 67 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
type: explanation
status: current
related:
- PLAN.md
- docs/design.md
- docs/runbook.md
---

# Gap to v1 — detective brain

Goal: a brain that does not assert without proof. Search is deduction
(`facts` ≥2 sources → `info` → `web`). `confirmed` only from the facts root.

**v1 is a living graph the agent can write and walk**, not “more RAG”.

Epic: [Gitea #16](https://git.produktor.io/eSlider/2dph/issues/16).
Milestone: [v1 detective brain](https://git.produktor.io/eSlider/2dph/milestone/12).
Decisions: [PLAN.md](../PLAN.md).

## In (do not reopen)

Read path Go + Zig CGO (D21). HTTP + OpenAPI + MCP (D20). PicoClaw compose
profile + CPU reasoner (D18). Mail sync → import → rebuild. D14 shebangs.
Compose `api` (no CPython) / `index` (Python write). Issues #1–#5, #7–#13.

`POST /ingest` is a rebuild **hint**. `add` is not implemented.

## Blockers

```
question
├─ FTS + HNSW ← in
├─ facts / info roots ← in
├─ web (D17) ← in
├─ Cypher hop ← #17 schema yes, search no
├─ brain/add ACID ← #14 rebuild only
└─ facts+chats corpus ← #18
```

1. **[#14](https://git.produktor.io/eSlider/2dph/issues/14) write** —
`bin/brain/index.go --rebuild` (Python `kblib`). No incremental
`brain/add`. Watch/mail/git cannot become facts “now”.
2. **[#17](https://git.produktor.io/eSlider/2dph/issues/17) hops** —
`Leaf-[:FROM_FILE]->File-[:HAS_VERSION]->Commit-[:AUTHORED]->Person`
exists; `--hop` still errors. Without a walk, D9/D10 are paper.
3. **[#18](https://git.produktor.io/eSlider/2dph/issues/18) corpus** —
rebuild loads repo markdown + mail as `info`. `facts/extract` pairing
and `bin/chats` are not indexed. WhatsApp is a stub. PII stays in `var/`.
4. **[#15](https://git.produktor.io/eSlider/2dph/issues/15) lever/loop** —
2dph is the lever (`search` → `get` → `audit`). PicoClaw is the loop.
Document the contour in-repo; CPU turns need a large context window.
5. **[#19](https://git.produktor.io/eSlider/2dph/issues/19) CI eval** —
recall SoT should be `bin/brain/eval.go` via Zig, not Python `bin/kb/eval`.

## Not v1

[#6](https://git.produktor.io/eSlider/2dph/issues/6) OCR (OQ2), OQ1
contradiction resolution, OQ3 duckdb-md export, OQ4 YAML-first leafs.

## Close epic #16 when

- facts+info can be written without a full rebuild for every leaf
- `--hop` stops erroring and runs a Cypher path from search hits
- ops pairing + chat import land as leafs on rebuild
- MCP tool order is documented and still gated by tests
1 change: 1 addition & 0 deletions docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ bin/brain/stats.go
```

`--hop` is not implemented. Empty web results are `throttled`, not absence.
Gap to v1: [roadmap](roadmap.md) / [epic #16](https://git.produktor.io/eSlider/2dph/issues/16).

Ladybug 0.19: never `DROP INDEX` FTS/VECTOR (ghost catalog). Fresh indexes =
delete `var/kb.lbug` then `--rebuild`.
Expand Down
5 changes: 3 additions & 2 deletions skills/brain/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,9 @@ bin/brain/stats.go # index health
bin/brain/eval.go # recall@5 >= 0.95 gate (Go; Python bin/kb/eval is CI fallback)
```

`bin/kb/search` is a deprecated wrapper. `--hop` errors (File/FROM_FILE edges
are not wired yet); do not treat it as a graph walk.
`bin/kb/search` is a deprecated wrapper. `--hop` errors (schema has
`FROM_FILE`; search does not walk it yet, [#17](https://git.produktor.io/eSlider/2dph/issues/17));
do not treat it as a graph walk.

## Rules

Expand Down
Loading