Fix correctness gaps and make the evaluation honest (v0.3.0)

.github/workflows/ci.yml

@@ -9,6 +9,7 @@ jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
         python-version: ["3.10", "3.12"]
     env:
@@ -20,8 +21,49 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install
+        # `dev` includes mcp and tiktoken: the MCP server is the headline
+        # feature and must be exercised, and the pinned tokenizer is what makes
+        # the published evaluation numbers reproducible.
         run: pip install -e ".[dev]"
       - name: Run tests
         run: pytest -q
       - name: Run evaluation
         run: python eval/run_eval.py
+      - name: Fail if the published results are stale
+        # The README quotes these numbers. If a change moves them, the results
+        # files must be regenerated in the same commit.
+        run: git diff --exit-code -- eval/results.md eval/results.json
+      - name: Run the quickstart example
+        run: python examples/quickstart.py
+
+  mcp-versions:
+    # The MCP server broke once because `mcp` 2.0 renamed the server class and
+    # nothing in CI imported it. Both majors are tested from now on.
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        mcp-version: ["mcp>=1.0,<2", "mcp>=2.0"]
+    env:
+      AGENT_MEMORY_EMBEDDER: hashing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install with ${{ matrix.mcp-version }}
+        run: |
+          pip install -e ".[dev]"
+          pip install "${{ matrix.mcp-version }}"
+      - name: Show resolved version
+        run: pip show mcp | head -2
+      - name: Test the MCP server
+        run: pytest -q tests/test_mcp_server.py
+      - name: Start the server entry point
+        # Smoke-test the console script an agent config actually launches:
+        # it must reach the stdio loop rather than exit on an import error.
+        run: |
+          timeout 10s agent-memory-mcp < /dev/null; status=$?
+          if [ $status -ne 0 ] && [ $status -ne 124 ]; then
+            echo "agent-memory-mcp failed to start (exit $status)"; exit 1
+          fi

.gitignore

@@ -6,6 +6,7 @@ __pycache__/
 build/
 dist/
 .venv/
+.venv*/
 venv/
 .env
 .DS_Store

README.md

@@ -1,6 +1,6 @@
 # Agent Memory Engine
 
-**One shared memory for all your coding agents — Claude Code, Codex CLI, Cursor — over MCP, with token-budgeted recall so it never floods a context window.**
+**One shared memory for your coding agents — Claude Code, Codex CLI, Cursor — over MCP, with token-budgeted recall so it never floods a context window.**
 
 Every coding agent forgets everything between sessions, and none of them can read another's notes: what Claude Code learned about your codebase is invisible to Codex, and vice versa. The common fix — a pile of Markdown memory files loaded into every prompt — has the opposite problem: it burns more tokens every week as the pile grows.
 
@@ -10,16 +10,43 @@ This engine is the small piece in between. Agents write atomic facts, decisions
 
 ## Results
 
-On a labeled benchmark of an agent working across many sessions on one codebase (14 durable memories, 7 fresh-session tasks, top-k = 3):
+On a hand-labeled benchmark of an agent working across many sessions on one codebase (14 durable memories, 7 fresh-session tasks, top-k = 3):
 
 | Arm | Avg context tokens | Recall | Precision | Tokens saved vs baseline |
 | --- | ---: | ---: | ---: | ---: |
 | No memory (control) | 0 | 0.00 | 0.00 | 100% |
-| Full context (baseline — load every memory) | 550 | 1.00 | 0.10 | 0% |
-| **Semantic recall (this engine)** | **120** | **0.93** | **0.43** | **78%** |
-| **Budget recall (≤ 120 tokens, hard cap)** | **113** | **0.93** | **0.43** | **80%** |
+| Full context (baseline — load every memory) | 424 | 1.00 | 0.10 | 0% |
+| *Random k (control)* | *93* | *0.07* | *0.05* | *78%* |
+| **Targeted retrieval (this engine)** | **93** | **0.93** | **0.43** | **78%** |
+| **Budget recall (≤ 120 tokens, hard cap)** | **93** | **0.93** | **0.43** | **78%** |
 
-**Semantic recall retrieves 93% of the relevant memories while loading 78% fewer context tokens** than dumping every memory file — and under a **hard 120-token cap recall is unchanged**, so the cost of memory stays fixed as the store grows. Numbers are reproduced by `python eval/run_eval.py` and regenerated in CI with the deterministic offline embedder, so they are byte-stable.
+**Read the random row first.** It loads three memories picked at random, so it reports the *same* 78% token saving with 0.07 recall. The saving is arithmetic — you loaded 3 of 14 memories — and proves nothing on its own. The claim worth making is the **0.86 recall gap between random and retrieval at identical token cost**.
+
+Reproduce with `python eval/run_eval.py`; the full report is in [eval/results.md](eval/results.md).
+
+### What these numbers don't show
+
+- **The default embedder is lexical, not semantic.** It is feature hashing over word and character n-grams, so it matches shared wording. Re-run the same tasks with paraphrased queries that avoid the memories' vocabulary and recall drops from **0.93 to 0.43**. That gap is published in the results, not tuned away, and it is the reason the optional `sentence-transformers` backend exists.
+
+  | Query phrasing | Word overlap with gold | Recall |
+  | --- | ---: | ---: |
+  | Developer phrasing (as labelled) | 40% | 0.93 |
+  | Outsider paraphrase | 3% | 0.43 |
+
+- **The precision column is nearly meaningless for the baseline.** Full context scores 0.10 because that is `|relevant| / |store|` — an artefact of loading everything.
+- **7 tasks, 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim.
+
+### Does the cost stay flat as the store grows?
+
+Adding unrelated memories from the same project, re-measuring against the same labels:
+
+| Memories in store | Full context tokens | Budget recall tokens | Budget recall |
+| ---: | ---: | ---: | ---: |
+| 14 | 424 | 93 | 0.93 |
+| 34 | 805 | 87 | 0.93 |
+| 54 | 1159 | 82 | 0.93 |
+
+The baseline grows with the store. The budgeted arm does not, and recall holds.
 
 ## The cross-agent loop
 
@@ -39,18 +66,20 @@ sequenceDiagram
 Three habits, three tools:
 
 - **`memory_boot(task, budget_tokens)`** — call once at session start: the latest handoff from the *previous agent, whichever tool it was*, plus the memories most relevant to the task, packed under one token budget.
-- **`memory_write(text, type)`** — persist a durable fact, decision or issue the moment it's learned. Near-duplicates are dropped automatically.
+- **`memory_write(text, type)`** — persist a durable fact, decision or issue the moment it's learned. Near-duplicates are dropped, and the tool says so rather than reporting a save that didn't happen.
 - **`memory_handoff(done, next_steps, warnings)`** — call at session end so the next agent continues instead of rediscovering.
 
-Tool outputs are deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window.
+Plus `memory_list`, `memory_update` and `memory_forget`, because memory you cannot correct is worse than no memory: a stale `state` entry keeps being recalled and quietly misleads every later session.
+
+Tool outputs are deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. The flip side is that the agent can't judge relevance itself, so weak matches are filtered out before they're returned (see [Relevance floor](#relevance-floor)).
 
 ## Hook it up to your agents
 
 ```bash
 pip install -e ".[mcp]"
 ```
 
-Point every agent at the **same store path**, and give each its own name:
+Works with both `mcp` 1.x and 2.x. Point every agent at the **same store path**, and give each its own name:
 
 **Claude Code**
 
@@ -83,24 +112,42 @@ env = { AGENT_MEMORY_PATH = "~/.agent_memory/store.json", AGENT_MEMORY_AGENT = "
 }
 ```
 
-Each memory now carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`.
+Each memory now carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`. Agents may run at the same time: writes take a lock, merge, and land atomically, and an already-running server picks up another agent's writes on its next read.
 
 ## How it works
 
-- **Pluggable embedder** ([embeddings.py](src/agent_memory/embeddings.py)) — a small `Embedder` protocol with two backends: a dependency-free, deterministic `HashingEmbedder` (default; runs offline and in CI) and a `SentenceTransformerEmbedder` for real semantic embeddings. Same API, swap by installing one package.
+- **Pluggable embedder** ([embeddings.py](src/agent_memory/embeddings.py)) — a small `Embedder` protocol with two backends: a dependency-free, deterministic `HashingEmbedder` (default; runs offline and in CI) and a `SentenceTransformerEmbedder` for real semantic embeddings. Same API, swap by installing one package. Each carries its own calibrated relevance floor, because the two score on different scales.
 - **Vector store** ([store.py](src/agent_memory/store.py)) — exact brute-force cosine over a numpy matrix, JSON persistence, near-duplicate suppression on write, and greedy **budget packing**: recall walks results in relevance order and skips anything that would overflow the caller's token budget. An agent's memory is hundreds of entries, not millions, so this is instant and exact; FAISS/Chroma can slot behind the same API later.
-- **MCP server** ([mcp_server.py](src/agent_memory/mcp_server.py)) — exposes `memory_boot`, `memory_recall`, `memory_write`, `memory_handoff`, `memory_stats` as agent tools.
-- **Evaluation harness** ([run_eval.py](eval/run_eval.py)) — measures token cost and retrieval precision/recall across four arms on a labeled dataset, including the hard-budget arm.
+- **Durable writes** — every write takes a cross-process lock, re-reads anything another agent appended, and replaces the file atomically. A crash mid-write cannot truncate the store, and two agents writing at once cannot silently overwrite each other. Embeddings are stored as base64 float16, which keeps the file about 5× smaller than JSON float lists.
+- **MCP server** ([mcp_server.py](src/agent_memory/mcp_server.py)) — exposes `memory_boot`, `memory_recall`, `memory_write`, `memory_handoff`, `memory_list`, `memory_update`, `memory_forget` and `memory_stats` as agent tools.
+- **Evaluation harness** ([run_eval.py](eval/run_eval.py)) — measures token cost and retrieval precision/recall across five arms, including the random control and the hard-budget arm, plus the paraphrase gap, a store-growth scaling test and the relevance-floor sweep.
+
+### Relevance floor
+
+Without a floor, a query about something the store knows nothing about still returns *k* memories — and since tool output hides the scores, the calling agent has no way to tell noise from signal. `min_score` drops weak matches. The shipped default comes from a published sweep:
+
+| min_score | Labelled recall | Off-topic memories returned |
+| ---: | ---: | ---: |
+| 0.00 | 0.93 | 12/12 |
+| 0.10 | 0.93 | 10/12 |
+| **0.15** | **0.93** | **6/12** |
+| 0.20 | 0.93 | 2/12 |
+
+The floor lives on the embedder (`HashingEmbedder.recommended_min_score = 0.15`), overridable with `AGENT_MEMORY_MIN_SCORE` or the `--min-score` flag. A higher floor cuts more noise on this benchmark, but the lowest-scoring genuinely-relevant memory in it sits at 0.13, so the default stays close to that observed edge. The `sentence-transformers` value is **not** calibrated here — measure it on your own store.
 
 ## Quickstart (60 seconds)
 
 ```bash
-pip install -e .            # core engine + CLI (numpy only)
-python eval/run_eval.py     # reproduce the results table above
+pip install -e ".[dev]"     # engine, CLI, MCP server, exact tokenizer
+python eval/run_eval.py     # reproduce the results above
+python examples/quickstart.py
 
 agent-memory --agent claude-code write "Bookings are stored in UTC." --type decision
 agent-memory --agent claude-code handoff --done "Fixed double emails." --next "Add rate limiting."
 agent-memory boot "continue the rate limiting work" --budget 200
+agent-memory list                                   # ids, so you can fix mistakes
+agent-memory update mem_0003 "Bookings are stored in UTC; the UI converts."
+agent-memory forget mem_0007
 ```
 
 In Python:
@@ -111,7 +158,9 @@ from agent_memory import MemoryStore
 store = MemoryStore(path="~/.agent_memory/store.json")
 store.write("The chatbot uses Google Gemini in server/gemini-chat.ts.",
             type="decision", agent="claude-code")
-for hit in store.recall("where is the chatbot configured", k=3, budget_tokens=150):
+
+handoff, hits = store.boot("where is the chatbot configured", budget_tokens=150)
+for hit in hits:
     print(hit.score, hit.entry.text)
 ```
 
@@ -121,33 +170,39 @@ for hit in store.recall("where is the chatbot configured", k=3, budget_tokens=15
 pip install -e ".[real]"    # sentence-transformers + tiktoken
 ```
 
-`default_embedder()` automatically prefers the real model when installed. Force the offline embedder anywhere with `AGENT_MEMORY_EMBEDDER=hashing` (CI does this for stable numbers).
+`default_embedder()` prefers the real model when installed, and warns on stderr if it falls back rather than switching silently. Force the offline embedder with `AGENT_MEMORY_EMBEDDER=hashing` (CI does this for stable numbers), or demand the real one with `AGENT_MEMORY_EMBEDDER=sentence-transformers` to turn a missing dependency into an error instead of a downgrade.
+
+Switching backends on an existing store is safe: vectors from a different embedder aren't comparable, so the store detects the mismatch on load and re-embeds from the stored text instead of returning meaningless scores.
 
 ## Migrate an existing Markdown scaffold
 
 ```bash
-python scripts/ingest_markdown.py path/to/agent-memory/   # one memory per file/section
+python scripts/ingest_markdown.py path/to/agent-memory/ --path ~/.agent_memory/store.json
 ```
 
+One memory per `##` section, with long sections split so that every ingested memory stays small enough to be recalled under a budget.
+
+## Limits and trust boundary
+
+- **Memories are injected into agent context verbatim.** Anything an agent writes to the store — including text it read from a webpage, an issue tracker or a dependency — will be replayed into a *different* agent's context later, where it reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally.
+- **The store is a local file.** No auth, no encryption, no server. It belongs next to your code, not on a shared host.
+- **Retrieval is lexical by default.** See the paraphrase gap above.
+- **The benchmark is small and self-authored.** It is a regression check for the engine, not evidence about your codebase.
+
 ## Develop
 
 ```bash
 pip install -e ".[dev]"
 pytest -q
 ```
 
-## Design notes
-
-- **Small on purpose.** A memory layer that itself eats context defeats its point. Recall output is compact, budget-capped, and score-free; the whole engine is a few hundred lines with one required dependency (numpy).
-- **Offline-safe by default.** No model download or network needed to run, test, or evaluate — the heavy backends are optional extras. This keeps CI deterministic and the repo cloneable-and-runnable in one step.
-- **Honest evaluation.** The benchmark is small and labeled by hand; the one missed memory (`task_1`) is reported rather than tuned away. The harness includes a no-memory control and a full-context baseline so the comparison is fair.
-- **Boring where it counts.** Brute-force exact search instead of an ANN index, JSON instead of a database — chosen deliberately for the actual scale and swappable behind the same API.
+CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the committed results are stale, and separately tests the MCP server against both `mcp` 1.x and 2.x.
 
 ## Roadmap
 
 - LLM-based compaction: summarize/dedup `state` and `worklog` memories, extract durable facts from a raw session transcript.
 - Recency- and type-aware ranking (decay old `state`, never drop `decision`).
-- `sentence-transformers` numbers published alongside the hashing baseline in CI.
+- `sentence-transformers` numbers published alongside the hashing baseline in CI, including a calibrated relevance floor.
 - Optional FAISS backend for large stores.
 
 ## License