diff --git a/README.md b/README.md
index 3c5e91a..d9f057a 100644
--- a/README.md
+++ b/README.md
@@ -1,54 +1,61 @@
# Agent Memory Engine
+[](https://github.com/Ninadnj/ai-agent-memory-scaffold/actions/workflows/ci.yml)
+[](https://www.python.org/downloads/)
+[](LICENSE)
+
**One shared memory for your coding agents — Claude Code, Codex CLI, Cursor — over MCP, with token-budgeted recall so it never floods a context window.**
-Every coding agent forgets everything between sessions, and none of them can read another's notes: what Claude Code learned about your codebase is invisible to Codex, and vice versa. The common fix — a pile of Markdown memory files loaded into every prompt — has the opposite problem: it burns more tokens every week as the pile grows.
+Coding agents forget everything between sessions, and none of them can read another's notes: what Claude Code learned about your codebase is invisible to Codex. The usual fix — a pile of Markdown files loaded into every prompt — has the opposite problem, costing more tokens every week as the pile grows.
-This engine is the small piece in between. Agents write atomic facts, decisions and handoffs to one store; at the start of a task any agent recalls **only the memories relevant to that task, under a hard token budget you set**. It runs as a [Model Context Protocol](https://modelcontextprotocol.io) server, so every MCP-capable agent shares the same memory — and every memory records which agent wrote it.
+This engine sits in between. Agents write short, atomic memories to one store; at the start of a task, any agent gets back **only the memories relevant to that task, under a token budget you set**. It runs as a [Model Context Protocol](https://modelcontextprotocol.io) server, so every MCP-capable tool shares the same memory, and every memory records which agent wrote it.
-> This engine is the working evolution of the Markdown memory *convention* this repository originally hosted — still included and usable in [`scaffold/`](scaffold/) — into a measured retrieval *system*.
+> The original Markdown *convention* this repository hosted is still here and still usable, in [`scaffold/`](scaffold/). This engine is its measured successor.
-## Results
+---
-On a hand-labeled benchmark of an agent working across many sessions on one codebase (14 durable memories, 7 fresh-session tasks, top-k = 3):
+## Quick start
-| Arm | Avg context tokens | Recall | Precision | Tokens saved vs baseline |
-| --- | ---: | ---: | ---: | ---: |
-| No memory (control) | 0 | 0.00 | 0.00 | 100% |
-| Full context (baseline — load every memory) | 424 | 1.00 | 0.10 | 0% |
-| *Random k (control)* | *93* | *0.07* | *0.05* | *78%* |
-| **Targeted retrieval (this engine)** | **93** | **0.93** | **0.43** | **78%** |
-| **Budget recall (≤ 120 tokens, hard cap)** | **93** | **0.93** | **0.43** | **78%** |
+```bash
+git clone https://github.com/Ninadnj/ai-agent-memory-scaffold.git
+cd ai-agent-memory-scaffold
+pip install -e ".[mcp,real]"
+```
-**Read the random row first.** It loads three memories picked at random, so it reports the *same* 78% token saving with 0.07 recall. The saving is arithmetic — you loaded 3 of 14 memories — and proves nothing on its own. The claim worth making is the **0.86 recall gap between random and retrieval at identical token cost**.
+| Install | Gets you |
+| --- | --- |
+| `pip install -e .` | Engine + CLI. numpy only, fully offline. |
+| `pip install -e ".[mcp,real]"` | **Recommended.** Adds the MCP server and semantic embeddings. |
+| `pip install -e ".[dev]"` | Everything above minus the model, plus pytest. What CI runs. |
-Reproduce with `python eval/run_eval.py`; the full report is in [eval/results.md](eval/results.md).
+Try it in your terminal:
-### What these numbers don't show
+```bash
+agent-memory write "Bookings are stored in UTC; the UI converts to local time." --type decision
+agent-memory write "Admin routes are guarded by requireAdmin in server/auth.ts." --type decision
-- **The default embedder is lexical, not semantic.** It is feature hashing over word and character n-grams, so it matches shared wording. Re-run the same tasks with paraphrased queries that avoid the memories' vocabulary and recall drops from **0.93 to 0.43**. That gap is published in the results, not tuned away, and it is the reason the optional `sentence-transformers` backend exists.
+agent-memory recall "a client saw the wrong hour for their appointment"
+# 0.41 [decision] Bookings are stored in UTC; the UI converts to local time.
- | Query phrasing | Word overlap with gold | Recall |
- | --- | ---: | ---: |
- | Developer phrasing (as labelled) | 40% | 0.93 |
- | Outsider paraphrase | 3% | 0.43 |
+agent-memory recall "best sourdough bread recipe"
+# No relevant memories.
+```
-- **The precision column is nearly meaningless for the baseline.** Full context scores 0.10 because that is `|relevant| / |store|` — an artefact of loading everything.
-- **7 tasks, 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim.
+That first query shares no words with the memory it found, and the second returns nothing rather than guessing. Both behaviours are the point.
-### Does the cost stay flat as the store grows?
+---
-Adding unrelated memories from the same project, re-measuring against the same labels:
+## How it works
-| Memories in store | Full context tokens | Budget recall tokens | Budget recall |
-| ---: | ---: | ---: | ---: |
-| 14 | 424 | 93 | 0.93 |
-| 34 | 805 | 87 | 0.93 |
-| 54 | 1159 | 82 | 0.93 |
+Three habits, three tools:
-The baseline grows with the store. The budgeted arm does not, and recall holds.
+| Tool | When | What it does |
+| --- | --- | --- |
+| `memory_boot(task, budget_tokens)` | Start of a session | Returns the previous agent's handoff plus the memories most relevant to your task, packed under one token budget. |
+| `memory_write(text, type)` | The moment something is learned | Saves one durable fact. Near-duplicates are skipped — and it tells you so instead of reporting a save that didn't happen. |
+| `memory_handoff(done, next, warnings)` | End of a session | Leaves a note so the next agent continues instead of rediscovering. |
-## The cross-agent loop
+Plus `memory_list`, `memory_update` and `memory_forget` — memory you can't correct is worse than no memory, because a stale note keeps being recalled with full confidence.
```mermaid
sequenceDiagram
@@ -63,60 +70,114 @@ sequenceDiagram
Note over CX: picks up exactly where Claude Code stopped
```
-Three habits, three tools:
-
-- **`memory_boot(task, budget_tokens)`** — call once at session start: the latest handoff from the *previous agent, whichever tool it was*, plus the memories most relevant to the task, packed under one token budget.
-- **`memory_write(text, type)`** — persist a durable fact, decision or issue the moment it's learned. Near-duplicates are dropped, and the tool says so rather than reporting a save that didn't happen.
-- **`memory_handoff(done, next_steps, warnings)`** — call at session end so the next agent continues instead of rediscovering.
-
-Plus `memory_list`, `memory_update` and `memory_forget`, because memory you cannot correct is worse than no memory: a stale `state` entry keeps being recalled and quietly misleads every later session.
+Tool output is deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. The trade-off is that the agent can't judge relevance itself, so weak matches are filtered out before they're returned.
-Tool outputs are deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. The flip side is that the agent can't judge relevance itself, so weak matches are filtered out before they're returned (see [Relevance floor](#relevance-floor)).
+---
-## Hook it up to your agents
-
-```bash
-pip install -e ".[mcp]"
-```
+## Connect your agents
-Works with both `mcp` 1.x and 2.x. Point every agent at the **same store path**, and give each its own name:
+Give each agent its own name. The store needs no configuration — it follows the project you're in.
**Claude Code**
```bash
-claude mcp add agent-memory -e AGENT_MEMORY_PATH=~/.agent_memory/store.json \
- -e AGENT_MEMORY_AGENT=claude-code -- agent-memory-mcp
+claude mcp add agent-memory -e AGENT_MEMORY_AGENT=claude-code -- agent-memory-mcp
```
-**Codex CLI** (`~/.codex/config.toml`)
+**Codex CLI** — `~/.codex/config.toml`
```toml
[mcp_servers.agent-memory]
command = "agent-memory-mcp"
-env = { AGENT_MEMORY_PATH = "~/.agent_memory/store.json", AGENT_MEMORY_AGENT = "codex" }
+env = { AGENT_MEMORY_AGENT = "codex" }
```
-**Cursor** (`.cursor/mcp.json`)
+**Cursor** — `.cursor/mcp.json`
```json
{
"mcpServers": {
"agent-memory": {
"command": "agent-memory-mcp",
- "env": {
- "AGENT_MEMORY_PATH": "~/.agent_memory/store.json",
- "AGENT_MEMORY_AGENT": "cursor"
- }
+ "env": { "AGENT_MEMORY_AGENT": "cursor" }
}
}
}
```
-Each memory now carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`. Agents may run at the same time: writes take a lock, merge, and land atomically, and an already-running server picks up another agent's writes on its next read.
+Works with `mcp` 1.x and 2.x. Every memory carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next. Agents may run at the same time: writes take a lock, merge, and land atomically, and a running server sees another agent's writes on its next read.
+
+### One store per project
+
+Memories are scoped to the repository you're working in:
+
+1. `AGENT_MEMORY_PATH`, if you set it
+2. `.agent_memory/store.json` in the current git repository — **the normal case**
+3. `~/.agent_memory/store.json`, when you're not in a repository
+
+This matters more than it sounds. Recall matches on similarity alone, so one shared store would let another project's answer to *"how do we deploy?"* surface while you work here.
+
+```bash
+agent-memory stats # which store am I using?
+agent-memory --global stats # the cross-project store, when you want it
+```
+
+Add `.agent_memory/` to your `.gitignore` unless you mean to commit the memories — sharing them with a team is reasonable, but it should be deliberate.
+
+---
+
+## Does it actually work?
+
+A hand-labelled benchmark: 14 memories from one codebase, 7 fresh-session tasks, top-k = 3.
+
+| Arm | Avg context tokens | Recall | Tokens saved |
+| --- | ---: | ---: | ---: |
+| No memory (control) | 0 | 0.00 | 100% |
+| Full context — load everything | 424 | 1.00 | 0% |
+| *Random k (control)* | *93* | *0.07* | *78%* |
+| **Targeted retrieval** | **93** | **0.93** | **78%** |
+| **Budget recall (≤ 120 tokens)** | **93** | **0.93** | **78%** |
+
+**Read the random row first.** Three memories picked at random report the *same* 78% saving at 0.07 recall. The saving is arithmetic — you loaded 3 of 14 — and proves nothing by itself. The real claim is the **0.86 recall gap at identical token cost**.
+
+Reproduce with `python eval/run_eval.py`. Full report: [eval/results.md](eval/results.md).
+
+### Cost stays flat as memory grows
+
+| Memories in store | Full context | Budget recall | Recall |
+| ---: | ---: | ---: | ---: |
+| 14 | 424 tokens | 93 tokens | 0.93 |
+| 34 | 805 tokens | 87 tokens | 0.93 |
+| 54 | 1159 tokens | 82 tokens | 0.93 |
+
+The baseline grows with the store. The budgeted arm doesn't, and recall holds.
+
+### Which embedder should you use?
+
+The offline default is **lexical** — it matches shared wording, not meaning. So every task in the benchmark carries a second phrasing that deliberately avoids the memories' vocabulary, and both backends are scored on both:
+
+| | Hashing (default, offline) | MiniLM (`real` extra) |
+| --- | ---: | ---: |
+| Developer phrasing — 40% word overlap | **0.93** | 0.86 |
+| Outsider paraphrase — 3% word overlap | 0.43 | **0.79** |
+| Average | 0.68 | **0.82** |
+| Off-topic queries rejected | 6/12 | **12/12** |
+
+It isn't a clean sweep, and that's the useful finding. **Exact word matching genuinely wins when the words match** — and needs no model download. But everyday use is the second row: you write a memory in March and ask about it in July, in different words.
+
+**Use the `real` extra day to day**; keep hashing for CI, air-gapped machines, or when a ~90 MB download isn't welcome. Switching is safe — a store embedded by one backend is re-embedded on load by the other, never compared across incompatible vectors. Full report: [results_sentence_transformers.md](eval/results_sentence_transformers.md).
+
+### What the numbers don't show
+
+- **7 tasks and 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim.
+- **Precision is misleading for the baseline.** Loading everything scores 0.10 simply because that's `|relevant| / |store|`.
+- **MiniLM numbers aren't reproduced in CI** — they need a model download. CI regenerates and diffs the hashing results only.
+
+---
## Architecture
-Every agent talks to one local store. **MCP is the integration surface** — a stdio server each agent launches as a subprocess — and the CLI and Python API are thin alternatives onto the same engine.
+Every agent talks to one local store. **MCP is the integration surface** — a stdio server each agent launches as a subprocess. The CLI and Python API are thin alternatives onto the same engine.
```mermaid
flowchart TB
@@ -149,20 +210,20 @@ flowchart TB
CORE --> DISK
```
-The guards sit between the tools and the store on purpose. Tool output is compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. That means the agent *cannot* judge relevance for itself, so the floor and the budget are enforced before anything is handed back.
+The guards sit between the tools and the store on purpose: the caller is a model that cannot see similarity scores, so the floor and the budget are enforced before anything is handed back. A Python caller *can* see scores, so `MemoryStore.recall()` applies no floor unless asked.
| Module | Responsibility |
| --- | --- |
-| [`mcp_server.py`](src/agent_memory/mcp_server.py) | MCP tools over stdio. Works with `mcp` 1.x and 2.x. |
+| [`mcp_server.py`](src/agent_memory/mcp_server.py) | MCP tools over stdio. Supports `mcp` 1.x and 2.x. |
| [`cli.py`](src/agent_memory/cli.py) | The same operations as shell commands. |
-| [`store.py`](src/agent_memory/store.py) | Retrieval, budget packing, durability, the on-disk format. |
+| [`store.py`](src/agent_memory/store.py) | Retrieval, budget packing, durability, on-disk format. |
| [`embeddings.py`](src/agent_memory/embeddings.py) | `Embedder` protocol, two backends, per-backend relevance floor. |
| [`tokens.py`](src/agent_memory/tokens.py) | Token accounting — the unit the budget is denominated in. |
| [`eval/run_eval.py`](eval/run_eval.py) | Five-arm benchmark, paraphrase gap, scaling test, floor sweep. |
### The recall path
-Retrieval is exact brute-force cosine over a numpy matrix — an agent's memory for one project is hundreds of entries, not millions, so this is instant and exact. FAISS or Chroma can slot in behind the same API if a store ever outgrows it.
+Retrieval is exact brute-force cosine over a numpy matrix. One project's memory is hundreds of entries, not millions, so this is both instant and exact; FAISS or Chroma can slot in behind the same API if that ever changes.
```mermaid
flowchart LR
@@ -177,7 +238,16 @@ flowchart LR
G --> H["≤ k memories,
≤ budget tokens"]
```
-Budget packing is greedy rather than all-or-nothing: an entry that would overflow the remaining budget is skipped and a smaller, lower-ranked one gets its chance. `memory_boot` spends one budget across both the handoff and the recalled memories, so the total is capped whatever the store contains.
+Budget packing is greedy, not all-or-nothing: an entry that would overflow the remaining budget is skipped so a smaller, lower-ranked one gets its chance. `memory_boot` spends one budget across both the handoff and the recalled memories, so the total is capped whatever the store holds.
+
+**The relevance floor** exists because without it, a query about something the store knows nothing about still returns *k* memories — and tool output hides the scores, so the agent can't tell. Each backend carries its own value, since they score on different scales:
+
+| Backend | Floor | Chosen because |
+| --- | ---: | --- |
+| `HashingEmbedder` | 0.15 | No labelled recall lost, off-topic matches halved. The lowest-scoring genuinely relevant memory sits at 0.13, so the floor stays near that observed edge. |
+| `SentenceTransformerEmbedder` | 0.20 | Off-topic queries are fully rejected from 0.15 and recall is flat to 0.35, so the sweep alone can't choose. Set just under 0.22 — the score of a real paraphrase against the memory that answers it. |
+
+Override with `AGENT_MEMORY_MIN_SCORE` or `--min-score`. The honest consequence: when a query is genuinely ambiguous, recall returns *nothing* rather than a coin flip the agent would read as fact.
### The write path
@@ -202,45 +272,51 @@ sequenceDiagram
CX->>L: release
```
-- **No lost updates.** A write re-reads the file under the lock, so an agent that has been idle for an hour still appends rather than overwrites. Verified with 8 processes writing concurrently.
-- **No torn files.** The new contents land via `os.replace`, which is atomic — a crash mid-write leaves the previous store intact, never a truncated one.
-- **Fresh reads.** A long-running MCP server reloads when the file changes on disk, so it sees another agent's writes without a restart.
-- **Compact.** Embeddings are stored as base64 float16, roughly 5× smaller than JSON float lists.
-
-### Relevance floor
+- **No lost updates.** A write re-reads under the lock, so an agent idle for an hour appends rather than overwrites. Verified with 8 concurrent processes.
+- **No torn files.** Contents land via `os.replace`, which is atomic — a crash mid-write leaves the previous store intact.
+- **Fresh reads.** A long-running server reloads when the file changes, so it sees other agents' writes without a restart.
+- **Compact.** Embeddings are base64 float16, roughly 5× smaller than JSON float lists.
-Without a floor, a query about something the store knows nothing about still returns *k* memories — and since tool output hides the scores, the calling agent has no way to tell noise from signal. `min_score` drops weak matches. The shipped default comes from a published sweep:
+---
-| min_score | Labelled recall | Off-topic memories returned |
-| ---: | ---: | ---: |
-| 0.00 | 0.93 | 12/12 |
-| 0.10 | 0.93 | 10/12 |
-| **0.15** | **0.93** | **6/12** |
-| 0.20 | 0.93 | 2/12 |
+## Reference
-The floor lives on the embedder (`HashingEmbedder.recommended_min_score = 0.15`), overridable with `AGENT_MEMORY_MIN_SCORE` or the `--min-score` flag. A higher floor cuts more noise on this benchmark, but the lowest-scoring genuinely-relevant memory in it sits at 0.13, so the default stays close to that observed edge. The `sentence-transformers` value is **not** calibrated here — measure it on your own store.
-
-## Quickstart (60 seconds)
+### CLI
```bash
-pip install -e ".[dev]" # engine, CLI, MCP server, exact tokenizer
-python eval/run_eval.py # reproduce the results above
-python examples/quickstart.py
-
-agent-memory --agent claude-code write "Bookings are stored in UTC." --type decision
-agent-memory --agent claude-code handoff --done "Fixed double emails." --next "Add rate limiting."
-agent-memory boot "continue the rate limiting work" --budget 200
-agent-memory list # ids, so you can fix mistakes
-agent-memory update mem_0003 "Bookings are stored in UTC; the UI converts."
-agent-memory forget mem_0007
+agent-memory write "" --type decision # save a memory
+agent-memory recall "" -k 3 --budget 200 # find relevant memories
+agent-memory boot "" --budget 300 # handoff + relevant memories
+agent-memory handoff --done "..." --next "..." # leave a note for the next session
+agent-memory list --type decision # ids, so you can fix mistakes
+agent-memory update mem_0003 "" # revise a memory
+agent-memory forget mem_0007 # delete a stale memory
+agent-memory stats # store path, counts, embedder
```
-In Python:
+Global flags: `--path` (explicit store), `--global` (cross-project store), `--agent` (who is writing).
+
+### Memory types
+
+`project` · `decision` · `issue` · `state` · `handoff` · `worklog` · `fact`
+
+They mirror the original Markdown scaffold's files, so migration is one-to-one.
+
+### Environment variables
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `AGENT_MEMORY_PATH` | project store | Force a specific store file. |
+| `AGENT_MEMORY_AGENT` | *(empty)* | Name recorded on every memory this agent writes. |
+| `AGENT_MEMORY_EMBEDDER` | `auto` | `hashing` forces offline; `sentence-transformers` makes a missing model an error instead of a silent downgrade. |
+| `AGENT_MEMORY_MIN_SCORE` | per-backend | Override the relevance floor. |
+
+### Python
```python
-from agent_memory import MemoryStore
+from agent_memory import MemoryStore, default_store_path
-store = MemoryStore(path="~/.agent_memory/store.json")
+store = MemoryStore(path=default_store_path()) # this project's store
store.write("The chatbot uses Google Gemini in server/gemini-chat.ts.",
type="decision", agent="claude-code")
@@ -249,30 +325,24 @@ for hit in hits:
print(hit.score, hit.entry.text)
```
-## Real semantic embeddings
+### Migrate an existing Markdown scaffold
```bash
-pip install -e ".[real]" # sentence-transformers + tiktoken
+python scripts/ingest_markdown.py path/to/agent-memory/ --path .agent_memory/store.json
```
-`default_embedder()` prefers the real model when installed, and warns on stderr if it falls back rather than switching silently. Force the offline embedder with `AGENT_MEMORY_EMBEDDER=hashing` (CI does this for stable numbers), or demand the real one with `AGENT_MEMORY_EMBEDDER=sentence-transformers` to turn a missing dependency into an error instead of a downgrade.
-
-Switching backends on an existing store is safe: vectors from a different embedder aren't comparable, so the store detects the mismatch on load and re-embeds from the stored text instead of returning meaningless scores.
-
-## Migrate an existing Markdown scaffold
-
-```bash
-python scripts/ingest_markdown.py path/to/agent-memory/ --path ~/.agent_memory/store.json
-```
+One memory per `##` section, with long sections split so every ingested memory stays small enough to be recalled under a budget.
-One memory per `##` section, with long sections split so that every ingested memory stays small enough to be recalled under a budget.
+---
-## Limits and trust boundary
+## Limits
-- **Memories are injected into agent context verbatim.** Anything an agent writes to the store — including text it read from a webpage, an issue tracker or a dependency — will be replayed into a *different* agent's context later, where it reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally.
+- **Nothing writes memories for you.** The agent has to choose to call `memory_write` and `memory_handoff`. If it doesn't, the store stays empty. Prompt instructions or a session hook make this reliable; that isn't built in yet.
+- **Memories never expire.** `state` and `worklog` entries go stale but keep being recalled as confidently as a fresh decision. Correct them with `memory_update` / `memory_forget` until decay lands.
+- **Retrieval is lexical unless you install the `real` extra.** See [the comparison](#which-embedder-should-you-use).
+- **Memories are replayed verbatim into other agents' context.** Anything an agent writes — including text it read from a webpage, an issue tracker or a dependency — later reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally.
- **The store is a local file.** No auth, no encryption, no server. It belongs next to your code, not on a shared host.
-- **Retrieval is lexical by default.** See the paraphrase gap above.
-- **The benchmark is small and self-authored.** It is a regression check for the engine, not evidence about your codebase.
+- **The benchmark is small and self-authored.** It's a regression check for the engine, not evidence about your codebase.
## Develop
@@ -281,13 +351,13 @@ pip install -e ".[dev]"
pytest -q
```
-CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the committed results are stale, and separately tests the MCP server against both `mcp` 1.x and 2.x.
+74 tests. CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the committed results are stale, and separately tests the MCP server against both `mcp` 1.x and 2.x. Tests for the optional semantic backend skip automatically unless the `real` extra is installed.
## Roadmap
-- LLM-based compaction: summarize/dedup `state` and `worklog` memories, extract durable facts from a raw session transcript.
+- Session hooks, so writing a handoff doesn't depend on the agent remembering to.
- Recency- and type-aware ranking (decay old `state`, never drop `decision`).
-- `sentence-transformers` numbers published alongside the hashing baseline in CI, including a calibrated relevance floor.
+- LLM-based compaction: summarise and dedup `state`/`worklog`, extract durable facts from a session transcript.
- Optional FAISS backend for large stores.
## License
diff --git a/eval/results.json b/eval/results.json
index 564d4a9..1a917eb 100644
--- a/eval/results.json
+++ b/eval/results.json
@@ -207,6 +207,24 @@
"labelled_recall": 0.929,
"off_topic_memories_returned": 2,
"off_topic_max": 12
+ },
+ {
+ "min_score": 0.3,
+ "labelled_recall": 0.714,
+ "off_topic_memories_returned": 1,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.35,
+ "labelled_recall": 0.571,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.4,
+ "labelled_recall": 0.5,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
}
]
}
diff --git a/eval/results.md b/eval/results.md
index bb6c09c..984e73c 100644
--- a/eval/results.md
+++ b/eval/results.md
@@ -44,6 +44,9 @@ Distractor memories are added from the same project. The baseline grows linearly
| 0.15 | 0.93 | 6/12 |
| 0.20 | 0.93 | 2/12 |
| 0.25 | 0.93 | 2/12 |
+| 0.30 | 0.71 | 1/12 |
+| 0.35 | 0.57 | 0/12 |
+| 0.40 | 0.50 | 0/12 |
Four deliberately off-topic queries stand in for a task the store knows nothing about. Without a floor the engine returns k memories anyway, and the MCP layer strips scores, so the agent cannot tell. `HashingEmbedder.recommended_min_score` is set from this sweep.
@@ -51,4 +54,4 @@ Four deliberately off-topic queries stand in for a task the store knows nothing
- 7 tasks and 10 gold labels, all hand-written by the author. This is an engineering check, not a production-scale claim; one retrieval either way moves recall by ~0.07.
- The labels, the queries and the retriever all come from the same person, which is exactly the setup that flatters a retriever. The paraphrase arm exists to push back on that.
-- Numbers use the offline hashing embedder so they are stable in CI. The `sentence-transformers` backend is not measured here.
+- Numbers use the offline hashing embedder so they are stable in CI. The `sentence-transformers` backend is measured separately in `results_sentence_transformers.md`.
diff --git a/eval/results_sentence_transformers.json b/eval/results_sentence_transformers.json
new file mode 100644
index 0000000..cd9289e
--- /dev/null
+++ b/eval/results_sentence_transformers.json
@@ -0,0 +1,230 @@
+{
+ "k": 3,
+ "budget": 120,
+ "seed": 0,
+ "exact_tokenizer": true,
+ "embedder": "SentenceTransformerEmbedder",
+ "n_memories": 14,
+ "n_tasks": 7,
+ "summary": {
+ "no_memory": {
+ "avg_context_tokens": 0.0,
+ "precision": 0.0,
+ "recall": 0.0,
+ "token_reduction_vs_baseline": 100.0
+ },
+ "full_context": {
+ "avg_context_tokens": 424.0,
+ "precision": 0.102,
+ "recall": 1.0,
+ "token_reduction_vs_baseline": 0.0
+ },
+ "random_k": {
+ "avg_context_tokens": 93.4,
+ "precision": 0.048,
+ "recall": 0.071,
+ "token_reduction_vs_baseline": 78.0
+ },
+ "retrieval": {
+ "avg_context_tokens": 93.4,
+ "precision": 0.381,
+ "recall": 0.857,
+ "token_reduction_vs_baseline": 78.0
+ },
+ "budget_recall": {
+ "avg_context_tokens": 93.4,
+ "precision": 0.381,
+ "recall": 0.857,
+ "token_reduction_vs_baseline": 78.0
+ }
+ },
+ "per_task": [
+ {
+ "task": "task_1",
+ "query": "Fix the bug where customers receive two confirmation emails for a single booking.",
+ "gold": [
+ "mem_0005",
+ "mem_0013"
+ ],
+ "retrieved": [
+ "mem_0005",
+ "mem_0014",
+ "mem_0008"
+ ],
+ "recall": 0.5,
+ "tokens": 94
+ },
+ {
+ "task": "task_2",
+ "query": "Add rate limiting to the chat endpoint so the Gemini API key cannot be abused.",
+ "gold": [
+ "mem_0004",
+ "mem_0011"
+ ],
+ "retrieved": [
+ "mem_0011",
+ "mem_0004",
+ "mem_0008"
+ ],
+ "recall": 1.0,
+ "tokens": 96
+ },
+ {
+ "task": "task_3",
+ "query": "A booking shows the wrong time to the customer. Investigate how timezones are handled for bookings.",
+ "gold": [
+ "mem_0003"
+ ],
+ "retrieved": [
+ "mem_0003",
+ "mem_0005",
+ "mem_0014"
+ ],
+ "recall": 1.0,
+ "tokens": 92
+ },
+ {
+ "task": "task_4",
+ "query": "Update the list of services and prices the chatbot tells customers about.",
+ "gold": [
+ "mem_0004",
+ "mem_0012"
+ ],
+ "retrieved": [
+ "mem_0004",
+ "mem_0011",
+ "mem_0010"
+ ],
+ "recall": 0.5,
+ "tokens": 101
+ },
+ {
+ "task": "task_5",
+ "query": "Make the gallery images publicly viewable while keeping all other uploads private.",
+ "gold": [
+ "mem_0007"
+ ],
+ "retrieved": [
+ "mem_0007",
+ "mem_0011",
+ "mem_0014"
+ ],
+ "recall": 1.0,
+ "tokens": 95
+ },
+ {
+ "task": "task_6",
+ "query": "Add a new admin-only settings page and make sure it is protected like the other admin routes.",
+ "gold": [
+ "mem_0006"
+ ],
+ "retrieved": [
+ "mem_0006",
+ "mem_0012",
+ "mem_0007"
+ ],
+ "recall": 1.0,
+ "tokens": 80
+ },
+ {
+ "task": "task_7",
+ "query": "The chat widget sometimes crashes when Gemini replies. Make the chat resilient to bad responses.",
+ "gold": [
+ "mem_0008"
+ ],
+ "retrieved": [
+ "mem_0008",
+ "mem_0011",
+ "mem_0004"
+ ],
+ "recall": 1.0,
+ "tokens": 96
+ }
+ ],
+ "phrasing": {
+ "query": {
+ "recall": 0.857,
+ "avg_query_word_overlap_with_gold": 0.403
+ },
+ "paraphrase": {
+ "recall": 0.786,
+ "avg_query_word_overlap_with_gold": 0.032
+ }
+ },
+ "scaling": [
+ {
+ "n_memories": 14,
+ "full_context_tokens": 424,
+ "budget_recall_tokens": 93.4,
+ "budget_recall": 0.857
+ },
+ {
+ "n_memories": 34,
+ "full_context_tokens": 805,
+ "budget_recall_tokens": 87.3,
+ "budget_recall": 0.857
+ },
+ {
+ "n_memories": 54,
+ "full_context_tokens": 1159,
+ "budget_recall_tokens": 86.3,
+ "budget_recall": 0.857
+ }
+ ],
+ "min_score_sweep": [
+ {
+ "min_score": 0.0,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 12,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.05,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 10,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.1,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 1,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.15,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.2,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.25,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.3,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.35,
+ "labelled_recall": 0.857,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ },
+ {
+ "min_score": 0.4,
+ "labelled_recall": 0.714,
+ "off_topic_memories_returned": 0,
+ "off_topic_max": 12
+ }
+ ]
+}
diff --git a/eval/results_sentence_transformers.md b/eval/results_sentence_transformers.md
new file mode 100644
index 0000000..602d30b
--- /dev/null
+++ b/eval/results_sentence_transformers.md
@@ -0,0 +1,57 @@
+# Evaluation results
+
+- Benchmark: **14 memories**, **7 tasks**, top-k = **3**, budget = **120 tokens**
+- Embedder: `SentenceTransformerEmbedder` · exact tokenizer: `True`
+
+| Arm | Avg context tokens | Recall | Precision | Tokens saved vs baseline |
+| --- | ---: | ---: | ---: | ---: |
+| No memory (control) | 0 | 0.00 | 0.00 | 100% |
+| Full context (baseline — load every memory) | 424 | 1.00 | 0.10 | 0% |
+| Random k (control — the saving without the retrieval) | 93 | 0.07 | 0.05 | 78% |
+| Targeted retrieval (this engine) | 93 | 0.86 | 0.38 | 78% |
+| Budget recall (≤ 120 tokens) | 93 | 0.86 | 0.38 | 78% |
+
+**Read the random arm first.** It loads the same number of memories as the engine, so it reports the same ~78% token saving — at **0.07** recall against the engine's **0.86**. The saving is arithmetic (k memories out of n); only the recall gap is evidence that retrieval does anything.
+
+Precision is reported for completeness, but the baseline's **0.10** is just `|relevant| / |store|` — an artefact of loading everything, not a meaningful comparison.
+
+## Does it survive a rephrase?
+
+| Query phrasing | Word overlap with gold memories | Recall |
+| --- | ---: | ---: |
+| Developer phrasing (as labelled) | 40% | 0.86 |
+| Outsider paraphrase (vocabulary avoided) | 3% | 0.79 |
+
+This backend embeds meaning rather than wording, so the drop is only **0.07**. It is the reason to install the `real` extra for day-to-day use: real questions rarely reuse the words a memory was written in.
+
+## Does the cost stay flat as the store grows?
+
+| Memories in store | Full context tokens | Budget recall tokens | Budget recall |
+| ---: | ---: | ---: | ---: |
+| 14 | 424 | 93 | 0.86 |
+| 34 | 805 | 87 | 0.86 |
+| 54 | 1159 | 86 | 0.86 |
+
+Distractor memories are added from the same project. The baseline grows linearly; the budgeted arm does not. Recall is measured against the same labels throughout, so any drop is real interference from the added memories.
+
+## Relevance floor (`min_score`)
+
+| min_score | Labelled recall | Off-topic memories returned |
+| ---: | ---: | ---: |
+| 0.00 | 0.86 | 12/12 |
+| 0.05 | 0.86 | 10/12 |
+| 0.10 | 0.86 | 1/12 |
+| 0.15 | 0.86 | 0/12 |
+| 0.20 | 0.86 | 0/12 |
+| 0.25 | 0.86 | 0/12 |
+| 0.30 | 0.86 | 0/12 |
+| 0.35 | 0.86 | 0/12 |
+| 0.40 | 0.71 | 0/12 |
+
+Four deliberately off-topic queries stand in for a task the store knows nothing about. Without a floor the engine returns k memories anyway, and the MCP layer strips scores, so the agent cannot tell. `SentenceTransformerEmbedder.recommended_min_score` is set from this sweep.
+
+## Honest limits
+
+- 7 tasks and 10 gold labels, all hand-written by the author. This is an engineering check, not a production-scale claim; one retrieval either way moves recall by ~0.07.
+- The labels, the queries and the retriever all come from the same person, which is exactly the setup that flatters a retriever. The paraphrase arm exists to push back on that.
+- Numbers use `sentence-transformers` (all-MiniLM-L6-v2), which needs a model download and is therefore not run in CI. Regenerate with: `python eval/run_eval.py --embedder sentence-transformers`.
diff --git a/eval/run_eval.py b/eval/run_eval.py
index 073de99..5390957 100644
--- a/eval/run_eval.py
+++ b/eval/run_eval.py
@@ -49,6 +49,8 @@
from agent_memory import HashingEmbedder, MemoryStore, count_tokens # noqa: E402
from agent_memory.tokens import using_exact_tokenizer # noqa: E402
+EMBEDDER_CHOICES = ("hashing", "sentence-transformers")
+
ARM_LABELS = {
"no_memory": "No memory (control)",
"full_context": "Full context (baseline — load every memory)",
@@ -75,18 +77,29 @@ def mean(xs: list[float]) -> float:
return sum(xs) / len(xs) if xs else 0.0
-def build_store(memories: list[dict]) -> MemoryStore:
+def make_embedder(name: str):
+ """Build an embedder by name, so the same benchmark can score both."""
+ if name == "hashing":
+ return HashingEmbedder()
+ from agent_memory import SentenceTransformerEmbedder
+
+ return SentenceTransformerEmbedder()
+
+
+def build_store(memories: list[dict], embedder=None) -> MemoryStore:
# Explicit embedder: the published numbers are the offline one's, and we do
# not want them to change silently on a machine with sentence-transformers.
- store = MemoryStore(embedder=HashingEmbedder())
+ store = MemoryStore(embedder=embedder or HashingEmbedder())
for m in memories:
store.write(m["text"], type=m["type"], id=m["id"])
return store
-def evaluate(dataset: dict, k: int = 3, budget: int = 120, seed: int = 0) -> dict:
+def evaluate(
+ dataset: dict, k: int = 3, budget: int = 120, seed: int = 0, embedder=None
+) -> dict:
memories = dataset["memories"]
- store = build_store(memories)
+ store = build_store(memories, embedder)
rng = random.Random(seed)
all_ids = [m["id"] for m in memories]
@@ -159,19 +172,19 @@ def record(name: str, ids: list[str], tokens: int) -> tuple[float, float]:
"n_tasks": len(dataset["tasks"]),
"summary": summary,
"per_task": per_task,
- "phrasing": phrasing_gap(dataset, k=k),
- "scaling": scaling(dataset, k=k, budget=budget),
- "min_score_sweep": min_score_sweep(dataset, k=k),
+ "phrasing": phrasing_gap(dataset, k=k, embedder=embedder),
+ "scaling": scaling(dataset, k=k, budget=budget, embedder=embedder),
+ "min_score_sweep": min_score_sweep(dataset, k=k, embedder=embedder),
}
-def phrasing_gap(dataset: dict, k: int = 3) -> dict:
+def phrasing_gap(dataset: dict, k: int = 3, embedder=None) -> dict:
"""Recall on the developer phrasing vs an outsider's paraphrase.
The default embedder matches shared wording. This is the number that says
how much of the headline recall comes from the benchmark's phrasing.
"""
- store = build_store(dataset["memories"])
+ store = build_store(dataset["memories"], embedder)
out = {}
for field in ("query", "paraphrase"):
recalls, overlaps = [], []
@@ -206,7 +219,7 @@ def _word_overlap(query: str, memories: list[dict], gold: set[str]) -> float:
return len(words & gold_words) / len(words)
-def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]:
+def scaling(dataset: dict, k: int = 3, budget: int = 120, embedder=None) -> list[dict]:
"""Grow the store with distractors and re-measure.
"The cost of memory stays fixed as the store grows" is a claim about a store
@@ -220,7 +233,7 @@ def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]:
{"id": f"dis_{i:04d}", "type": d["type"], "text": d["text"]}
for i, d in enumerate(distractors[:extra])
]
- store = build_store(memories)
+ store = build_store(memories, embedder)
full_tokens = sum(count_tokens(m["text"]) for m in memories)
recalls, tokens = [], []
for task in dataset["tasks"]:
@@ -240,13 +253,13 @@ def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]:
return rows
-def min_score_sweep(dataset: dict, k: int = 3) -> list[dict]:
+def min_score_sweep(dataset: dict, k: int = 3, embedder=None) -> list[dict]:
"""Calibrate the relevance floor used by the CLI and MCP server.
Labelled recall on real queries must not drop, while queries about things
the store knows nothing about should return nothing at all.
"""
- store = build_store(dataset["memories"])
+ store = build_store(dataset["memories"], embedder)
off_topic = [
"how do I bake sourdough bread at home",
"what is the capital of Peru",
@@ -254,7 +267,7 @@ def min_score_sweep(dataset: dict, k: int = 3) -> list[dict]:
"best hiking boots for winter walking",
]
rows = []
- for threshold in (0.0, 0.05, 0.10, 0.15, 0.20, 0.25):
+ for threshold in (0.0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40):
recalls = []
for task in dataset["tasks"]:
gold = set(task["relevant_ids"])
@@ -327,13 +340,23 @@ def render_markdown(results: dict) -> str:
f"| {label} | {row['avg_query_word_overlap_with_gold']:.0%} | "
f"{row['recall']:.2f} |"
)
+ gap = ph["query"]["recall"] - ph["paraphrase"]["recall"]
+ if results["embedder"] == "HashingEmbedder":
+ verdict = (
+ "The default embedder is feature hashing — lexical, not semantic. When "
+ f"the query stops sharing words with the memory, recall drops by "
+ f"**{gap:.2f}**. That gap is the honest limit of the offline default, "
+ "and the reason the optional `sentence-transformers` backend exists."
+ )
+ else:
+ verdict = (
+ f"This backend embeds meaning rather than wording, so the drop is only "
+ f"**{gap:.2f}**. It is the reason to install the `real` extra for day-to-day "
+ "use: real questions rarely reuse the words a memory was written in."
+ )
lines += [
"",
- "The default embedder is feature hashing — lexical, not semantic. When the "
- "query stops sharing words with the memory, recall drops by "
- f"**{(ph['query']['recall'] - ph['paraphrase']['recall']):.2f}**. That gap is "
- "the honest limit of the offline default, and the reason the optional "
- "`sentence-transformers` backend exists.",
+ verdict,
"",
"## Does the cost stay flat as the store grows?",
"",
@@ -365,8 +388,8 @@ def render_markdown(results: dict) -> str:
"",
"Four deliberately off-topic queries stand in for a task the store knows "
"nothing about. Without a floor the engine returns k memories anyway, and "
- "the MCP layer strips scores, so the agent cannot tell. `HashingEmbedder."
- "recommended_min_score` is set from this sweep.",
+ "the MCP layer strips scores, so the agent cannot tell. "
+ f"`{results['embedder']}.recommended_min_score` is set from this sweep.",
"",
"## Honest limits",
"",
@@ -377,8 +400,15 @@ def render_markdown(results: dict) -> str:
"- The labels, the queries and the retriever all come from the same person, "
"which is exactly the setup that flatters a retriever. The paraphrase arm "
"exists to push back on that.",
- "- Numbers use the offline hashing embedder so they are stable in CI. The "
- "`sentence-transformers` backend is not measured here.",
+ (
+ "- Numbers use the offline hashing embedder so they are stable in CI. "
+ "The `sentence-transformers` backend is measured separately in "
+ "`results_sentence_transformers.md`."
+ if results["embedder"] == "HashingEmbedder"
+ else "- Numbers use `sentence-transformers` (all-MiniLM-L6-v2), which "
+ "needs a model download and is therefore not run in CI. Regenerate "
+ "with: `python eval/run_eval.py --embedder sentence-transformers`."
+ ),
]
return "\n".join(lines)
@@ -395,15 +425,29 @@ def main() -> None:
parser.add_argument(
"--dataset", type=Path, default=Path(__file__).parent / "dataset.json"
)
+ parser.add_argument(
+ "--embedder",
+ choices=EMBEDDER_CHOICES,
+ default="hashing",
+ help="which backend to score. The committed results.md is the hashing "
+ "run, because it is the one that reproduces offline and in CI.",
+ )
parser.add_argument("--out-dir", type=Path, default=Path(__file__).parent)
args = parser.parse_args()
dataset = load_dataset(args.dataset)
- results = evaluate(dataset, k=args.k, budget=args.budget, seed=args.seed)
+ embedder = make_embedder(args.embedder)
+ results = evaluate(
+ dataset, k=args.k, budget=args.budget, seed=args.seed, embedder=embedder
+ )
+ # Keep the two runs in separate files: results.md is the reproducible one
+ # CI regenerates and diffs, so a machine with the model installed must not
+ # silently overwrite it with numbers CI can never reproduce.
+ stem = "results" if args.embedder == "hashing" else "results_sentence_transformers"
md = render_markdown(results)
- (args.out_dir / "results.md").write_text(md + "\n")
- (args.out_dir / "results.json").write_text(json.dumps(results, indent=2) + "\n")
+ (args.out_dir / f"{stem}.md").write_text(md + "\n")
+ (args.out_dir / f"{stem}.json").write_text(json.dumps(results, indent=2) + "\n")
print(md)
if not results["exact_tokenizer"]:
print(
diff --git a/src/agent_memory/__init__.py b/src/agent_memory/__init__.py
index 21358fa..3527914 100644
--- a/src/agent_memory/__init__.py
+++ b/src/agent_memory/__init__.py
@@ -8,11 +8,14 @@
default_min_score,
)
from .store import (
+ GLOBAL_STORE,
MEMORY_TYPES,
STORE_FORMAT,
MemoryEntry,
MemoryStore,
RecallHit,
+ default_store_path,
+ find_project_root,
)
from .tokens import count_tokens
@@ -24,6 +27,9 @@
"RecallHit",
"MEMORY_TYPES",
"STORE_FORMAT",
+ "GLOBAL_STORE",
+ "default_store_path",
+ "find_project_root",
"Embedder",
"HashingEmbedder",
"SentenceTransformerEmbedder",
diff --git a/src/agent_memory/cli.py b/src/agent_memory/cli.py
index 6c4a34b..78685ff 100644
--- a/src/agent_memory/cli.py
+++ b/src/agent_memory/cli.py
@@ -18,22 +18,39 @@
import argparse
import os
+import sys
from pathlib import Path
from .embeddings import default_min_score
-from .store import MEMORY_TYPES, MemoryStore
+from .store import (
+ GLOBAL_STORE,
+ MEMORY_TYPES,
+ MemoryStore,
+ default_store_path,
+ relocation_notice,
+)
-DEFAULT_STORE = Path(
- os.environ.get("AGENT_MEMORY_PATH", "~/.agent_memory/store.json")
-).expanduser()
DEFAULT_AGENT = os.environ.get("AGENT_MEMORY_AGENT", "")
# Sentinel: resolved per-embedder once the store is open (see default_min_score).
AUTO_MIN_SCORE = -1.0
+def _resolve_path(args) -> Path:
+ """Explicit --path, then --global, then the project/global default."""
+ if args.path is not None:
+ return Path(args.path).expanduser()
+ if getattr(args, "use_global", False):
+ return GLOBAL_STORE
+ path = default_store_path()
+ notice = relocation_notice(path)
+ if notice:
+ print(notice, file=sys.stderr)
+ return path
+
+
def _store(args) -> MemoryStore:
- return MemoryStore(path=args.path)
+ return MemoryStore(path=_resolve_path(args))
def _min_score(args, store: MemoryStore) -> float:
@@ -115,7 +132,9 @@ def cmd_forget(args) -> None:
def cmd_stats(args) -> None:
- s = _store(args).stats()
+ path = _resolve_path(args)
+ s = MemoryStore(path=path).stats()
+ print(f"store: {path}")
print(f"{s['count']} memories | {s['total_tokens']} tokens | {s['embedder']}")
for t, n in sorted(s["by_type"].items()):
print(f" {t}: {n}")
@@ -123,7 +142,18 @@ def cmd_stats(args) -> None:
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(prog="agent-memory")
- parser.add_argument("--path", type=Path, default=DEFAULT_STORE)
+ parser.add_argument(
+ "--path",
+ type=Path,
+ default=None,
+ help="store file (default: this project's .agent_memory/store.json)",
+ )
+ parser.add_argument(
+ "--global",
+ dest="use_global",
+ action="store_true",
+ help=f"use the cross-project store at {GLOBAL_STORE}",
+ )
parser.add_argument("--agent", default=DEFAULT_AGENT, help="who is writing")
sub = parser.add_subparsers(dest="command", required=True)
diff --git a/src/agent_memory/embeddings.py b/src/agent_memory/embeddings.py
index 89219de..29a48ac 100644
--- a/src/agent_memory/embeddings.py
+++ b/src/agent_memory/embeddings.py
@@ -101,17 +101,33 @@ def embed(self, texts: list[str]) -> np.ndarray:
class SentenceTransformerEmbedder:
"""Real semantic embeddings. Optional: needs ``sentence-transformers``."""
- # NOT calibrated in this repo — the offline benchmark runs on the hashing
- # embedder, and MiniLM cosines sit on a higher scale (unrelated pairs often
- # score 0.1-0.3). Treat this as a conservative starting point and measure on
- # your own store before relying on it.
- recommended_min_score = 0.25
+ # Calibrated on eval/dataset.json — see eval/results_sentence_transformers.md.
+ # Every off-topic query is already rejected at 0.15, and labelled recall is
+ # flat until it falls at 0.40, so anything in 0.15-0.35 scores identically on
+ # the benchmark. The tie is broken by headroom rather than by the sweep:
+ # genuine paraphrases can land low ("which AI model answers customer
+ # questions" scores 0.22 against a memory that answers it), so the floor sits
+ # just under that rather than in the middle of the safe band. MiniLM cosines
+ # run higher than the hashing embedder's, which is why this differs from it.
+ recommended_min_score = 0.20
def __init__(self, model_name: str = "all-MiniLM-L6-v2") -> None:
from sentence_transformers import SentenceTransformer # lazy import
self._model = SentenceTransformer(model_name)
- self.dim = int(self._model.get_sentence_embedding_dimension())
+ # Renamed in sentence-transformers 5.x; support both so an upgrade of an
+ # optional dependency cannot break the backend.
+ get_dim = getattr(
+ self._model,
+ "get_embedding_dimension",
+ getattr(self._model, "get_sentence_embedding_dimension", None),
+ )
+ if get_dim is None: # pragma: no cover - depends on the installed version
+ raise RuntimeError(
+ "could not determine the embedding dimension from "
+ f"sentence-transformers model {model_name!r}"
+ )
+ self.dim = int(get_dim())
self.model_name = model_name
def embed(self, texts: list[str]) -> np.ndarray:
diff --git a/src/agent_memory/mcp_server.py b/src/agent_memory/mcp_server.py
index 32e1b13..890756d 100644
--- a/src/agent_memory/mcp_server.py
+++ b/src/agent_memory/mcp_server.py
@@ -21,15 +21,12 @@
from __future__ import annotations
import os
+import sys
from pathlib import Path
from typing import Optional
from .embeddings import default_min_score
-from .store import MEMORY_TYPES, MemoryStore
-
-DEFAULT_STORE = Path(
- os.environ.get("AGENT_MEMORY_PATH", "~/.agent_memory/store.json")
-).expanduser()
+from .store import MEMORY_TYPES, MemoryStore, default_store_path, relocation_notice
# Who is talking to the store — "claude-code", "codex", "cursor", ...
DEFAULT_AGENT = os.environ.get("AGENT_MEMORY_AGENT", "")
@@ -69,14 +66,26 @@ def _render(hits) -> str:
def build_server(
- store_path: Path = DEFAULT_STORE,
+ store_path: Optional[Path] = None,
agent: str = DEFAULT_AGENT,
min_score: Optional[float] = None,
):
"""Construct the MCP server. Imports `mcp` lazily so importing this module
- never hard-fails when the optional dependency is absent."""
+ never hard-fails when the optional dependency is absent.
+
+ With no `store_path`, the store is resolved per project: the agent launches
+ this server from the working directory, so `.agent_memory/store.json` in the
+ enclosing repository is the store its memories belong to.
+ """
server_class = _load_server_class()
+ if store_path is None:
+ store_path = default_store_path()
+ notice = relocation_notice(store_path)
+ if notice:
+ # stdout is the JSON-RPC channel; anything printed there corrupts it.
+ print(notice, file=sys.stderr)
+
store = MemoryStore(path=store_path)
if min_score is None:
min_score = default_min_score(store.embedder)
diff --git a/src/agent_memory/store.py b/src/agent_memory/store.py
index 1896b6f..df3a7f8 100644
--- a/src/agent_memory/store.py
+++ b/src/agent_memory/store.py
@@ -47,11 +47,68 @@
_ID_RE = re.compile(r"^mem_(\d+)$")
+# Where memories live when nothing is configured. One store per project, not one
+# store for everything you have ever worked on: recall matches on similarity
+# alone, so a single global file lets one project's deploy notes surface while
+# you are working on another.
+PROJECT_STORE_DIR = ".agent_memory"
+STORE_FILENAME = "store.json"
+GLOBAL_STORE = Path.home() / PROJECT_STORE_DIR / STORE_FILENAME
+
def _now_iso() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
+def find_project_root(start: Optional[str | Path] = None) -> Optional[Path]:
+ """Nearest ancestor directory containing `.git`, or None outside a repo."""
+ current = Path(start).expanduser().resolve() if start else Path.cwd().resolve()
+ for candidate in (current, *current.parents):
+ # `.git` is a directory in a normal clone and a file in a worktree or
+ # submodule, so test for existence rather than for a directory.
+ if (candidate / ".git").exists():
+ return candidate
+ return None
+
+
+def default_store_path(start: Optional[str | Path] = None) -> Path:
+ """Resolve which store to use.
+
+ In precedence order: an explicit ``AGENT_MEMORY_PATH``, then the current
+ project's ``.agent_memory/store.json``, then a global store for work that
+ isn't in a repository.
+ """
+ configured = os.environ.get("AGENT_MEMORY_PATH")
+ if configured:
+ return Path(configured).expanduser()
+ root = find_project_root(start)
+ if root is not None:
+ return root / PROJECT_STORE_DIR / STORE_FILENAME
+ return GLOBAL_STORE
+
+
+def relocation_notice(path: Path) -> Optional[str]:
+ """Warn once when a fresh project store is used but a global one has data.
+
+ Stores used to default to a single global file. Without this, upgrading
+ looks like every memory was deleted. Callers must print it to stderr — on
+ the MCP server stdout carries the protocol.
+ """
+ if path == GLOBAL_STORE or path.exists() or not GLOBAL_STORE.exists():
+ return None
+ try:
+ count = len(json.loads(GLOBAL_STORE.read_text()).get("entries", []))
+ except (OSError, ValueError):
+ return None
+ if not count:
+ return None
+ return (
+ f"[agent-memory] Starting an empty store for this project at {path}. "
+ f"Your global store still holds {count} memories — use them here with "
+ f"AGENT_MEMORY_PATH={GLOBAL_STORE}"
+ )
+
+
@dataclass
class MemoryEntry:
id: str
diff --git a/tests/test_sentence_transformers.py b/tests/test_sentence_transformers.py
new file mode 100644
index 0000000..bd82c80
--- /dev/null
+++ b/tests/test_sentence_transformers.py
@@ -0,0 +1,83 @@
+"""The optional sentence-transformers backend.
+
+Skipped unless the `real` extra is installed, so CI stays offline and fast.
+Run locally with: pip install -e ".[real]" && pytest tests/test_sentence_transformers.py
+"""
+
+import pytest
+
+pytest.importorskip(
+ "sentence_transformers", reason='needs the optional "real" extra'
+)
+
+from agent_memory import MemoryStore, SentenceTransformerEmbedder # noqa: E402
+
+
+@pytest.fixture(scope="module")
+def embedder():
+ return SentenceTransformerEmbedder()
+
+
+@pytest.fixture(scope="module")
+def store(embedder):
+ s = MemoryStore(embedder=embedder)
+ s.write("Bookings are stored in UTC and converted in the UI layer.", type="decision")
+ s.write("The customer chatbot uses Google Gemini in server/gemini-chat.ts.", type="decision")
+ s.write("Admin routes are guarded by requireAdmin in server/auth.ts.", type="decision")
+ s.write("Uploaded images are private unless they sit under the gallery/ prefix.", type="project")
+ return s
+
+
+def test_dimension_is_discovered_across_library_versions(embedder):
+ """`get_sentence_embedding_dimension` was renamed in 5.x; both must work."""
+ assert embedder.dim == 384
+ assert embedder.embed(["hello"]).shape == (1, 384)
+
+
+def test_vectors_are_unit_norm(embedder):
+ vecs = embedder.embed(["one sentence", "another entirely different sentence"])
+ norms = (vecs**2).sum(axis=1) ** 0.5
+ assert all(abs(n - 1.0) < 1e-4 for n in norms)
+
+
+@pytest.mark.parametrize(
+ "paraphrase, expected",
+ [
+ ("a client saw the wrong hour for their appointment", "UTC"),
+ ("make sure only administrators can reach the new settings page", "requireAdmin"),
+ ("how do we restrict a route to administrators", "requireAdmin"),
+ ("which AI model answers customer questions", "Gemini"),
+ ],
+)
+def test_recall_survives_a_paraphrase(store, paraphrase, expected):
+ """The whole reason this backend exists: no shared vocabulary."""
+ hits = store.recall(paraphrase, k=1)
+ assert expected in hits[0].entry.text
+
+
+def test_a_genuinely_ambiguous_query_is_not_answered_confidently(store):
+ """Two memories are about access control, and this query fits both equally.
+
+ Both score ~0.16 — below the configured floor — so the honest outcome is
+ nothing rather than a coin-flip presented to the agent as fact.
+ """
+ floor = SentenceTransformerEmbedder.recommended_min_score
+ assert store.recall("only staff should be able to open this screen",
+ k=2, min_score=floor) == []
+
+
+def test_configured_floor_keeps_real_matches_and_drops_junk(store):
+ floor = SentenceTransformerEmbedder.recommended_min_score
+ assert store.recall("how are timezones handled", k=3, min_score=floor)
+ assert store.recall("best sourdough bread recipe for beginners", k=3, min_score=floor) == []
+
+
+def test_a_store_written_by_this_backend_reloads(tmp_path, embedder):
+ path = tmp_path / "st.json"
+ first = MemoryStore(path=path, embedder=embedder)
+ first.write("Calendar sync must never block a booking from saving.", type="decision")
+
+ reopened = MemoryStore(path=path, embedder=embedder)
+ assert reopened.stats()["count"] == 1
+ hits = reopened.recall("what happens if the calendar integration fails", k=1)
+ assert "Calendar sync" in hits[0].entry.text
diff --git a/tests/test_store_location.py b/tests/test_store_location.py
new file mode 100644
index 0000000..1b2ae6e
--- /dev/null
+++ b/tests/test_store_location.py
@@ -0,0 +1,115 @@
+"""Which store a command uses.
+
+Memories used to land in one global file shared by every project, so recall
+could surface another codebase's answers. The store now follows the project.
+"""
+
+import json
+
+import pytest
+
+from agent_memory import GLOBAL_STORE, default_store_path, find_project_root
+from agent_memory.cli import build_parser, main
+from agent_memory.store import relocation_notice
+
+
+@pytest.fixture
+def repo(tmp_path, monkeypatch):
+ """A directory that looks like a git checkout, with a nested subdirectory."""
+ monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False)
+ root = tmp_path / "my-project"
+ (root / ".git").mkdir(parents=True)
+ (root / "src" / "deep").mkdir(parents=True)
+ return root
+
+
+def test_store_follows_the_project(repo, monkeypatch):
+ monkeypatch.chdir(repo)
+ assert default_store_path() == repo / ".agent_memory" / "store.json"
+
+
+def test_store_is_found_from_a_nested_directory(repo, monkeypatch):
+ monkeypatch.chdir(repo / "src" / "deep")
+ assert default_store_path() == repo / ".agent_memory" / "store.json"
+
+
+def test_two_projects_get_two_stores(tmp_path, monkeypatch):
+ monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False)
+ first, second = tmp_path / "alpha", tmp_path / "beta"
+ for project in (first, second):
+ (project / ".git").mkdir(parents=True)
+ monkeypatch.chdir(first)
+ a = default_store_path()
+ monkeypatch.chdir(second)
+ b = default_store_path()
+ assert a != b
+
+
+def test_git_worktrees_and_submodules_are_recognised(tmp_path, monkeypatch):
+ """`.git` is a file, not a directory, in a worktree or submodule."""
+ monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False)
+ root = tmp_path / "worktree"
+ root.mkdir()
+ (root / ".git").write_text("gitdir: /elsewhere/.git/worktrees/wt\n")
+ monkeypatch.chdir(root)
+ assert find_project_root() == root
+ assert default_store_path() == root / ".agent_memory" / "store.json"
+
+
+def test_outside_a_repository_falls_back_to_the_global_store(tmp_path, monkeypatch):
+ monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False)
+ loose = tmp_path / "not-a-repo"
+ loose.mkdir()
+ monkeypatch.chdir(loose)
+ assert default_store_path() == GLOBAL_STORE
+
+
+def test_explicit_env_var_still_wins(repo, monkeypatch, tmp_path):
+ monkeypatch.chdir(repo)
+ monkeypatch.setenv("AGENT_MEMORY_PATH", str(tmp_path / "chosen.json"))
+ assert default_store_path() == tmp_path / "chosen.json"
+
+
+def test_memories_written_in_a_project_stay_in_that_project(repo, monkeypatch, capsys):
+ monkeypatch.setenv("AGENT_MEMORY_EMBEDDER", "hashing")
+ monkeypatch.chdir(repo)
+ main(["write", "Bookings are stored in UTC.", "--type", "decision"])
+
+ store_file = repo / ".agent_memory" / "store.json"
+ assert store_file.exists()
+ assert "UTC" in store_file.read_text()
+
+ other = repo.parent / "other-project"
+ (other / ".git").mkdir(parents=True)
+ monkeypatch.chdir(other)
+ capsys.readouterr()
+ main(["recall", "how are timezones handled"])
+ assert "UTC" not in capsys.readouterr().out, "another project's memory leaked in"
+
+
+def test_global_flag_opts_back_into_the_shared_store(repo, monkeypatch):
+ monkeypatch.chdir(repo)
+ args = build_parser().parse_args(["--global", "stats"])
+ assert args.use_global is True
+
+
+def test_relocation_notice_points_at_existing_global_memories(tmp_path, monkeypatch):
+ """Upgrading must not look like every memory was deleted."""
+ fake_global = tmp_path / "global.json"
+ fake_global.write_text(json.dumps({"entries": [{"id": "mem_0001", "text": "x"}]}))
+ monkeypatch.setattr("agent_memory.store.GLOBAL_STORE", fake_global)
+
+ fresh = tmp_path / "proj" / ".agent_memory" / "store.json"
+ notice = relocation_notice(fresh)
+ assert notice and "1 memories" in notice and str(fake_global) in notice
+
+ # Silent once the project store exists, and silent for the global store itself.
+ fresh.parent.mkdir(parents=True)
+ fresh.write_text(json.dumps({"entries": []}))
+ assert relocation_notice(fresh) is None
+ assert relocation_notice(fake_global) is None
+
+
+def test_notice_is_silent_when_there_is_nothing_to_migrate(tmp_path, monkeypatch):
+ monkeypatch.setattr("agent_memory.store.GLOBAL_STORE", tmp_path / "absent.json")
+ assert relocation_notice(tmp_path / "proj" / "store.json") is None