From 918992997d7732d075ae5b5905742155dd442a98 Mon Sep 17 00:00:00 2001 From: Nina Doinjashvili Date: Wed, 12 Aug 2026 00:59:11 +0300 Subject: [PATCH 1/2] Scope stores to the project, and measure the semantic backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes aimed at making this usable day to day rather than just correct on a benchmark. Per-project stores - The default was a single global ~/.agent_memory/store.json shared by every project, and all three README configs pointed at it. Recall matches on similarity alone, so one project's "how do we deploy" answer could surface while working on another. - The store now resolves as: AGENT_MEMORY_PATH, then .agent_memory/ store.json in the enclosing git repository, then the global file when outside a repo. `.git` is tested for existence, so worktrees and submodules (where it is a file) work too. - `--global` opts back into the shared store, `agent-memory stats` prints which store is in use, and a one-time stderr notice points at existing global memories so upgrading does not look like data loss. It goes to stderr because on the MCP server stdout carries the protocol. Measured the sentence-transformers backend - It shipped unmeasured with a guessed relevance floor. Now scored on the same benchmark and published in results_sentence_transformers.md: hashing MiniLM developer phrasing 0.93 0.86 paraphrase 0.43 0.79 off-topic rejected 6/12 12/12 Not a clean sweep, which is the interesting part: exact word matching genuinely wins when the words match. MiniLM wins where real use lives. The README now recommends the `real` extra for daily use and keeps hashing as the offline/CI default. - Calibrated its floor from the sweep plus observed behaviour: off-topic queries are fully rejected by 0.15 and recall is flat to 0.35, so the sweep alone cannot choose. Set to 0.20, just under the 0.22 scored by a real paraphrase against the memory that answers it. The previous 0.25 guess would have rejected it. - Fixed a latent break: sentence-transformers 5.x renamed get_sentence_embedding_dimension. Both names are now supported — the same failure mode as the mcp 1.x/2.x rename. - eval/run_eval.py takes --embedder and writes the two runs to separate files, so a machine with the model installed cannot overwrite the results CI regenerates and diffs. Tests 65 -> 74, plus 9 sentence-transformers tests that skip without the extra. Verified over a real JSON-RPC stdio session: 8 tools, paraphrased query resolved, off-topic rejected, no protocol corruption. Co-Authored-By: Claude Opus 5 --- README.md | 78 +++++--- eval/results.json | 18 ++ eval/results.md | 5 +- eval/results_sentence_transformers.json | 230 ++++++++++++++++++++++++ eval/results_sentence_transformers.md | 57 ++++++ eval/run_eval.py | 96 +++++++--- src/agent_memory/__init__.py | 6 + src/agent_memory/cli.py | 44 ++++- src/agent_memory/embeddings.py | 28 ++- src/agent_memory/mcp_server.py | 23 ++- src/agent_memory/store.py | 57 ++++++ tests/test_sentence_transformers.py | 83 +++++++++ tests/test_store_location.py | 115 ++++++++++++ 13 files changed, 771 insertions(+), 69 deletions(-) create mode 100644 eval/results_sentence_transformers.json create mode 100644 eval/results_sentence_transformers.md create mode 100644 tests/test_sentence_transformers.py create mode 100644 tests/test_store_location.py diff --git a/README.md b/README.md index 3c5e91a..ff9e9df 100644 --- a/README.md +++ b/README.md @@ -24,17 +24,26 @@ On a hand-labeled benchmark of an agent working across many sessions on one code Reproduce with `python eval/run_eval.py`; the full report is in [eval/results.md](eval/results.md). -### What these numbers don't show +### Which embedder should you use? -- **The default embedder is lexical, not semantic.** It is feature hashing over word and character n-grams, so it matches shared wording. Re-run the same tasks with paraphrased queries that avoid the memories' vocabulary and recall drops from **0.93 to 0.43**. That gap is published in the results, not tuned away, and it is the reason the optional `sentence-transformers` backend exists. +The offline default is **lexical** — feature hashing over word and character n-grams — so it matches shared wording. That flatters it on a benchmark whose queries reuse the memories' vocabulary. Every task therefore carries a second phrasing that deliberately avoids that vocabulary, and both backends are measured on both: - | Query phrasing | Word overlap with gold | Recall | - | --- | ---: | ---: | - | Developer phrasing (as labelled) | 40% | 0.93 | - | Outsider paraphrase | 3% | 0.43 | +| | Hashing (offline default) | MiniLM (`real` extra) | +| --- | ---: | ---: | +| Developer phrasing — 40% word overlap | **0.93** | 0.86 | +| Outsider paraphrase — 3% word overlap | 0.43 | **0.79** | +| Average of the two | 0.68 | **0.82** | +| Off-topic queries correctly rejected | 6/12 | **12/12** | + +It isn't a clean sweep, and that's the interesting part. **Exact word matching genuinely wins when the words match** — if you write memories and query them in the same vocabulary, hashing is better *and* needs no model. But real use is mostly the second row: you write a memory in March and ask about it in July, in different words. + +**Recommendation: install the `real` extra for day-to-day use** (`pip install -e ".[real]"`), and keep the hashing default for CI, air-gapped machines, and anywhere a 90 MB model download isn't welcome. Full reports: [results.md](eval/results.md) · [results_sentence_transformers.md](eval/results_sentence_transformers.md). + +### What these numbers still don't show - **The precision column is nearly meaningless for the baseline.** Full context scores 0.10 because that is `|relevant| / |store|` — an artefact of loading everything. - **7 tasks, 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim. +- **MiniLM numbers are not reproduced in CI**, because they need a model download. CI regenerates and diffs the hashing results only. ### Does the cost stay flat as the store grows? @@ -76,16 +85,15 @@ Tool outputs are deliberately compact — no scores, no timestamps — because e ## Hook it up to your agents ```bash -pip install -e ".[mcp]" +pip install -e ".[mcp,real]" # drop `real` to stay fully offline ``` -Works with both `mcp` 1.x and 2.x. Point every agent at the **same store path**, and give each its own name: +Works with both `mcp` 1.x and 2.x. Give each agent its own name — the store itself needs no configuration: **Claude Code** ```bash -claude mcp add agent-memory -e AGENT_MEMORY_PATH=~/.agent_memory/store.json \ - -e AGENT_MEMORY_AGENT=claude-code -- agent-memory-mcp +claude mcp add agent-memory -e AGENT_MEMORY_AGENT=claude-code -- agent-memory-mcp ``` **Codex CLI** (`~/.codex/config.toml`) @@ -93,7 +101,7 @@ claude mcp add agent-memory -e AGENT_MEMORY_PATH=~/.agent_memory/store.json \ ```toml [mcp_servers.agent-memory] command = "agent-memory-mcp" -env = { AGENT_MEMORY_PATH = "~/.agent_memory/store.json", AGENT_MEMORY_AGENT = "codex" } +env = { AGENT_MEMORY_AGENT = "codex" } ``` **Cursor** (`.cursor/mcp.json`) @@ -103,16 +111,30 @@ env = { AGENT_MEMORY_PATH = "~/.agent_memory/store.json", AGENT_MEMORY_AGENT = " "mcpServers": { "agent-memory": { "command": "agent-memory-mcp", - "env": { - "AGENT_MEMORY_PATH": "~/.agent_memory/store.json", - "AGENT_MEMORY_AGENT": "cursor" - } + "env": { "AGENT_MEMORY_AGENT": "cursor" } } } } ``` -Each memory now carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`. Agents may run at the same time: writes take a lock, merge, and land atomically, and an already-running server picks up another agent's writes on its next read. +Each memory carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`. Agents may run at the same time: writes take a lock, merge, and land atomically, and an already-running server picks up another agent's writes on its next read. + +### One store per project + +Memories are scoped to the repository you're working in. The store resolves in this order: + +1. `AGENT_MEMORY_PATH`, if you set it +2. `.agent_memory/store.json` in the current git repository — **the normal case** +3. `~/.agent_memory/store.json`, when you're not inside a repository + +This matters more than it sounds. Recall matches on similarity alone, so a single shared store lets one project's answer to "how do we deploy?" surface while you're working on a different project. Per-project stores make that impossible. + +```bash +agent-memory stats # prints which store is in use +agent-memory --global stats # the cross-project store, when you want it +``` + +Add `.agent_memory/` to your `.gitignore` unless you intend to commit the memories — sharing them with a team is a legitimate choice, but it should be a deliberate one. ## Architecture @@ -217,8 +239,16 @@ Without a floor, a query about something the store knows nothing about still ret | 0.10 | 0.93 | 10/12 | | **0.15** | **0.93** | **6/12** | | 0.20 | 0.93 | 2/12 | +| 0.25 | 0.93 | 2/12 | + +The floor lives on the embedder, because the two score on different scales — a single constant would be wrong for one of them: -The floor lives on the embedder (`HashingEmbedder.recommended_min_score = 0.15`), overridable with `AGENT_MEMORY_MIN_SCORE` or the `--min-score` flag. A higher floor cuts more noise on this benchmark, but the lowest-scoring genuinely-relevant memory in it sits at 0.13, so the default stays close to that observed edge. The `sentence-transformers` value is **not** calibrated here — measure it on your own store. +| Backend | Floor | Chosen because | +| --- | ---: | --- | +| `HashingEmbedder` | 0.15 | No labelled recall lost, junk halved. The lowest-scoring genuinely relevant memory sits at 0.13, so the floor stays close to that observed edge. | +| `SentenceTransformerEmbedder` | 0.20 | Off-topic queries are already fully rejected at 0.15 and recall is flat to 0.35, so the sweep alone can't choose. Set just under 0.22 — the score of a real paraphrase ("which AI model answers customer questions") against the memory that answers it. | + +Override with `AGENT_MEMORY_MIN_SCORE` or `--min-score`. Note the honest consequence of a floor: when a query is genuinely ambiguous, recall returns *nothing* rather than a coin flip the agent would read as fact. ## Quickstart (60 seconds) @@ -238,9 +268,9 @@ agent-memory forget mem_0007 In Python: ```python -from agent_memory import MemoryStore +from agent_memory import MemoryStore, default_store_path -store = MemoryStore(path="~/.agent_memory/store.json") +store = MemoryStore(path=default_store_path()) # this project's store store.write("The chatbot uses Google Gemini in server/gemini-chat.ts.", type="decision", agent="claude-code") @@ -255,6 +285,8 @@ for hit in hits: pip install -e ".[real]" # sentence-transformers + tiktoken ``` +Recommended for day-to-day use — it roughly doubles recall on paraphrased questions (0.43 → 0.79) and rejects every off-topic query in the benchmark. Costs a ~90 MB model download on first run. See [the comparison](#which-embedder-should-you-use). + `default_embedder()` prefers the real model when installed, and warns on stderr if it falls back rather than switching silently. Force the offline embedder with `AGENT_MEMORY_EMBEDDER=hashing` (CI does this for stable numbers), or demand the real one with `AGENT_MEMORY_EMBEDDER=sentence-transformers` to turn a missing dependency into an error instead of a downgrade. Switching backends on an existing store is safe: vectors from a different embedder aren't comparable, so the store detects the mismatch on load and re-embeds from the stored text instead of returning meaningless scores. @@ -271,7 +303,9 @@ One memory per `##` section, with long sections split so that every ingested mem - **Memories are injected into agent context verbatim.** Anything an agent writes to the store — including text it read from a webpage, an issue tracker or a dependency — will be replayed into a *different* agent's context later, where it reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally. - **The store is a local file.** No auth, no encryption, no server. It belongs next to your code, not on a shared host. -- **Retrieval is lexical by default.** See the paraphrase gap above. +- **Retrieval is lexical unless you install the `real` extra.** The offline default matches wording, not meaning — see [the comparison](#which-embedder-should-you-use). +- **Nothing writes memories for you.** The agent has to choose to call `memory_write` and `memory_handoff`. If it doesn't, the store stays empty and the next session boots into nothing. Prompt instructions or a session hook make this reliable; that isn't built in yet. +- **Memories never expire.** `state` and `worklog` entries go stale but keep being recalled with the same authority as a fresh decision. Correct them with `memory_update`/`memory_forget` until decay lands. - **The benchmark is small and self-authored.** It is a regression check for the engine, not evidence about your codebase. ## Develop @@ -285,9 +319,9 @@ CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the ## Roadmap -- LLM-based compaction: summarize/dedup `state` and `worklog` memories, extract durable facts from a raw session transcript. +- Session hooks so writing a handoff doesn't depend on the agent remembering to. - Recency- and type-aware ranking (decay old `state`, never drop `decision`). -- `sentence-transformers` numbers published alongside the hashing baseline in CI, including a calibrated relevance floor. +- LLM-based compaction: summarize/dedup `state` and `worklog` memories, extract durable facts from a raw session transcript. - Optional FAISS backend for large stores. ## License diff --git a/eval/results.json b/eval/results.json index 564d4a9..1a917eb 100644 --- a/eval/results.json +++ b/eval/results.json @@ -207,6 +207,24 @@ "labelled_recall": 0.929, "off_topic_memories_returned": 2, "off_topic_max": 12 + }, + { + "min_score": 0.3, + "labelled_recall": 0.714, + "off_topic_memories_returned": 1, + "off_topic_max": 12 + }, + { + "min_score": 0.35, + "labelled_recall": 0.571, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.4, + "labelled_recall": 0.5, + "off_topic_memories_returned": 0, + "off_topic_max": 12 } ] } diff --git a/eval/results.md b/eval/results.md index bb6c09c..984e73c 100644 --- a/eval/results.md +++ b/eval/results.md @@ -44,6 +44,9 @@ Distractor memories are added from the same project. The baseline grows linearly | 0.15 | 0.93 | 6/12 | | 0.20 | 0.93 | 2/12 | | 0.25 | 0.93 | 2/12 | +| 0.30 | 0.71 | 1/12 | +| 0.35 | 0.57 | 0/12 | +| 0.40 | 0.50 | 0/12 | Four deliberately off-topic queries stand in for a task the store knows nothing about. Without a floor the engine returns k memories anyway, and the MCP layer strips scores, so the agent cannot tell. `HashingEmbedder.recommended_min_score` is set from this sweep. @@ -51,4 +54,4 @@ Four deliberately off-topic queries stand in for a task the store knows nothing - 7 tasks and 10 gold labels, all hand-written by the author. This is an engineering check, not a production-scale claim; one retrieval either way moves recall by ~0.07. - The labels, the queries and the retriever all come from the same person, which is exactly the setup that flatters a retriever. The paraphrase arm exists to push back on that. -- Numbers use the offline hashing embedder so they are stable in CI. The `sentence-transformers` backend is not measured here. +- Numbers use the offline hashing embedder so they are stable in CI. The `sentence-transformers` backend is measured separately in `results_sentence_transformers.md`. diff --git a/eval/results_sentence_transformers.json b/eval/results_sentence_transformers.json new file mode 100644 index 0000000..cd9289e --- /dev/null +++ b/eval/results_sentence_transformers.json @@ -0,0 +1,230 @@ +{ + "k": 3, + "budget": 120, + "seed": 0, + "exact_tokenizer": true, + "embedder": "SentenceTransformerEmbedder", + "n_memories": 14, + "n_tasks": 7, + "summary": { + "no_memory": { + "avg_context_tokens": 0.0, + "precision": 0.0, + "recall": 0.0, + "token_reduction_vs_baseline": 100.0 + }, + "full_context": { + "avg_context_tokens": 424.0, + "precision": 0.102, + "recall": 1.0, + "token_reduction_vs_baseline": 0.0 + }, + "random_k": { + "avg_context_tokens": 93.4, + "precision": 0.048, + "recall": 0.071, + "token_reduction_vs_baseline": 78.0 + }, + "retrieval": { + "avg_context_tokens": 93.4, + "precision": 0.381, + "recall": 0.857, + "token_reduction_vs_baseline": 78.0 + }, + "budget_recall": { + "avg_context_tokens": 93.4, + "precision": 0.381, + "recall": 0.857, + "token_reduction_vs_baseline": 78.0 + } + }, + "per_task": [ + { + "task": "task_1", + "query": "Fix the bug where customers receive two confirmation emails for a single booking.", + "gold": [ + "mem_0005", + "mem_0013" + ], + "retrieved": [ + "mem_0005", + "mem_0014", + "mem_0008" + ], + "recall": 0.5, + "tokens": 94 + }, + { + "task": "task_2", + "query": "Add rate limiting to the chat endpoint so the Gemini API key cannot be abused.", + "gold": [ + "mem_0004", + "mem_0011" + ], + "retrieved": [ + "mem_0011", + "mem_0004", + "mem_0008" + ], + "recall": 1.0, + "tokens": 96 + }, + { + "task": "task_3", + "query": "A booking shows the wrong time to the customer. Investigate how timezones are handled for bookings.", + "gold": [ + "mem_0003" + ], + "retrieved": [ + "mem_0003", + "mem_0005", + "mem_0014" + ], + "recall": 1.0, + "tokens": 92 + }, + { + "task": "task_4", + "query": "Update the list of services and prices the chatbot tells customers about.", + "gold": [ + "mem_0004", + "mem_0012" + ], + "retrieved": [ + "mem_0004", + "mem_0011", + "mem_0010" + ], + "recall": 0.5, + "tokens": 101 + }, + { + "task": "task_5", + "query": "Make the gallery images publicly viewable while keeping all other uploads private.", + "gold": [ + "mem_0007" + ], + "retrieved": [ + "mem_0007", + "mem_0011", + "mem_0014" + ], + "recall": 1.0, + "tokens": 95 + }, + { + "task": "task_6", + "query": "Add a new admin-only settings page and make sure it is protected like the other admin routes.", + "gold": [ + "mem_0006" + ], + "retrieved": [ + "mem_0006", + "mem_0012", + "mem_0007" + ], + "recall": 1.0, + "tokens": 80 + }, + { + "task": "task_7", + "query": "The chat widget sometimes crashes when Gemini replies. Make the chat resilient to bad responses.", + "gold": [ + "mem_0008" + ], + "retrieved": [ + "mem_0008", + "mem_0011", + "mem_0004" + ], + "recall": 1.0, + "tokens": 96 + } + ], + "phrasing": { + "query": { + "recall": 0.857, + "avg_query_word_overlap_with_gold": 0.403 + }, + "paraphrase": { + "recall": 0.786, + "avg_query_word_overlap_with_gold": 0.032 + } + }, + "scaling": [ + { + "n_memories": 14, + "full_context_tokens": 424, + "budget_recall_tokens": 93.4, + "budget_recall": 0.857 + }, + { + "n_memories": 34, + "full_context_tokens": 805, + "budget_recall_tokens": 87.3, + "budget_recall": 0.857 + }, + { + "n_memories": 54, + "full_context_tokens": 1159, + "budget_recall_tokens": 86.3, + "budget_recall": 0.857 + } + ], + "min_score_sweep": [ + { + "min_score": 0.0, + "labelled_recall": 0.857, + "off_topic_memories_returned": 12, + "off_topic_max": 12 + }, + { + "min_score": 0.05, + "labelled_recall": 0.857, + "off_topic_memories_returned": 10, + "off_topic_max": 12 + }, + { + "min_score": 0.1, + "labelled_recall": 0.857, + "off_topic_memories_returned": 1, + "off_topic_max": 12 + }, + { + "min_score": 0.15, + "labelled_recall": 0.857, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.2, + "labelled_recall": 0.857, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.25, + "labelled_recall": 0.857, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.3, + "labelled_recall": 0.857, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.35, + "labelled_recall": 0.857, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + }, + { + "min_score": 0.4, + "labelled_recall": 0.714, + "off_topic_memories_returned": 0, + "off_topic_max": 12 + } + ] +} diff --git a/eval/results_sentence_transformers.md b/eval/results_sentence_transformers.md new file mode 100644 index 0000000..602d30b --- /dev/null +++ b/eval/results_sentence_transformers.md @@ -0,0 +1,57 @@ +# Evaluation results + +- Benchmark: **14 memories**, **7 tasks**, top-k = **3**, budget = **120 tokens** +- Embedder: `SentenceTransformerEmbedder` · exact tokenizer: `True` + +| Arm | Avg context tokens | Recall | Precision | Tokens saved vs baseline | +| --- | ---: | ---: | ---: | ---: | +| No memory (control) | 0 | 0.00 | 0.00 | 100% | +| Full context (baseline — load every memory) | 424 | 1.00 | 0.10 | 0% | +| Random k (control — the saving without the retrieval) | 93 | 0.07 | 0.05 | 78% | +| Targeted retrieval (this engine) | 93 | 0.86 | 0.38 | 78% | +| Budget recall (≤ 120 tokens) | 93 | 0.86 | 0.38 | 78% | + +**Read the random arm first.** It loads the same number of memories as the engine, so it reports the same ~78% token saving — at **0.07** recall against the engine's **0.86**. The saving is arithmetic (k memories out of n); only the recall gap is evidence that retrieval does anything. + +Precision is reported for completeness, but the baseline's **0.10** is just `|relevant| / |store|` — an artefact of loading everything, not a meaningful comparison. + +## Does it survive a rephrase? + +| Query phrasing | Word overlap with gold memories | Recall | +| --- | ---: | ---: | +| Developer phrasing (as labelled) | 40% | 0.86 | +| Outsider paraphrase (vocabulary avoided) | 3% | 0.79 | + +This backend embeds meaning rather than wording, so the drop is only **0.07**. It is the reason to install the `real` extra for day-to-day use: real questions rarely reuse the words a memory was written in. + +## Does the cost stay flat as the store grows? + +| Memories in store | Full context tokens | Budget recall tokens | Budget recall | +| ---: | ---: | ---: | ---: | +| 14 | 424 | 93 | 0.86 | +| 34 | 805 | 87 | 0.86 | +| 54 | 1159 | 86 | 0.86 | + +Distractor memories are added from the same project. The baseline grows linearly; the budgeted arm does not. Recall is measured against the same labels throughout, so any drop is real interference from the added memories. + +## Relevance floor (`min_score`) + +| min_score | Labelled recall | Off-topic memories returned | +| ---: | ---: | ---: | +| 0.00 | 0.86 | 12/12 | +| 0.05 | 0.86 | 10/12 | +| 0.10 | 0.86 | 1/12 | +| 0.15 | 0.86 | 0/12 | +| 0.20 | 0.86 | 0/12 | +| 0.25 | 0.86 | 0/12 | +| 0.30 | 0.86 | 0/12 | +| 0.35 | 0.86 | 0/12 | +| 0.40 | 0.71 | 0/12 | + +Four deliberately off-topic queries stand in for a task the store knows nothing about. Without a floor the engine returns k memories anyway, and the MCP layer strips scores, so the agent cannot tell. `SentenceTransformerEmbedder.recommended_min_score` is set from this sweep. + +## Honest limits + +- 7 tasks and 10 gold labels, all hand-written by the author. This is an engineering check, not a production-scale claim; one retrieval either way moves recall by ~0.07. +- The labels, the queries and the retriever all come from the same person, which is exactly the setup that flatters a retriever. The paraphrase arm exists to push back on that. +- Numbers use `sentence-transformers` (all-MiniLM-L6-v2), which needs a model download and is therefore not run in CI. Regenerate with: `python eval/run_eval.py --embedder sentence-transformers`. diff --git a/eval/run_eval.py b/eval/run_eval.py index 073de99..5390957 100644 --- a/eval/run_eval.py +++ b/eval/run_eval.py @@ -49,6 +49,8 @@ from agent_memory import HashingEmbedder, MemoryStore, count_tokens # noqa: E402 from agent_memory.tokens import using_exact_tokenizer # noqa: E402 +EMBEDDER_CHOICES = ("hashing", "sentence-transformers") + ARM_LABELS = { "no_memory": "No memory (control)", "full_context": "Full context (baseline — load every memory)", @@ -75,18 +77,29 @@ def mean(xs: list[float]) -> float: return sum(xs) / len(xs) if xs else 0.0 -def build_store(memories: list[dict]) -> MemoryStore: +def make_embedder(name: str): + """Build an embedder by name, so the same benchmark can score both.""" + if name == "hashing": + return HashingEmbedder() + from agent_memory import SentenceTransformerEmbedder + + return SentenceTransformerEmbedder() + + +def build_store(memories: list[dict], embedder=None) -> MemoryStore: # Explicit embedder: the published numbers are the offline one's, and we do # not want them to change silently on a machine with sentence-transformers. - store = MemoryStore(embedder=HashingEmbedder()) + store = MemoryStore(embedder=embedder or HashingEmbedder()) for m in memories: store.write(m["text"], type=m["type"], id=m["id"]) return store -def evaluate(dataset: dict, k: int = 3, budget: int = 120, seed: int = 0) -> dict: +def evaluate( + dataset: dict, k: int = 3, budget: int = 120, seed: int = 0, embedder=None +) -> dict: memories = dataset["memories"] - store = build_store(memories) + store = build_store(memories, embedder) rng = random.Random(seed) all_ids = [m["id"] for m in memories] @@ -159,19 +172,19 @@ def record(name: str, ids: list[str], tokens: int) -> tuple[float, float]: "n_tasks": len(dataset["tasks"]), "summary": summary, "per_task": per_task, - "phrasing": phrasing_gap(dataset, k=k), - "scaling": scaling(dataset, k=k, budget=budget), - "min_score_sweep": min_score_sweep(dataset, k=k), + "phrasing": phrasing_gap(dataset, k=k, embedder=embedder), + "scaling": scaling(dataset, k=k, budget=budget, embedder=embedder), + "min_score_sweep": min_score_sweep(dataset, k=k, embedder=embedder), } -def phrasing_gap(dataset: dict, k: int = 3) -> dict: +def phrasing_gap(dataset: dict, k: int = 3, embedder=None) -> dict: """Recall on the developer phrasing vs an outsider's paraphrase. The default embedder matches shared wording. This is the number that says how much of the headline recall comes from the benchmark's phrasing. """ - store = build_store(dataset["memories"]) + store = build_store(dataset["memories"], embedder) out = {} for field in ("query", "paraphrase"): recalls, overlaps = [], [] @@ -206,7 +219,7 @@ def _word_overlap(query: str, memories: list[dict], gold: set[str]) -> float: return len(words & gold_words) / len(words) -def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]: +def scaling(dataset: dict, k: int = 3, budget: int = 120, embedder=None) -> list[dict]: """Grow the store with distractors and re-measure. "The cost of memory stays fixed as the store grows" is a claim about a store @@ -220,7 +233,7 @@ def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]: {"id": f"dis_{i:04d}", "type": d["type"], "text": d["text"]} for i, d in enumerate(distractors[:extra]) ] - store = build_store(memories) + store = build_store(memories, embedder) full_tokens = sum(count_tokens(m["text"]) for m in memories) recalls, tokens = [], [] for task in dataset["tasks"]: @@ -240,13 +253,13 @@ def scaling(dataset: dict, k: int = 3, budget: int = 120) -> list[dict]: return rows -def min_score_sweep(dataset: dict, k: int = 3) -> list[dict]: +def min_score_sweep(dataset: dict, k: int = 3, embedder=None) -> list[dict]: """Calibrate the relevance floor used by the CLI and MCP server. Labelled recall on real queries must not drop, while queries about things the store knows nothing about should return nothing at all. """ - store = build_store(dataset["memories"]) + store = build_store(dataset["memories"], embedder) off_topic = [ "how do I bake sourdough bread at home", "what is the capital of Peru", @@ -254,7 +267,7 @@ def min_score_sweep(dataset: dict, k: int = 3) -> list[dict]: "best hiking boots for winter walking", ] rows = [] - for threshold in (0.0, 0.05, 0.10, 0.15, 0.20, 0.25): + for threshold in (0.0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40): recalls = [] for task in dataset["tasks"]: gold = set(task["relevant_ids"]) @@ -327,13 +340,23 @@ def render_markdown(results: dict) -> str: f"| {label} | {row['avg_query_word_overlap_with_gold']:.0%} | " f"{row['recall']:.2f} |" ) + gap = ph["query"]["recall"] - ph["paraphrase"]["recall"] + if results["embedder"] == "HashingEmbedder": + verdict = ( + "The default embedder is feature hashing — lexical, not semantic. When " + f"the query stops sharing words with the memory, recall drops by " + f"**{gap:.2f}**. That gap is the honest limit of the offline default, " + "and the reason the optional `sentence-transformers` backend exists." + ) + else: + verdict = ( + f"This backend embeds meaning rather than wording, so the drop is only " + f"**{gap:.2f}**. It is the reason to install the `real` extra for day-to-day " + "use: real questions rarely reuse the words a memory was written in." + ) lines += [ "", - "The default embedder is feature hashing — lexical, not semantic. When the " - "query stops sharing words with the memory, recall drops by " - f"**{(ph['query']['recall'] - ph['paraphrase']['recall']):.2f}**. That gap is " - "the honest limit of the offline default, and the reason the optional " - "`sentence-transformers` backend exists.", + verdict, "", "## Does the cost stay flat as the store grows?", "", @@ -365,8 +388,8 @@ def render_markdown(results: dict) -> str: "", "Four deliberately off-topic queries stand in for a task the store knows " "nothing about. Without a floor the engine returns k memories anyway, and " - "the MCP layer strips scores, so the agent cannot tell. `HashingEmbedder." - "recommended_min_score` is set from this sweep.", + "the MCP layer strips scores, so the agent cannot tell. " + f"`{results['embedder']}.recommended_min_score` is set from this sweep.", "", "## Honest limits", "", @@ -377,8 +400,15 @@ def render_markdown(results: dict) -> str: "- The labels, the queries and the retriever all come from the same person, " "which is exactly the setup that flatters a retriever. The paraphrase arm " "exists to push back on that.", - "- Numbers use the offline hashing embedder so they are stable in CI. The " - "`sentence-transformers` backend is not measured here.", + ( + "- Numbers use the offline hashing embedder so they are stable in CI. " + "The `sentence-transformers` backend is measured separately in " + "`results_sentence_transformers.md`." + if results["embedder"] == "HashingEmbedder" + else "- Numbers use `sentence-transformers` (all-MiniLM-L6-v2), which " + "needs a model download and is therefore not run in CI. Regenerate " + "with: `python eval/run_eval.py --embedder sentence-transformers`." + ), ] return "\n".join(lines) @@ -395,15 +425,29 @@ def main() -> None: parser.add_argument( "--dataset", type=Path, default=Path(__file__).parent / "dataset.json" ) + parser.add_argument( + "--embedder", + choices=EMBEDDER_CHOICES, + default="hashing", + help="which backend to score. The committed results.md is the hashing " + "run, because it is the one that reproduces offline and in CI.", + ) parser.add_argument("--out-dir", type=Path, default=Path(__file__).parent) args = parser.parse_args() dataset = load_dataset(args.dataset) - results = evaluate(dataset, k=args.k, budget=args.budget, seed=args.seed) + embedder = make_embedder(args.embedder) + results = evaluate( + dataset, k=args.k, budget=args.budget, seed=args.seed, embedder=embedder + ) + # Keep the two runs in separate files: results.md is the reproducible one + # CI regenerates and diffs, so a machine with the model installed must not + # silently overwrite it with numbers CI can never reproduce. + stem = "results" if args.embedder == "hashing" else "results_sentence_transformers" md = render_markdown(results) - (args.out_dir / "results.md").write_text(md + "\n") - (args.out_dir / "results.json").write_text(json.dumps(results, indent=2) + "\n") + (args.out_dir / f"{stem}.md").write_text(md + "\n") + (args.out_dir / f"{stem}.json").write_text(json.dumps(results, indent=2) + "\n") print(md) if not results["exact_tokenizer"]: print( diff --git a/src/agent_memory/__init__.py b/src/agent_memory/__init__.py index 21358fa..3527914 100644 --- a/src/agent_memory/__init__.py +++ b/src/agent_memory/__init__.py @@ -8,11 +8,14 @@ default_min_score, ) from .store import ( + GLOBAL_STORE, MEMORY_TYPES, STORE_FORMAT, MemoryEntry, MemoryStore, RecallHit, + default_store_path, + find_project_root, ) from .tokens import count_tokens @@ -24,6 +27,9 @@ "RecallHit", "MEMORY_TYPES", "STORE_FORMAT", + "GLOBAL_STORE", + "default_store_path", + "find_project_root", "Embedder", "HashingEmbedder", "SentenceTransformerEmbedder", diff --git a/src/agent_memory/cli.py b/src/agent_memory/cli.py index 6c4a34b..78685ff 100644 --- a/src/agent_memory/cli.py +++ b/src/agent_memory/cli.py @@ -18,22 +18,39 @@ import argparse import os +import sys from pathlib import Path from .embeddings import default_min_score -from .store import MEMORY_TYPES, MemoryStore +from .store import ( + GLOBAL_STORE, + MEMORY_TYPES, + MemoryStore, + default_store_path, + relocation_notice, +) -DEFAULT_STORE = Path( - os.environ.get("AGENT_MEMORY_PATH", "~/.agent_memory/store.json") -).expanduser() DEFAULT_AGENT = os.environ.get("AGENT_MEMORY_AGENT", "") # Sentinel: resolved per-embedder once the store is open (see default_min_score). AUTO_MIN_SCORE = -1.0 +def _resolve_path(args) -> Path: + """Explicit --path, then --global, then the project/global default.""" + if args.path is not None: + return Path(args.path).expanduser() + if getattr(args, "use_global", False): + return GLOBAL_STORE + path = default_store_path() + notice = relocation_notice(path) + if notice: + print(notice, file=sys.stderr) + return path + + def _store(args) -> MemoryStore: - return MemoryStore(path=args.path) + return MemoryStore(path=_resolve_path(args)) def _min_score(args, store: MemoryStore) -> float: @@ -115,7 +132,9 @@ def cmd_forget(args) -> None: def cmd_stats(args) -> None: - s = _store(args).stats() + path = _resolve_path(args) + s = MemoryStore(path=path).stats() + print(f"store: {path}") print(f"{s['count']} memories | {s['total_tokens']} tokens | {s['embedder']}") for t, n in sorted(s["by_type"].items()): print(f" {t}: {n}") @@ -123,7 +142,18 @@ def cmd_stats(args) -> None: def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(prog="agent-memory") - parser.add_argument("--path", type=Path, default=DEFAULT_STORE) + parser.add_argument( + "--path", + type=Path, + default=None, + help="store file (default: this project's .agent_memory/store.json)", + ) + parser.add_argument( + "--global", + dest="use_global", + action="store_true", + help=f"use the cross-project store at {GLOBAL_STORE}", + ) parser.add_argument("--agent", default=DEFAULT_AGENT, help="who is writing") sub = parser.add_subparsers(dest="command", required=True) diff --git a/src/agent_memory/embeddings.py b/src/agent_memory/embeddings.py index 89219de..29a48ac 100644 --- a/src/agent_memory/embeddings.py +++ b/src/agent_memory/embeddings.py @@ -101,17 +101,33 @@ def embed(self, texts: list[str]) -> np.ndarray: class SentenceTransformerEmbedder: """Real semantic embeddings. Optional: needs ``sentence-transformers``.""" - # NOT calibrated in this repo — the offline benchmark runs on the hashing - # embedder, and MiniLM cosines sit on a higher scale (unrelated pairs often - # score 0.1-0.3). Treat this as a conservative starting point and measure on - # your own store before relying on it. - recommended_min_score = 0.25 + # Calibrated on eval/dataset.json — see eval/results_sentence_transformers.md. + # Every off-topic query is already rejected at 0.15, and labelled recall is + # flat until it falls at 0.40, so anything in 0.15-0.35 scores identically on + # the benchmark. The tie is broken by headroom rather than by the sweep: + # genuine paraphrases can land low ("which AI model answers customer + # questions" scores 0.22 against a memory that answers it), so the floor sits + # just under that rather than in the middle of the safe band. MiniLM cosines + # run higher than the hashing embedder's, which is why this differs from it. + recommended_min_score = 0.20 def __init__(self, model_name: str = "all-MiniLM-L6-v2") -> None: from sentence_transformers import SentenceTransformer # lazy import self._model = SentenceTransformer(model_name) - self.dim = int(self._model.get_sentence_embedding_dimension()) + # Renamed in sentence-transformers 5.x; support both so an upgrade of an + # optional dependency cannot break the backend. + get_dim = getattr( + self._model, + "get_embedding_dimension", + getattr(self._model, "get_sentence_embedding_dimension", None), + ) + if get_dim is None: # pragma: no cover - depends on the installed version + raise RuntimeError( + "could not determine the embedding dimension from " + f"sentence-transformers model {model_name!r}" + ) + self.dim = int(get_dim()) self.model_name = model_name def embed(self, texts: list[str]) -> np.ndarray: diff --git a/src/agent_memory/mcp_server.py b/src/agent_memory/mcp_server.py index 32e1b13..890756d 100644 --- a/src/agent_memory/mcp_server.py +++ b/src/agent_memory/mcp_server.py @@ -21,15 +21,12 @@ from __future__ import annotations import os +import sys from pathlib import Path from typing import Optional from .embeddings import default_min_score -from .store import MEMORY_TYPES, MemoryStore - -DEFAULT_STORE = Path( - os.environ.get("AGENT_MEMORY_PATH", "~/.agent_memory/store.json") -).expanduser() +from .store import MEMORY_TYPES, MemoryStore, default_store_path, relocation_notice # Who is talking to the store — "claude-code", "codex", "cursor", ... DEFAULT_AGENT = os.environ.get("AGENT_MEMORY_AGENT", "") @@ -69,14 +66,26 @@ def _render(hits) -> str: def build_server( - store_path: Path = DEFAULT_STORE, + store_path: Optional[Path] = None, agent: str = DEFAULT_AGENT, min_score: Optional[float] = None, ): """Construct the MCP server. Imports `mcp` lazily so importing this module - never hard-fails when the optional dependency is absent.""" + never hard-fails when the optional dependency is absent. + + With no `store_path`, the store is resolved per project: the agent launches + this server from the working directory, so `.agent_memory/store.json` in the + enclosing repository is the store its memories belong to. + """ server_class = _load_server_class() + if store_path is None: + store_path = default_store_path() + notice = relocation_notice(store_path) + if notice: + # stdout is the JSON-RPC channel; anything printed there corrupts it. + print(notice, file=sys.stderr) + store = MemoryStore(path=store_path) if min_score is None: min_score = default_min_score(store.embedder) diff --git a/src/agent_memory/store.py b/src/agent_memory/store.py index 1896b6f..df3a7f8 100644 --- a/src/agent_memory/store.py +++ b/src/agent_memory/store.py @@ -47,11 +47,68 @@ _ID_RE = re.compile(r"^mem_(\d+)$") +# Where memories live when nothing is configured. One store per project, not one +# store for everything you have ever worked on: recall matches on similarity +# alone, so a single global file lets one project's deploy notes surface while +# you are working on another. +PROJECT_STORE_DIR = ".agent_memory" +STORE_FILENAME = "store.json" +GLOBAL_STORE = Path.home() / PROJECT_STORE_DIR / STORE_FILENAME + def _now_iso() -> str: return datetime.now(timezone.utc).isoformat(timespec="seconds") +def find_project_root(start: Optional[str | Path] = None) -> Optional[Path]: + """Nearest ancestor directory containing `.git`, or None outside a repo.""" + current = Path(start).expanduser().resolve() if start else Path.cwd().resolve() + for candidate in (current, *current.parents): + # `.git` is a directory in a normal clone and a file in a worktree or + # submodule, so test for existence rather than for a directory. + if (candidate / ".git").exists(): + return candidate + return None + + +def default_store_path(start: Optional[str | Path] = None) -> Path: + """Resolve which store to use. + + In precedence order: an explicit ``AGENT_MEMORY_PATH``, then the current + project's ``.agent_memory/store.json``, then a global store for work that + isn't in a repository. + """ + configured = os.environ.get("AGENT_MEMORY_PATH") + if configured: + return Path(configured).expanduser() + root = find_project_root(start) + if root is not None: + return root / PROJECT_STORE_DIR / STORE_FILENAME + return GLOBAL_STORE + + +def relocation_notice(path: Path) -> Optional[str]: + """Warn once when a fresh project store is used but a global one has data. + + Stores used to default to a single global file. Without this, upgrading + looks like every memory was deleted. Callers must print it to stderr — on + the MCP server stdout carries the protocol. + """ + if path == GLOBAL_STORE or path.exists() or not GLOBAL_STORE.exists(): + return None + try: + count = len(json.loads(GLOBAL_STORE.read_text()).get("entries", [])) + except (OSError, ValueError): + return None + if not count: + return None + return ( + f"[agent-memory] Starting an empty store for this project at {path}. " + f"Your global store still holds {count} memories — use them here with " + f"AGENT_MEMORY_PATH={GLOBAL_STORE}" + ) + + @dataclass class MemoryEntry: id: str diff --git a/tests/test_sentence_transformers.py b/tests/test_sentence_transformers.py new file mode 100644 index 0000000..bd82c80 --- /dev/null +++ b/tests/test_sentence_transformers.py @@ -0,0 +1,83 @@ +"""The optional sentence-transformers backend. + +Skipped unless the `real` extra is installed, so CI stays offline and fast. +Run locally with: pip install -e ".[real]" && pytest tests/test_sentence_transformers.py +""" + +import pytest + +pytest.importorskip( + "sentence_transformers", reason='needs the optional "real" extra' +) + +from agent_memory import MemoryStore, SentenceTransformerEmbedder # noqa: E402 + + +@pytest.fixture(scope="module") +def embedder(): + return SentenceTransformerEmbedder() + + +@pytest.fixture(scope="module") +def store(embedder): + s = MemoryStore(embedder=embedder) + s.write("Bookings are stored in UTC and converted in the UI layer.", type="decision") + s.write("The customer chatbot uses Google Gemini in server/gemini-chat.ts.", type="decision") + s.write("Admin routes are guarded by requireAdmin in server/auth.ts.", type="decision") + s.write("Uploaded images are private unless they sit under the gallery/ prefix.", type="project") + return s + + +def test_dimension_is_discovered_across_library_versions(embedder): + """`get_sentence_embedding_dimension` was renamed in 5.x; both must work.""" + assert embedder.dim == 384 + assert embedder.embed(["hello"]).shape == (1, 384) + + +def test_vectors_are_unit_norm(embedder): + vecs = embedder.embed(["one sentence", "another entirely different sentence"]) + norms = (vecs**2).sum(axis=1) ** 0.5 + assert all(abs(n - 1.0) < 1e-4 for n in norms) + + +@pytest.mark.parametrize( + "paraphrase, expected", + [ + ("a client saw the wrong hour for their appointment", "UTC"), + ("make sure only administrators can reach the new settings page", "requireAdmin"), + ("how do we restrict a route to administrators", "requireAdmin"), + ("which AI model answers customer questions", "Gemini"), + ], +) +def test_recall_survives_a_paraphrase(store, paraphrase, expected): + """The whole reason this backend exists: no shared vocabulary.""" + hits = store.recall(paraphrase, k=1) + assert expected in hits[0].entry.text + + +def test_a_genuinely_ambiguous_query_is_not_answered_confidently(store): + """Two memories are about access control, and this query fits both equally. + + Both score ~0.16 — below the configured floor — so the honest outcome is + nothing rather than a coin-flip presented to the agent as fact. + """ + floor = SentenceTransformerEmbedder.recommended_min_score + assert store.recall("only staff should be able to open this screen", + k=2, min_score=floor) == [] + + +def test_configured_floor_keeps_real_matches_and_drops_junk(store): + floor = SentenceTransformerEmbedder.recommended_min_score + assert store.recall("how are timezones handled", k=3, min_score=floor) + assert store.recall("best sourdough bread recipe for beginners", k=3, min_score=floor) == [] + + +def test_a_store_written_by_this_backend_reloads(tmp_path, embedder): + path = tmp_path / "st.json" + first = MemoryStore(path=path, embedder=embedder) + first.write("Calendar sync must never block a booking from saving.", type="decision") + + reopened = MemoryStore(path=path, embedder=embedder) + assert reopened.stats()["count"] == 1 + hits = reopened.recall("what happens if the calendar integration fails", k=1) + assert "Calendar sync" in hits[0].entry.text diff --git a/tests/test_store_location.py b/tests/test_store_location.py new file mode 100644 index 0000000..1b2ae6e --- /dev/null +++ b/tests/test_store_location.py @@ -0,0 +1,115 @@ +"""Which store a command uses. + +Memories used to land in one global file shared by every project, so recall +could surface another codebase's answers. The store now follows the project. +""" + +import json + +import pytest + +from agent_memory import GLOBAL_STORE, default_store_path, find_project_root +from agent_memory.cli import build_parser, main +from agent_memory.store import relocation_notice + + +@pytest.fixture +def repo(tmp_path, monkeypatch): + """A directory that looks like a git checkout, with a nested subdirectory.""" + monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False) + root = tmp_path / "my-project" + (root / ".git").mkdir(parents=True) + (root / "src" / "deep").mkdir(parents=True) + return root + + +def test_store_follows_the_project(repo, monkeypatch): + monkeypatch.chdir(repo) + assert default_store_path() == repo / ".agent_memory" / "store.json" + + +def test_store_is_found_from_a_nested_directory(repo, monkeypatch): + monkeypatch.chdir(repo / "src" / "deep") + assert default_store_path() == repo / ".agent_memory" / "store.json" + + +def test_two_projects_get_two_stores(tmp_path, monkeypatch): + monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False) + first, second = tmp_path / "alpha", tmp_path / "beta" + for project in (first, second): + (project / ".git").mkdir(parents=True) + monkeypatch.chdir(first) + a = default_store_path() + monkeypatch.chdir(second) + b = default_store_path() + assert a != b + + +def test_git_worktrees_and_submodules_are_recognised(tmp_path, monkeypatch): + """`.git` is a file, not a directory, in a worktree or submodule.""" + monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False) + root = tmp_path / "worktree" + root.mkdir() + (root / ".git").write_text("gitdir: /elsewhere/.git/worktrees/wt\n") + monkeypatch.chdir(root) + assert find_project_root() == root + assert default_store_path() == root / ".agent_memory" / "store.json" + + +def test_outside_a_repository_falls_back_to_the_global_store(tmp_path, monkeypatch): + monkeypatch.delenv("AGENT_MEMORY_PATH", raising=False) + loose = tmp_path / "not-a-repo" + loose.mkdir() + monkeypatch.chdir(loose) + assert default_store_path() == GLOBAL_STORE + + +def test_explicit_env_var_still_wins(repo, monkeypatch, tmp_path): + monkeypatch.chdir(repo) + monkeypatch.setenv("AGENT_MEMORY_PATH", str(tmp_path / "chosen.json")) + assert default_store_path() == tmp_path / "chosen.json" + + +def test_memories_written_in_a_project_stay_in_that_project(repo, monkeypatch, capsys): + monkeypatch.setenv("AGENT_MEMORY_EMBEDDER", "hashing") + monkeypatch.chdir(repo) + main(["write", "Bookings are stored in UTC.", "--type", "decision"]) + + store_file = repo / ".agent_memory" / "store.json" + assert store_file.exists() + assert "UTC" in store_file.read_text() + + other = repo.parent / "other-project" + (other / ".git").mkdir(parents=True) + monkeypatch.chdir(other) + capsys.readouterr() + main(["recall", "how are timezones handled"]) + assert "UTC" not in capsys.readouterr().out, "another project's memory leaked in" + + +def test_global_flag_opts_back_into_the_shared_store(repo, monkeypatch): + monkeypatch.chdir(repo) + args = build_parser().parse_args(["--global", "stats"]) + assert args.use_global is True + + +def test_relocation_notice_points_at_existing_global_memories(tmp_path, monkeypatch): + """Upgrading must not look like every memory was deleted.""" + fake_global = tmp_path / "global.json" + fake_global.write_text(json.dumps({"entries": [{"id": "mem_0001", "text": "x"}]})) + monkeypatch.setattr("agent_memory.store.GLOBAL_STORE", fake_global) + + fresh = tmp_path / "proj" / ".agent_memory" / "store.json" + notice = relocation_notice(fresh) + assert notice and "1 memories" in notice and str(fake_global) in notice + + # Silent once the project store exists, and silent for the global store itself. + fresh.parent.mkdir(parents=True) + fresh.write_text(json.dumps({"entries": []})) + assert relocation_notice(fresh) is None + assert relocation_notice(fake_global) is None + + +def test_notice_is_silent_when_there_is_nothing_to_migrate(tmp_path, monkeypatch): + monkeypatch.setattr("agent_memory.store.GLOBAL_STORE", tmp_path / "absent.json") + assert relocation_notice(tmp_path / "proj" / "store.json") is None From 657f3b0e614f76e0b34af2b3f053ef56fbb9c16a Mon Sep 17 00:00:00 2001 From: Nina Doinjashvili Date: Wed, 12 Aug 2026 23:36:24 +0300 Subject: [PATCH 2/2] README: restructure for readers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The page had grown by accretion — heavy evaluation analysis before the reader knew what the thing was, quickstart buried two thirds down, and install instructions in three different places. Reordered to try it -> understand it -> evidence -> reference: - Quick start first, with a real terminal session whose output is copied from an actual run: a paraphrased query that shares no words with the memory it finds, and an off-topic query that returns nothing. - One install table replacing three scattered pip lines. - "How it works" as a three-row tool table before the sequence diagram. - Evaluation consolidated under "Does it actually work?", keeping the random control, the embedder comparison and the caveats. - New Reference section: CLI commands, memory types, environment variables, Python API, migration — previously scattered or unwritten. - Added CI/Python/licence badges. Every number in the page is checked against eval/results*.json, the shipped floors, the test count and the tool count. Co-Authored-By: Claude Opus 5 --- README.md | 284 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 160 insertions(+), 124 deletions(-) diff --git a/README.md b/README.md index ff9e9df..d9f057a 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,61 @@ # Agent Memory Engine -**One shared memory for your coding agents — Claude Code, Codex CLI, Cursor — over MCP, with token-budgeted recall so it never floods a context window.** - -Every coding agent forgets everything between sessions, and none of them can read another's notes: what Claude Code learned about your codebase is invisible to Codex, and vice versa. The common fix — a pile of Markdown memory files loaded into every prompt — has the opposite problem: it burns more tokens every week as the pile grows. +[![CI](https://github.com/Ninadnj/ai-agent-memory-scaffold/actions/workflows/ci.yml/badge.svg)](https://github.com/Ninadnj/ai-agent-memory-scaffold/actions/workflows/ci.yml) +[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/) +[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) -This engine is the small piece in between. Agents write atomic facts, decisions and handoffs to one store; at the start of a task any agent recalls **only the memories relevant to that task, under a hard token budget you set**. It runs as a [Model Context Protocol](https://modelcontextprotocol.io) server, so every MCP-capable agent shares the same memory — and every memory records which agent wrote it. - -> This engine is the working evolution of the Markdown memory *convention* this repository originally hosted — still included and usable in [`scaffold/`](scaffold/) — into a measured retrieval *system*. +**One shared memory for your coding agents — Claude Code, Codex CLI, Cursor — over MCP, with token-budgeted recall so it never floods a context window.** -## Results +Coding agents forget everything between sessions, and none of them can read another's notes: what Claude Code learned about your codebase is invisible to Codex. The usual fix — a pile of Markdown files loaded into every prompt — has the opposite problem, costing more tokens every week as the pile grows. -On a hand-labeled benchmark of an agent working across many sessions on one codebase (14 durable memories, 7 fresh-session tasks, top-k = 3): +This engine sits in between. Agents write short, atomic memories to one store; at the start of a task, any agent gets back **only the memories relevant to that task, under a token budget you set**. It runs as a [Model Context Protocol](https://modelcontextprotocol.io) server, so every MCP-capable tool shares the same memory, and every memory records which agent wrote it. -| Arm | Avg context tokens | Recall | Precision | Tokens saved vs baseline | -| --- | ---: | ---: | ---: | ---: | -| No memory (control) | 0 | 0.00 | 0.00 | 100% | -| Full context (baseline — load every memory) | 424 | 1.00 | 0.10 | 0% | -| *Random k (control)* | *93* | *0.07* | *0.05* | *78%* | -| **Targeted retrieval (this engine)** | **93** | **0.93** | **0.43** | **78%** | -| **Budget recall (≤ 120 tokens, hard cap)** | **93** | **0.93** | **0.43** | **78%** | +> The original Markdown *convention* this repository hosted is still here and still usable, in [`scaffold/`](scaffold/). This engine is its measured successor. -**Read the random row first.** It loads three memories picked at random, so it reports the *same* 78% token saving with 0.07 recall. The saving is arithmetic — you loaded 3 of 14 memories — and proves nothing on its own. The claim worth making is the **0.86 recall gap between random and retrieval at identical token cost**. +--- -Reproduce with `python eval/run_eval.py`; the full report is in [eval/results.md](eval/results.md). +## Quick start -### Which embedder should you use? +```bash +git clone https://github.com/Ninadnj/ai-agent-memory-scaffold.git +cd ai-agent-memory-scaffold +pip install -e ".[mcp,real]" +``` -The offline default is **lexical** — feature hashing over word and character n-grams — so it matches shared wording. That flatters it on a benchmark whose queries reuse the memories' vocabulary. Every task therefore carries a second phrasing that deliberately avoids that vocabulary, and both backends are measured on both: +| Install | Gets you | +| --- | --- | +| `pip install -e .` | Engine + CLI. numpy only, fully offline. | +| `pip install -e ".[mcp,real]"` | **Recommended.** Adds the MCP server and semantic embeddings. | +| `pip install -e ".[dev]"` | Everything above minus the model, plus pytest. What CI runs. | -| | Hashing (offline default) | MiniLM (`real` extra) | -| --- | ---: | ---: | -| Developer phrasing — 40% word overlap | **0.93** | 0.86 | -| Outsider paraphrase — 3% word overlap | 0.43 | **0.79** | -| Average of the two | 0.68 | **0.82** | -| Off-topic queries correctly rejected | 6/12 | **12/12** | +Try it in your terminal: -It isn't a clean sweep, and that's the interesting part. **Exact word matching genuinely wins when the words match** — if you write memories and query them in the same vocabulary, hashing is better *and* needs no model. But real use is mostly the second row: you write a memory in March and ask about it in July, in different words. +```bash +agent-memory write "Bookings are stored in UTC; the UI converts to local time." --type decision +agent-memory write "Admin routes are guarded by requireAdmin in server/auth.ts." --type decision -**Recommendation: install the `real` extra for day-to-day use** (`pip install -e ".[real]"`), and keep the hashing default for CI, air-gapped machines, and anywhere a 90 MB model download isn't welcome. Full reports: [results.md](eval/results.md) · [results_sentence_transformers.md](eval/results_sentence_transformers.md). +agent-memory recall "a client saw the wrong hour for their appointment" +# 0.41 [decision] Bookings are stored in UTC; the UI converts to local time. -### What these numbers still don't show +agent-memory recall "best sourdough bread recipe" +# No relevant memories. +``` -- **The precision column is nearly meaningless for the baseline.** Full context scores 0.10 because that is `|relevant| / |store|` — an artefact of loading everything. -- **7 tasks, 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim. -- **MiniLM numbers are not reproduced in CI**, because they need a model download. CI regenerates and diffs the hashing results only. +That first query shares no words with the memory it found, and the second returns nothing rather than guessing. Both behaviours are the point. -### Does the cost stay flat as the store grows? +--- -Adding unrelated memories from the same project, re-measuring against the same labels: +## How it works -| Memories in store | Full context tokens | Budget recall tokens | Budget recall | -| ---: | ---: | ---: | ---: | -| 14 | 424 | 93 | 0.93 | -| 34 | 805 | 87 | 0.93 | -| 54 | 1159 | 82 | 0.93 | +Three habits, three tools: -The baseline grows with the store. The budgeted arm does not, and recall holds. +| Tool | When | What it does | +| --- | --- | --- | +| `memory_boot(task, budget_tokens)` | Start of a session | Returns the previous agent's handoff plus the memories most relevant to your task, packed under one token budget. | +| `memory_write(text, type)` | The moment something is learned | Saves one durable fact. Near-duplicates are skipped — and it tells you so instead of reporting a save that didn't happen. | +| `memory_handoff(done, next, warnings)` | End of a session | Leaves a note so the next agent continues instead of rediscovering. | -## The cross-agent loop +Plus `memory_list`, `memory_update` and `memory_forget` — memory you can't correct is worse than no memory, because a stale note keeps being recalled with full confidence. ```mermaid sequenceDiagram @@ -72,23 +70,13 @@ sequenceDiagram Note over CX: picks up exactly where Claude Code stopped ``` -Three habits, three tools: +Tool output is deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. The trade-off is that the agent can't judge relevance itself, so weak matches are filtered out before they're returned. -- **`memory_boot(task, budget_tokens)`** — call once at session start: the latest handoff from the *previous agent, whichever tool it was*, plus the memories most relevant to the task, packed under one token budget. -- **`memory_write(text, type)`** — persist a durable fact, decision or issue the moment it's learned. Near-duplicates are dropped, and the tool says so rather than reporting a save that didn't happen. -- **`memory_handoff(done, next_steps, warnings)`** — call at session end so the next agent continues instead of rediscovering. +--- -Plus `memory_list`, `memory_update` and `memory_forget`, because memory you cannot correct is worse than no memory: a stale `state` entry keeps being recalled and quietly misleads every later session. +## Connect your agents -Tool outputs are deliberately compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. The flip side is that the agent can't judge relevance itself, so weak matches are filtered out before they're returned (see [Relevance floor](#relevance-floor)). - -## Hook it up to your agents - -```bash -pip install -e ".[mcp,real]" # drop `real` to stay fully offline -``` - -Works with both `mcp` 1.x and 2.x. Give each agent its own name — the store itself needs no configuration: +Give each agent its own name. The store needs no configuration — it follows the project you're in. **Claude Code** @@ -96,7 +84,7 @@ Works with both `mcp` 1.x and 2.x. Give each agent its own name — the store it claude mcp add agent-memory -e AGENT_MEMORY_AGENT=claude-code -- agent-memory-mcp ``` -**Codex CLI** (`~/.codex/config.toml`) +**Codex CLI** — `~/.codex/config.toml` ```toml [mcp_servers.agent-memory] @@ -104,7 +92,7 @@ command = "agent-memory-mcp" env = { AGENT_MEMORY_AGENT = "codex" } ``` -**Cursor** (`.cursor/mcp.json`) +**Cursor** — `.cursor/mcp.json` ```json { @@ -117,28 +105,79 @@ env = { AGENT_MEMORY_AGENT = "codex" } } ``` -Each memory carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next via `memory_boot`. Agents may run at the same time: writes take a lock, merge, and land atomically, and an already-running server picks up another agent's writes on its next read. +Works with `mcp` 1.x and 2.x. Every memory carries its origin (`[decision · claude-code]`, `[handoff · codex]`), and a handoff written in one tool is picked up by the next. Agents may run at the same time: writes take a lock, merge, and land atomically, and a running server sees another agent's writes on its next read. ### One store per project -Memories are scoped to the repository you're working in. The store resolves in this order: +Memories are scoped to the repository you're working in: 1. `AGENT_MEMORY_PATH`, if you set it 2. `.agent_memory/store.json` in the current git repository — **the normal case** -3. `~/.agent_memory/store.json`, when you're not inside a repository +3. `~/.agent_memory/store.json`, when you're not in a repository -This matters more than it sounds. Recall matches on similarity alone, so a single shared store lets one project's answer to "how do we deploy?" surface while you're working on a different project. Per-project stores make that impossible. +This matters more than it sounds. Recall matches on similarity alone, so one shared store would let another project's answer to *"how do we deploy?"* surface while you work here. ```bash -agent-memory stats # prints which store is in use +agent-memory stats # which store am I using? agent-memory --global stats # the cross-project store, when you want it ``` -Add `.agent_memory/` to your `.gitignore` unless you intend to commit the memories — sharing them with a team is a legitimate choice, but it should be a deliberate one. +Add `.agent_memory/` to your `.gitignore` unless you mean to commit the memories — sharing them with a team is reasonable, but it should be deliberate. + +--- + +## Does it actually work? + +A hand-labelled benchmark: 14 memories from one codebase, 7 fresh-session tasks, top-k = 3. + +| Arm | Avg context tokens | Recall | Tokens saved | +| --- | ---: | ---: | ---: | +| No memory (control) | 0 | 0.00 | 100% | +| Full context — load everything | 424 | 1.00 | 0% | +| *Random k (control)* | *93* | *0.07* | *78%* | +| **Targeted retrieval** | **93** | **0.93** | **78%** | +| **Budget recall (≤ 120 tokens)** | **93** | **0.93** | **78%** | + +**Read the random row first.** Three memories picked at random report the *same* 78% saving at 0.07 recall. The saving is arithmetic — you loaded 3 of 14 — and proves nothing by itself. The real claim is the **0.86 recall gap at identical token cost**. + +Reproduce with `python eval/run_eval.py`. Full report: [eval/results.md](eval/results.md). + +### Cost stays flat as memory grows + +| Memories in store | Full context | Budget recall | Recall | +| ---: | ---: | ---: | ---: | +| 14 | 424 tokens | 93 tokens | 0.93 | +| 34 | 805 tokens | 87 tokens | 0.93 | +| 54 | 1159 tokens | 82 tokens | 0.93 | + +The baseline grows with the store. The budgeted arm doesn't, and recall holds. + +### Which embedder should you use? + +The offline default is **lexical** — it matches shared wording, not meaning. So every task in the benchmark carries a second phrasing that deliberately avoids the memories' vocabulary, and both backends are scored on both: + +| | Hashing (default, offline) | MiniLM (`real` extra) | +| --- | ---: | ---: | +| Developer phrasing — 40% word overlap | **0.93** | 0.86 | +| Outsider paraphrase — 3% word overlap | 0.43 | **0.79** | +| Average | 0.68 | **0.82** | +| Off-topic queries rejected | 6/12 | **12/12** | + +It isn't a clean sweep, and that's the useful finding. **Exact word matching genuinely wins when the words match** — and needs no model download. But everyday use is the second row: you write a memory in March and ask about it in July, in different words. + +**Use the `real` extra day to day**; keep hashing for CI, air-gapped machines, or when a ~90 MB download isn't welcome. Switching is safe — a store embedded by one backend is re-embedded on load by the other, never compared across incompatible vectors. Full report: [results_sentence_transformers.md](eval/results_sentence_transformers.md). + +### What the numbers don't show + +- **7 tasks and 10 gold labels, written by the same person who wrote the retriever.** One retrieval either way moves recall by ~0.07. This is an engineering check, not a production-scale claim. +- **Precision is misleading for the baseline.** Loading everything scores 0.10 simply because that's `|relevant| / |store|`. +- **MiniLM numbers aren't reproduced in CI** — they need a model download. CI regenerates and diffs the hashing results only. + +--- ## Architecture -Every agent talks to one local store. **MCP is the integration surface** — a stdio server each agent launches as a subprocess — and the CLI and Python API are thin alternatives onto the same engine. +Every agent talks to one local store. **MCP is the integration surface** — a stdio server each agent launches as a subprocess. The CLI and Python API are thin alternatives onto the same engine. ```mermaid flowchart TB @@ -171,20 +210,20 @@ flowchart TB CORE --> DISK ``` -The guards sit between the tools and the store on purpose. Tool output is compact — no scores, no timestamps — because everything a memory tool returns is paid for again in the calling agent's context window. That means the agent *cannot* judge relevance for itself, so the floor and the budget are enforced before anything is handed back. +The guards sit between the tools and the store on purpose: the caller is a model that cannot see similarity scores, so the floor and the budget are enforced before anything is handed back. A Python caller *can* see scores, so `MemoryStore.recall()` applies no floor unless asked. | Module | Responsibility | | --- | --- | -| [`mcp_server.py`](src/agent_memory/mcp_server.py) | MCP tools over stdio. Works with `mcp` 1.x and 2.x. | +| [`mcp_server.py`](src/agent_memory/mcp_server.py) | MCP tools over stdio. Supports `mcp` 1.x and 2.x. | | [`cli.py`](src/agent_memory/cli.py) | The same operations as shell commands. | -| [`store.py`](src/agent_memory/store.py) | Retrieval, budget packing, durability, the on-disk format. | +| [`store.py`](src/agent_memory/store.py) | Retrieval, budget packing, durability, on-disk format. | | [`embeddings.py`](src/agent_memory/embeddings.py) | `Embedder` protocol, two backends, per-backend relevance floor. | | [`tokens.py`](src/agent_memory/tokens.py) | Token accounting — the unit the budget is denominated in. | | [`eval/run_eval.py`](eval/run_eval.py) | Five-arm benchmark, paraphrase gap, scaling test, floor sweep. | ### The recall path -Retrieval is exact brute-force cosine over a numpy matrix — an agent's memory for one project is hundreds of entries, not millions, so this is instant and exact. FAISS or Chroma can slot in behind the same API if a store ever outgrows it. +Retrieval is exact brute-force cosine over a numpy matrix. One project's memory is hundreds of entries, not millions, so this is both instant and exact; FAISS or Chroma can slot in behind the same API if that ever changes. ```mermaid flowchart LR @@ -199,7 +238,16 @@ flowchart LR G --> H["≤ k memories,
≤ budget tokens"] ``` -Budget packing is greedy rather than all-or-nothing: an entry that would overflow the remaining budget is skipped and a smaller, lower-ranked one gets its chance. `memory_boot` spends one budget across both the handoff and the recalled memories, so the total is capped whatever the store contains. +Budget packing is greedy, not all-or-nothing: an entry that would overflow the remaining budget is skipped so a smaller, lower-ranked one gets its chance. `memory_boot` spends one budget across both the handoff and the recalled memories, so the total is capped whatever the store holds. + +**The relevance floor** exists because without it, a query about something the store knows nothing about still returns *k* memories — and tool output hides the scores, so the agent can't tell. Each backend carries its own value, since they score on different scales: + +| Backend | Floor | Chosen because | +| --- | ---: | --- | +| `HashingEmbedder` | 0.15 | No labelled recall lost, off-topic matches halved. The lowest-scoring genuinely relevant memory sits at 0.13, so the floor stays near that observed edge. | +| `SentenceTransformerEmbedder` | 0.20 | Off-topic queries are fully rejected from 0.15 and recall is flat to 0.35, so the sweep alone can't choose. Set just under 0.22 — the score of a real paraphrase against the memory that answers it. | + +Override with `AGENT_MEMORY_MIN_SCORE` or `--min-score`. The honest consequence: when a query is genuinely ambiguous, recall returns *nothing* rather than a coin flip the agent would read as fact. ### The write path @@ -224,48 +272,46 @@ sequenceDiagram CX->>L: release ``` -- **No lost updates.** A write re-reads the file under the lock, so an agent that has been idle for an hour still appends rather than overwrites. Verified with 8 processes writing concurrently. -- **No torn files.** The new contents land via `os.replace`, which is atomic — a crash mid-write leaves the previous store intact, never a truncated one. -- **Fresh reads.** A long-running MCP server reloads when the file changes on disk, so it sees another agent's writes without a restart. -- **Compact.** Embeddings are stored as base64 float16, roughly 5× smaller than JSON float lists. +- **No lost updates.** A write re-reads under the lock, so an agent idle for an hour appends rather than overwrites. Verified with 8 concurrent processes. +- **No torn files.** Contents land via `os.replace`, which is atomic — a crash mid-write leaves the previous store intact. +- **Fresh reads.** A long-running server reloads when the file changes, so it sees other agents' writes without a restart. +- **Compact.** Embeddings are base64 float16, roughly 5× smaller than JSON float lists. -### Relevance floor +--- -Without a floor, a query about something the store knows nothing about still returns *k* memories — and since tool output hides the scores, the calling agent has no way to tell noise from signal. `min_score` drops weak matches. The shipped default comes from a published sweep: +## Reference -| min_score | Labelled recall | Off-topic memories returned | -| ---: | ---: | ---: | -| 0.00 | 0.93 | 12/12 | -| 0.10 | 0.93 | 10/12 | -| **0.15** | **0.93** | **6/12** | -| 0.20 | 0.93 | 2/12 | -| 0.25 | 0.93 | 2/12 | +### CLI -The floor lives on the embedder, because the two score on different scales — a single constant would be wrong for one of them: +```bash +agent-memory write "" --type decision # save a memory +agent-memory recall "" -k 3 --budget 200 # find relevant memories +agent-memory boot "" --budget 300 # handoff + relevant memories +agent-memory handoff --done "..." --next "..." # leave a note for the next session +agent-memory list --type decision # ids, so you can fix mistakes +agent-memory update mem_0003 "" # revise a memory +agent-memory forget mem_0007 # delete a stale memory +agent-memory stats # store path, counts, embedder +``` -| Backend | Floor | Chosen because | -| --- | ---: | --- | -| `HashingEmbedder` | 0.15 | No labelled recall lost, junk halved. The lowest-scoring genuinely relevant memory sits at 0.13, so the floor stays close to that observed edge. | -| `SentenceTransformerEmbedder` | 0.20 | Off-topic queries are already fully rejected at 0.15 and recall is flat to 0.35, so the sweep alone can't choose. Set just under 0.22 — the score of a real paraphrase ("which AI model answers customer questions") against the memory that answers it. | +Global flags: `--path` (explicit store), `--global` (cross-project store), `--agent` (who is writing). -Override with `AGENT_MEMORY_MIN_SCORE` or `--min-score`. Note the honest consequence of a floor: when a query is genuinely ambiguous, recall returns *nothing* rather than a coin flip the agent would read as fact. +### Memory types -## Quickstart (60 seconds) +`project` · `decision` · `issue` · `state` · `handoff` · `worklog` · `fact` -```bash -pip install -e ".[dev]" # engine, CLI, MCP server, exact tokenizer -python eval/run_eval.py # reproduce the results above -python examples/quickstart.py - -agent-memory --agent claude-code write "Bookings are stored in UTC." --type decision -agent-memory --agent claude-code handoff --done "Fixed double emails." --next "Add rate limiting." -agent-memory boot "continue the rate limiting work" --budget 200 -agent-memory list # ids, so you can fix mistakes -agent-memory update mem_0003 "Bookings are stored in UTC; the UI converts." -agent-memory forget mem_0007 -``` +They mirror the original Markdown scaffold's files, so migration is one-to-one. + +### Environment variables -In Python: +| Variable | Default | Purpose | +| --- | --- | --- | +| `AGENT_MEMORY_PATH` | project store | Force a specific store file. | +| `AGENT_MEMORY_AGENT` | *(empty)* | Name recorded on every memory this agent writes. | +| `AGENT_MEMORY_EMBEDDER` | `auto` | `hashing` forces offline; `sentence-transformers` makes a missing model an error instead of a silent downgrade. | +| `AGENT_MEMORY_MIN_SCORE` | per-backend | Override the relevance floor. | + +### Python ```python from agent_memory import MemoryStore, default_store_path @@ -279,34 +325,24 @@ for hit in hits: print(hit.score, hit.entry.text) ``` -## Real semantic embeddings +### Migrate an existing Markdown scaffold ```bash -pip install -e ".[real]" # sentence-transformers + tiktoken +python scripts/ingest_markdown.py path/to/agent-memory/ --path .agent_memory/store.json ``` -Recommended for day-to-day use — it roughly doubles recall on paraphrased questions (0.43 → 0.79) and rejects every off-topic query in the benchmark. Costs a ~90 MB model download on first run. See [the comparison](#which-embedder-should-you-use). - -`default_embedder()` prefers the real model when installed, and warns on stderr if it falls back rather than switching silently. Force the offline embedder with `AGENT_MEMORY_EMBEDDER=hashing` (CI does this for stable numbers), or demand the real one with `AGENT_MEMORY_EMBEDDER=sentence-transformers` to turn a missing dependency into an error instead of a downgrade. - -Switching backends on an existing store is safe: vectors from a different embedder aren't comparable, so the store detects the mismatch on load and re-embeds from the stored text instead of returning meaningless scores. - -## Migrate an existing Markdown scaffold - -```bash -python scripts/ingest_markdown.py path/to/agent-memory/ --path ~/.agent_memory/store.json -``` +One memory per `##` section, with long sections split so every ingested memory stays small enough to be recalled under a budget. -One memory per `##` section, with long sections split so that every ingested memory stays small enough to be recalled under a budget. +--- -## Limits and trust boundary +## Limits -- **Memories are injected into agent context verbatim.** Anything an agent writes to the store — including text it read from a webpage, an issue tracker or a dependency — will be replayed into a *different* agent's context later, where it reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally. +- **Nothing writes memories for you.** The agent has to choose to call `memory_write` and `memory_handoff`. If it doesn't, the store stays empty. Prompt instructions or a session hook make this reliable; that isn't built in yet. +- **Memories never expire.** `state` and `worklog` entries go stale but keep being recalled as confidently as a fresh decision. Correct them with `memory_update` / `memory_forget` until decay lands. +- **Retrieval is lexical unless you install the `real` extra.** See [the comparison](#which-embedder-should-you-use). +- **Memories are replayed verbatim into other agents' context.** Anything an agent writes — including text it read from a webpage, an issue tracker or a dependency — later reads as trusted project knowledge. Don't point a shared store at untrusted input, and skim `agent-memory list` occasionally. - **The store is a local file.** No auth, no encryption, no server. It belongs next to your code, not on a shared host. -- **Retrieval is lexical unless you install the `real` extra.** The offline default matches wording, not meaning — see [the comparison](#which-embedder-should-you-use). -- **Nothing writes memories for you.** The agent has to choose to call `memory_write` and `memory_handoff`. If it doesn't, the store stays empty and the next session boots into nothing. Prompt instructions or a session hook make this reliable; that isn't built in yet. -- **Memories never expire.** `state` and `worklog` entries go stale but keep being recalled with the same authority as a fresh decision. Correct them with `memory_update`/`memory_forget` until decay lands. -- **The benchmark is small and self-authored.** It is a regression check for the engine, not evidence about your codebase. +- **The benchmark is small and self-authored.** It's a regression check for the engine, not evidence about your codebase. ## Develop @@ -315,13 +351,13 @@ pip install -e ".[dev]" pytest -q ``` -CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the committed results are stale, and separately tests the MCP server against both `mcp` 1.x and 2.x. +74 tests. CI runs the suite on Python 3.10 and 3.12, runs the evaluation and fails if the committed results are stale, and separately tests the MCP server against both `mcp` 1.x and 2.x. Tests for the optional semantic backend skip automatically unless the `real` extra is installed. ## Roadmap -- Session hooks so writing a handoff doesn't depend on the agent remembering to. +- Session hooks, so writing a handoff doesn't depend on the agent remembering to. - Recency- and type-aware ranking (decay old `state`, never drop `decision`). -- LLM-based compaction: summarize/dedup `state` and `worklog` memories, extract durable facts from a raw session transcript. +- LLM-based compaction: summarise and dedup `state`/`worklog`, extract durable facts from a session transcript. - Optional FAISS backend for large stores. ## License