From 7957dcc61b3b718312aad970e41f414b46e9eefa Mon Sep 17 00:00:00 2001 From: Lakshman Patel Date: Mon, 24 Aug 2026 17:48:00 +0530 Subject: [PATCH] docs: add event-sourced memory architecture notes Design study porting Autohand Code CLI memory subsystem concepts (event log, projection, summary tree, session extraction) mapped onto yaad engine/graph/storage layers, with a three-step adoption path. --- docs/design/memory-event-sourcing-notes.md | 71 ++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/design/memory-event-sourcing-notes.md diff --git a/docs/design/memory-event-sourcing-notes.md b/docs/design/memory-event-sourcing-notes.md new file mode 100644 index 0000000..33e267a --- /dev/null +++ b/docs/design/memory-event-sourcing-notes.md @@ -0,0 +1,71 @@ +# Memory Architecture Notes — Event-Sourced Memory with Projections + +Ported design study from Autohand Code CLI's memory subsystem +(`src/memory/`, Apache-2.0, Copyright 2025 Autohand AI LLC) mapped onto +yaad's existing engine. This is a design document, not an implementation. + +## What code-cli does + +Four files carry the whole design: + +| Component | File | Responsibility | +|---|---|---| +| Event log | `MemoryEventLog.ts` | Append-only log of raw memory events (add, update, delete, access). Source of truth. | +| Projection | `MemoryProjection.ts` | Deterministic fold of the event log into the current memory view. Rebuildable at any time; never mutated directly. | +| Summary tree | `MemorySummaryTree.ts` | Hierarchical summarization: leaf memories are summarized into parent nodes so a bounded prompt can carry unbounded history. | +| Session extraction | `extractSessionMemories.ts` | End-of-session pass that mines the transcript for durable facts and emits events into the log. | +| Safety | `MemoryPathSafety.ts` | Path validation for memory file operations. | + +## Why it matters + +The event-log/projection split gives three properties yaad's current +direct-write storage does not: + +1. **Auditable history** — every memory change is an immutable event with + provenance (which session, which tool call). Today `storage/` keeps only + latest state. +2. **Deterministic replay** — the projection is a pure fold; corruption or + schema migration becomes "replay from event N" instead of data surgery. +3. **Cheap undo/branching** — time-travel is reading the log at offset K. + This composes naturally with yaad's temporal decay work (`temporal/`, + `engine/decay.go`): decay becomes a projection-time view, not a mutation. + +## Mapping onto yaad + +| code-cli concept | yaad home | Notes | +|---|---|---| +| MemoryEventLog | new `events/` package, SQLite table `memory_events(seq INTEGER PRIMARY KEY, ns, type, payload JSON, session_id, ts)` | Append-only; namespace-scoped like the rest of `engine/namespace.go`. | +| MemoryProjection | fold inside `engine/`; replaces direct fact writes from `ingest/extract.go` | Keep the current graph tables as *the* projection target so MCP tools are unchanged. | +| MemorySummaryTree | extends `compact/` | Leaf = individual facts; parents = cluster summaries. yaad's Louvain communities (`graph/community.go`) are natural tree groupings. | +| extractSessionMemories | hook in `ingest/` triggered on session end / compaction boundary | Emits events; extraction quality stays in one place. | +| MemoryPathSafety | mostly N/A | yaad stores in SQLite, not files; keep the idea as parameterized query validation in `internal/server`. | + +## Interaction with existing subsystems + +- **Decay/GC** (`temporal/`, `engine/decay.go`): decay should mark events + suppressed in the projection rather than deleting rows. Deletion loses + the ability to re-rank later; suppression is reversible and auditable. +- **Dedup** (`dedup/`): dedup moves from write-time gate to + projection-time merge — two near-duplicate events can still be folded + differently by future projections without losing either input. +- **Graph engine** (`graph/`): PageRank/community detection run on the + projected state exactly as today; nothing upstream changes. +- **Export** (`exportimport/`): `.yaadpack` gains a new section — the raw + event slice — enabling signed team packs to be merged by replay instead + of record-level conflict resolution (`conflict/`). + +## Adoption path + +1. **Shadow mode**: write events alongside current writes; projection runs + but is not authoritative. Compare snapshots in tests. +2. **Cutover**: make ingest emit events only; projection builds state. + MCP tool surface unchanged. +3. **Summaries**: add the summary tree over projected clusters; wire into + context assembly for bounded prompts. + +Step 1 is low-risk and independently valuable (audit trail). + +## References + +- Autohand Code CLI `src/memory/*` — https://github.com/autohandai/code-cli +- yaad `ARCHITECTURE.md` — engine, graph, storage layers this plugs into