feat: add conversation materialization

docs/.gitignore

@@ -6,3 +6,4 @@
 *
 !.gitignore
 !golden-principles.md
+!conversation-materialization.md

docs/conversation-materialization.md

@@ -0,0 +1,172 @@
+# Conversation materialization
+
+## Purpose
+
+This package converts caller-supplied closed conversation windows into immutable source documents.
+
+The caller remains the source of truth. The source documents are a read-only projection. Each turn keeps its conversation identity, speaker, authority, time, text, and source anchor.
+
+The package has no source-specific rules. The caller selects the input window, cleaning hooks, limits, and authority values.
+
+## Data model
+
+`ConversationTurn` contains:
+
+- conversation ID
+- turn ID
+- speaker
+- authority
+- timestamp
+- text
+- source anchor
+
+Authority values are `first_party`, `assistant`, and `document`. These values come from public OpenKB claims rules.
+
+`ConversationWindow` contains a caller-defined window ID, source ID, creation time, closed state, and ordered turns. OpenKB does not calculate a delay. OpenKB does not use a local time zone.
+
+Each timestamp must use ISO format and include a UTC offset. OpenKB normalizes timestamps to UTC for sorting. When two normalized times are equal, OpenKB keeps the caller order. Compile selection normalizes each creation time to UTC before it applies the source, window, segment, and path tie breaks.
+
+## Cleaning and limits
+
+`clean_text()` normalizes line endings, removes NUL characters, and runs pure caller hooks in order.
+
+The default materializer fails when one turn exceeds `max_turn_chars`. It also fails when one complete rendered source document exceeds `max_document_chars`. The document limit includes headers, evidence fields, turn fields, text, and line breaks. The materializer never truncates by default.
+
+A caller can use `truncate_with_digest()` as an explicit choice. The result keeps the beginning and end. It records the original character count and the SHA-256 digest of the omitted middle.
+
+The splitter stops only at turn boundaries. It preserves UTC order and caller order for equal UTC times. After each split, it verifies that the flattened turn identities exactly match the ordered input identities. A coverage error stops materialization before a file or manifest mutation.
+
+## Closed-window identity
+
+The manifest stores one fingerprint for the canonical closed-window bytes. It also stores one fingerprint for the active document set.
+
+A normal replay must match both fingerprints. A change to a turn, creation time, rendered bytes, segment count, segment path, or document hash fails. An empty closed window also gets a fingerprint and a window record. A later non-empty replay of that identity fails.
+
+Only `recover_uncompiled()` can replace the active document set. The canonical closed-window fingerprint must still match.
+
+Path tokens contain a readable prefix and the full SHA-256 digest of the opaque ID. Different IDs cannot collide only because their readable forms are the same.
+
+## Immutable documents and manifest rules
+
+Each materialized document contains:
+
+- window ID
+- source ID
+- creation time
+- segment index and count
+- evidence rules
+- turn ID
+- speaker
+- authority
+- timestamp
+- source anchor
+- text
+
+The manifest stores relative paths and lowercase hexadecimal SHA-256 values. Each document record also stores the full `(conversation_id, turn_id)` identities.
+
+Manifest loading rejects:
+
+- malformed timestamps
+- malformed hashes
+- corrupt UTF-8 manifest text
+- duplicate document paths
+- duplicate window records
+- conflicting window document sets
+- unknown completed hashes
+- receipts for incomplete hashes
+
+A replay with the same bytes is a no-op. A replay with different bytes fails. `verify_manifest()` checks every active file.
+
+`select_compile_inputs()` sorts by normalized UTC creation time, source ID, window ID, segment index, and relative path. It skips completed hashes and returns the caller limit. The optional `filename` value selects a non-default manifest. The function does not call a compiler.
+
+`check_compile_budget()` validates positive values and requires:
+
+`batch_limit * document_timeout < scheduler_guard`
+
+## Compile receipts
+
+`mark_compiled()` can store optional receipt metadata for each content hash. The allowed metadata is:
+
+- compile time
+- duration in seconds
+- result SHA-256
+
+The manifest JSON is deterministic. Receipt values do not contain source-specific fields.
+
+## Recovery
+
+`recover_uncompiled()` uses a smaller document limit. It selects turns with the full `(conversation_id, turn_id)` identity. Equal turn IDs from different conversations cannot cross-select.
+
+Recovery does not replace a compiled document. It writes the new recovery documents first. It then writes the new manifest. After both writes are durable, it removes the replaced uncompiled files from the active source tree. The manifest and compile selector cannot return the old files.
+
+If a pending turn cannot fit the recovery limit, recovery stops before mutation. The pending source file and manifest remain unchanged.
+
+Residual risk: A process stop after a new recovery file write and before the manifest write can leave an untracked recovery file. The file is not selectable because it is not in the manifest. Automatic cleanup of this crash orphan is outside this change.
+
+Recovery does not rewrite wiki output.
+
+## Shared locking
+
+Manifest read-modify-write functions use the shared OpenKB mutation lock. They use unlocked helpers only while the public function holds that lock.
+
+`apply_verified_claims()` uses the same lock rule. The function reads, merges, and writes one page while it holds the lock. Concurrent claim updates cannot overwrite each other.
+
+## Verified claims
+
+`apply_verified_claims()` accepts caller-verified claim objects. It calls `openkb.claims.merge_claims()`. It uses the shared frontmatter helpers and atomic writer.
+
+The function keeps the page body and claim history. Repeated input is idempotent. Public claim rules apply:
+
+- A first-party claim can supersede an older assistant proposal when the claim rules permit it.
+- An assistant claim cannot remain validated.
+- An assistant claim cannot supersede a first-party claim.
+
+## Deployment examples
+
+The following values are aggregate deployment examples. They are not code defaults:
+
+- One deployment processed 280 conversations into 537 documents.
+- Initial materialization used a 240,000-character document limit.
+- Recovery used a 40,000-character document limit.
+- A later live materializer used a 120,000-character document limit and a 40,000-character turn limit.
+- One document exceeded 600 seconds and completed in 885 seconds.
+- The validated operating point used a 1,200-second document timeout, a two-document batch, and a 3,600-second scheduler guard.
+
+These values show why callers must set limits for their own workload.
+
+## Relation to public OpenKB work
+
+This change is stacked on PR #223. It uses the claims API from that change.
+
+This change does not depend on PR #179. It does not copy the ingest-bundle pipeline. A future PR #179 adapter can call this materializer.
+
+The package does not add live input, source-specific filters, source data, scheduler wiring, or model calls.
+
+## Synthetic example
+
+```python
+window = ConversationWindow(
+    window_id="example-window",
+    source_id="example-source",
+    created_at="2026-01-01T00:00:00Z",
+    closed=True,
+    turns=(
+        ConversationTurn(
+            conversation_id="example-conversation",
+            turn_id="turn-1",
+            speaker="first-party",
+            authority="first_party",
+            timestamp="2026-01-01T00:00:00Z",
+            text="A synthetic value.",
+            source_anchor="example:turn-1",
+        ),
+    ),
+)
+
+materialize_window(
+    window,
+    output_dir,
+    max_turn_chars=40000,
+    max_document_chars=120000,
+)
+```

openkb/conversation/__init__.py

@@ -0,0 +1,65 @@
+"""Transport-neutral closed conversation materialization."""
+
+from openkb.conversation.authority import apply_verified_claims
+from openkb.conversation.manifest import (
+    Manifest,
+    ManifestConflictError,
+    ManifestError,
+    check_compile_budget,
+    load_manifest,
+    mark_compiled,
+    register_documents,
+    save_manifest,
+    select_compile_inputs,
+    verify_manifest,
+)
+from openkb.conversation.materialize import (
+    DuplicateTurnError,
+    OversizeTurnError,
+    clean_text,
+    deduplicate_turns,
+    materialize_window,
+    recover_uncompiled,
+    render_document,
+    split_turns,
+    truncate_with_digest,
+)
+from openkb.conversation.models import (
+    CompileReceipt,
+    CompileReceiptMetadata,
+    ConversationTurn,
+    ConversationWindow,
+    DocumentRef,
+    TruncationResult,
+    WindowRecord,
+)
+
+__all__ = [
+    "CompileReceipt",
+    "CompileReceiptMetadata",
+    "ConversationTurn",
+    "ConversationWindow",
+    "DocumentRef",
+    "TruncationResult",
+    "WindowRecord",
+    "DuplicateTurnError",
+    "OversizeTurnError",
+    "Manifest",
+    "ManifestError",
+    "ManifestConflictError",
+    "apply_verified_claims",
+    "check_compile_budget",
+    "clean_text",
+    "deduplicate_turns",
+    "load_manifest",
+    "mark_compiled",
+    "materialize_window",
+    "recover_uncompiled",
+    "register_documents",
+    "render_document",
+    "save_manifest",
+    "select_compile_inputs",
+    "split_turns",
+    "truncate_with_digest",
+    "verify_manifest",
+]

openkb/conversation/authority.py

@@ -0,0 +1,53 @@
+"""Apply caller-verified claims through the public claims API."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Mapping, Sequence
+
+from openkb import frontmatter
+from openkb.claims import merge_claims
+from openkb.locks import atomic_write_text, kb_ingest_lock
+
+
+def _default_openkb_dir(page_path: Path) -> Path:
+    for parent in page_path.parents:
+        candidate = parent / ".openkb"
+        if candidate.is_dir():
+            return candidate
+    return page_path.parent / ".openkb"
+
+
+def _apply_verified_claims_unlocked(
+    path: Path,
+    claims: Sequence[Mapping[str, object]],
+) -> list[dict]:
+    text = path.read_text(encoding="utf-8")
+    parts = frontmatter.split(text)
+    if parts is None:
+        raise ValueError("page must contain valid frontmatter")
+    fm_block, body = parts
+    existing = frontmatter.parse(text).get("claims")
+    merged = merge_claims(existing if isinstance(existing, list) else [], list(claims))
+    if merged:
+        fm_block = frontmatter.set_json_line(fm_block, "claims", merged)
+    else:
+        fm_block = frontmatter.drop_line(fm_block, "claims")
+    atomic_write_text(path, fm_block + body)
+    return merged
+
+
+def apply_verified_claims(
+    page_path: Path | str,
+    claims: Sequence[Mapping[str, object]],
+    *,
+    openkb_dir: Path | str | None = None,
+) -> list[dict]:
+    """Merge verified claims under the shared mutation lock."""
+    path = Path(page_path).resolve()
+    lock_dir = Path(openkb_dir) if openkb_dir is not None else _default_openkb_dir(path)
+    with kb_ingest_lock(lock_dir):
+        return _apply_verified_claims_unlocked(path, claims)
+
+
+apply_claims = apply_verified_claims