Add durable HNSW snapshots and persistent DocumentStore (RAG_STATE_PATH)

[1816x]

Motivation

• The in-memory-only retrieval state made the corpus and HNSW index ephemeral across restarts, so the service must be able to persist and reload the exact retrieval state without introducing an external vector DB.
• Persistence must preserve deterministic behaviour (including RNG continuation for level assignment) and avoid unsafe/opaque serialization.
• Writes must be atomic/transactional so a failed persist does not destroy the last good snapshot and the public API must remain compatible and hardened.

Description

• Implemented a compact, dependency-free binary HNSW snapshot format with an HNSWSNP\0 magic header, explicit SNAPSHOT_VERSION, little-endian fields, checksum, bounds checks, and rejection of truncated/corrupt/invalid snapshots, and added Hnsw::to_bytes, Hnsw::from_bytes, Hnsw::save(path), and Hnsw::load(path) APIs in Rust.
• Persisted the SplitMix64 RNG state and all durable index fields (dim, metric, HnswParams, contiguous vector data, per-node per-level links, entry point/top level) while reconstructing ephemeral scratch buffers on load.
• Exposed snapshot save/load and byte APIs through the PyO3 Hnsw binding with clear IOError/ValueError mappings, and added binding-level save/load/to_bytes/from_bytes helpers.
• Added an end-to-end service-level DocumentStore snapshot container (RAGSTATE) that wraps deterministic JSON metadata (chunks, documents, _next_doc_id, index configuration, embedder identity) together with the HNSW bytes, and implemented transactional atomic commits via write-to-temp → fsync → atomic replace → parent fsync while staging changes before publishing in memory.
• Integrated opt-in persistence via RAG_STATE_PATH, added embedder compatibility and metadata validation on load, updated /stats to report persistence: { enabled, loaded, format_version }, and extended the Next.js types to include the persistence fields.
• Added comprehensive tests: Rust round-trip and corruption tests for snapshots, Python binding snapshot tests, and service-level persistence regression tests that exercise restart/load, incompatible embedder rejection, and failed-commit rollback semantics; also documented behavior and limitations in README.md.

Testing

• Rust checks and unit tests: ran cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, and cargo test --workspace, and all checks and tests completed successfully.
• Python bindings and service tests: built and installed the wheel (python -m pip install --force-reinstall ./bindings) and ran PYTHONPATH=service pytest -q bindings/tests service/tests, which passed (63 passed) with one Starlette/httpx deprecation warning only.
• End-to-end UI build: ran cd app && npm ci && npm run build && npx tsc --noEmit, which completed a successful production build and type-check; npm ci reported existing vulnerability notices but the build and type checks passed.