Fix post-merge correctness and hardening gaps

[1816x]

Motivation

• Close post-merge correctness and hardening gaps around public-facing inputs, request framing, and cross-process interactions so the service behaves predictably under malformed or concurrent requests.
• Prevent silent failures from invalid runtime configuration (non-finite thresholds, timeouts) and ensure the UI and proxy observe the same enforced limits.

Description

• Harden FastAPI request ingestion by rejecting malformed or negative Content-Length, enforcing RAG_MAX_REQUEST_BYTES before JSON parsing via BodySizeLimitMiddleware, and streaming/cancelling oversized request bodies in service/rag_service/app.py.
• Tighten client-side (Next.js) validation and proxying by enforcing request limits in app/app/lib/limits.ts, streaming JSON parsing with a byte budget, and adding RAG_SERVICE_TIMEOUT_MS-driven upstream timeouts in app/app/lib/rag.ts that map timeouts to 504.
• Validate runtime floats and timeouts via _finite_env_float and ensure _positive_env_float rejects non-finite values in service/rag_service/generation.py and service/rag_service/app.py to avoid silent misconfiguration.
• Improve store/index correctness by validating embedder cardinality, verifying insert_batch returned ids, and holding the store lock across search + metadata lookup to prevent concurrent mutation races in service/rag_service/store.py.
• Add regression tests for framed-request handling, CORS parsing, upload enable/disable behavior, input bounds, and Claude timeout mapping in service/tests/test_e2e.py and a new service/tests/test_generation_timeout.py; document the new RAG_SERVICE_TIMEOUT_MS setting in README.md and update the Next.js client types to surface limits and uploads_enabled.

Testing

• Ran Rust unit tests and engine integration: cargo test --workspace — all Rust tests passed.
• Installed Python bindings locally and ran the Python test-suite: PYTHONPATH=service pytest -q bindings/tests service/tests — 59 passed, 1 warning.
• Built the Next.js app and type-checked: cd app && npm run build and npx tsc --noEmit — build and type-check succeeded.
• Lint and formatting checks: ruff passed for modified Python code and cargo fmt/rust checks passed.
• Local commits were created on branch fix/post-merge-correctness with commit 2a967be38ed57a24e2844ed6ccc9750b8edfd741 containing the changes above.