Upload a PDF or TXT document, ask questions about it in plain English, and get an answer generated only from the document's own contents — returned together with the exact passages and page numbers it was drawn from.
Measured on the bundled sample document: 0% hallucination rate, 100% retrieval recall@4, ~$0.004 per question. See Evaluation.
PDF / TXT
│
▼
extract text per page ──► clean ──► split into overlapping chunks
│
▼
embed locally (all-MiniLM-L6-v2)
│
▼
store in ChromaDB
(vector + text + page number)
question
│
├──► embed ──► vector search ────┐
│ ├──► fuse (RRF) ──► top-k chunks ──► Claude ──► answer + sources
└──► tokenize ──► BM25 search ───┘
The retrieved chunks are the only thing the model is allowed to answer from. If they do not contain the answer, the system says so instead of inventing one.
Answer with sources — every passage the answer was built from, with its page number and relevance score.
Refusing rather than inventing — the funding figure is not in the document, so the system says so instead of producing a plausible number.
Follow-up questions — "Who proposed it?" is rewritten into a standalone question before retrieval runs, and the rewrite is shown rather than applied silently.
The shortest path — no Python, no virtualenv, no model download:
git clone https://github.com/DevnilMaster/RAG-Document-QA-System.git
cd RAG-Document-QA-System
cp .env.example .env # then add your ANTHROPIC_API_KEY
docker compose up --buildThe first build takes a few minutes (it installs PyTorch and bakes the embedding model into the image so the container needs no network access to Hugging Face at runtime). Subsequent starts are immediate.
docker compose down # stop
docker compose down -v # stop and delete the indexed documentAssumes Python 3.10+ and git.
git clone https://github.com/DevnilMaster/RAG-Document-QA-System.git
cd RAG-Document-QA-System
python -m venv .venv# Windows (PowerShell)
.venv\Scripts\Activate.ps1# macOS / Linux
source .venv/bin/activatepip install -r backend/requirements.txt
cp .env.example .env # then add your ANTHROPIC_API_KEY
uvicorn backend.main:app --reloadOpen http://127.0.0.1:8000 · interactive API docs at /docs
The first upload downloads the embedding model (
all-MiniLM-L6-v2, ~90 MB) from Hugging Face and caches it locally. Once only.
Set RAG_DRY_RUN=1 in .env. Everything except the final generation step runs
normally — the document is extracted, cleaned, chunked, embedded and searched,
and the real retrieved passages are shown — with a placeholder in place of the
model's answer. It verifies the retrieval half of the pipeline with no key and no
cost.
samples/sample-research-paper.pdf is a short fictional paper included for
testing. Facts are spread deliberately across its three pages, so page citations
can be checked at a glance:
| Ask this | Expect a citation to |
|---|---|
| "What kappa score did the model achieve?" | Page 3 |
| "How many fundus photographs were in the dataset?" | Page 2 |
| "Why did earlier handcrafted approaches fail?" | Page 1 |
| "What is the capital city of France?" | a refusal — it is not in the document |
| Variable | Required | Default | Purpose |
|---|---|---|---|
ANTHROPIC_API_KEY |
Yes (unless RAG_DRY_RUN=1) |
— | Key for the Claude API. Get one at console.anthropic.com. |
RAG_DRY_RUN |
No | 0 |
1 skips the LLM call and returns a placeholder. Retrieval still runs in full. |
.env is gitignored; .env.example is the committed template. Docker Compose
reads the same .env and passes both values into the container — the key is
never baked into an image layer.
- Upload — choose a PDF or TXT file and press Upload. The status line reports how many chunks were indexed and from how many pages.
- Ask — type a question and press Ask (or hit Enter).
- Check the answer — the answer appears first, followed by the passages it was grounded in, each labelled with its page number and a relevance score.
Uploading a new document replaces the previous one.
POST /upload — index a document
Request: multipart/form-data with a single file field.
curl -F "file=@samples/sample-research-paper.pdf" http://127.0.0.1:8000/upload{
"filename": "sample-research-paper.pdf",
"pages": 3,
"chunk_count": 11,
"message": "Indexed 11 chunks from 3 page(s). You can ask questions now."
}POST /ask — ask a question
curl -X POST http://127.0.0.1:8000/ask \
-H "Content-Type: application/json" \
-d '{"question": "What are the three objectives of this study?"}'mode is optional and defaults to hybrid; semantic and keyword run a
single retrieval strategy, which is what makes the three directly comparable on
the same question.
{
"answer": "The three objectives are:\n\n1. To design a convolutional architecture that classifies retinal fundus images into five severity grades (Page 2).\n2. To constrain the model to fewer than twelve million parameters (Page 2).\n3. To quantify how much of the remaining accuracy gap is attributable to image acquisition variability (Page 2).",
"sources": [
{
"page": 2,
"snippet": "2. OBJECTIVES The objective of this study is threefold...",
"chunk_id": "chunk-0004",
"score": 0.032266
}
],
"mode": "hybrid",
"resolved_question": null
}score means different things per mode — cosine similarity for semantic, the
raw BM25 score for keyword, the fused RRF score for hybrid — so compare
scores within a mode, never across them.
POST /ask/stream — the same answer, streamed
Server-Sent Events. Same request body as /ask. Retrieval finishes before the
stream opens, so the sources arrive first and the UI can show the evidence while
the answer is still being written.
curl -N -X POST http://127.0.0.1:8000/ask/stream \
-H "Content-Type: application/json" \
-d '{"question": "What limitations do the authors acknowledge?"}'event: sources
data: {"sources": [...], "mode": "hybrid", "resolved_question": null}
event: delta
data: {"text": "The authors acknowledge three limitations: (1) the dataset..."}
event: done
data: {}
Failures detectable before generation starts (no document indexed, empty
question) still return a normal HTTP error code. Once the stream is open the
status code has already been sent, so a later failure arrives as an error
event instead.
The browser UI uses this endpoint; /ask is the non-streaming equivalent and is
what the evaluation script calls.
POST /api/reset — forget the conversation
Clears follow-up context without re-uploading the document.
curl -X POST http://127.0.0.1:8000/api/resetGET /api/status — what is currently indexed
{
"indexed": true,
"filename": "sample-research-paper.pdf",
"chunk_count": 11,
"model": "claude-sonnet-5",
"dry_run": false,
"conversation_turns": 2
}python evaluation/run_eval.pyevaluation/questions.json holds 13 questions against the sample document: ten
whose answer sits on a known page, and three that the document cannot answer at
all — two of them deliberately plausible for a paper of this kind, which is
exactly where a model is most tempted to invent something.
Results on the bundled sample (claude-sonnet-5, top_k=4, hybrid retrieval):
| Metric | Value |
|---|---|
| Retrieval recall@4 | 100% (10/10) |
| Retrieval precision@1 | 100% (10/10) |
| Answer keyword coverage | 100% (10/10) |
| Correct refusal rate | 100% (3/3) |
| Hallucination rate | 0% (0/3) |
| False refusal rate | 0% (0/10) |
| Retrieval latency | median ~50 ms |
| Generation latency | median 1.5–2.7 s across runs (network-dominated) |
| Cost | $0.0044 per question |
The full run writes evaluation/results.md with every question, its retrieved
pages, and the generated answer.
Retrieval strategy comparison — same questions, same document, retrieval only (so it costs nothing to regenerate):
| Mode | Recall@4 | Top-1 | Returned nothing |
|---|---|---|---|
hybrid |
10/10 | 10/10 | 0/10 |
semantic |
10/10 | 8/10 | 0/10 |
keyword |
10/10 | 9/10 | 0/10 |
Q13 is the case worth reading. Asked "What are the three objectives of this study?", semantic-only retrieval never returns the chunk containing the objectives, and the model — correctly, given what it was handed — refuses. BM25 finds that chunk at rank 1 because the heading literally reads "OBJECTIVES", and hybrid recovers the full three-part answer. A false refusal caused by retrieval, not by generation, and a concrete reason for the extra machinery.
RAG-Document-QA-System/
├── backend/
│ ├── main.py # FastAPI app: /upload, /ask, /ask/stream, /api/*
│ ├── document_processor.py # extraction, cleaning, chunking
│ ├── embeddings.py # sentence-transformers wrapper
│ ├── vector_store.py # ChromaDB wrapper
│ ├── keyword_search.py # BM25, implemented directly
│ ├── retrieval.py # semantic / keyword / hybrid + RRF fusion
│ ├── conversation.py # follow-up detection + short-term memory
│ ├── llm.py # Claude call, streaming, follow-up rewriting
│ ├── prompts.py # the prompt, isolated for inspection
│ ├── config.py # every tunable value, in one place
│ └── requirements.txt
├── frontend/ # index.html, app.js, style.css
├── evaluation/
│ ├── questions.json # ground truth
│ ├── run_eval.py # metrics + mode comparison
│ └── results.md # generated
├── samples/ # a sample document to test with
├── data/ # local ChromaDB store (contents gitignored)
├── Dockerfile
├── docker-compose.yml
├── .env.example
└── README.md
800 characters is roughly 150–200 tokens — large enough to hold a complete thought, small enough that four of them still leave the prompt cheap. The 150-character overlap means a fact sitting on a chunk boundary is still retrievable: it appears complete in at least one chunk instead of being cut in half by both.
Chunking happens within each page, so every chunk belongs to exactly one page and the page number cited as evidence is always exact. The trade-off is that a sentence spanning a page break gets split; since page-accurate attribution is the whole point of the system, exactness was the priority.
The splitter is a small hand-written recursive splitter rather than LangChain's
RecursiveCharacterTextSplitter. It tries the most natural boundary available
first — paragraph, then line, then sentence, then word — and only falls back to a
finer one for parts still too long. Writing it directly (~40 lines) was preferred
over adding LangChain's dependency tree for one function.
Every chunk of every uploaded document has to be embedded, so a hosted embedding API would mean a per-upload cost and a second key to manage. This model is 22M parameters, produces 384-dimensional vectors, and embeds a query in ~15 ms on CPU — it never shows up next to the model's network round-trip.
requirements.txt pulls the CPU-only PyTorch wheel via an extra index. The
default build bundles ~2 GB of CUDA libraries this project never uses, and a
container has no GPU to use them with anyway.
A larger model (bge-large, e5-large) would retrieve somewhat better at
roughly 10× the size and embedding time. Retrieval quality was not the
bottleneck here.
ChromaDB stores the vector, the chunk text, and the metadata together and persists to disk with no server to run. With FAISS the index holds vectors only, so chunk text and page numbers live in a parallel structure kept in sync by hand. Since page-accurate attribution matters here, keeping metadata attached to the vector removes a whole category of bug.
Distance is cosine, which measures the angle between vectors and ignores their length. That matters for text: a long passage and a short question about it should count as similar because they point the same way, not be pushed apart because one has more words.
Semantic search has a specific blind spot. An embedding compresses a passage into
384 numbers, and exact tokens — 0.847, AdamW, a section heading — do not
survive that compression intact. BM25 is the mirror image: excellent on exact
tokens, blind to paraphrase.
They are fused with Reciprocal Rank Fusion. The obvious alternative, adding the two scores, does not work off the shelf: cosine similarity lives in roughly 0–1 while BM25 is unbounded and corpus-dependent, so any fixed weighting is really a constant tuned to one document. RRF discards scores and uses only rank position, which is directly comparable between the two lists and needs no tuning. The measured effect is in the table above.
k=4 keeps the prompt at roughly 1,300 input tokens. Raising it improves recall
on questions whose answer is spread across a document, at a linear cost in tokens
and a real risk of diluting context with weakly-related passages. k is a single
constant in config.py.
The task is extraction from supplied context, not open-ended reasoning, so the model is configured for exactly that:
thinking: {"type": "disabled"}— extended thinking is on by default on Claude Sonnet 5 and would spend tokens deliberating over a task that does not need it.effort: "low"— the same reasoning applied to overall token spend.max_tokens: 600— a hard ceiling; an unbounded response is the easiest way to spend API credit by accident.max_retries: 1(SDK default is 2) — a persistent failure should surface quickly rather than be billed three times.
Measured result: $0.0044 per question, with generation latency dominated by the network round-trip rather than by anything on this side.
Anthropic is the only provider, with no fallback backend. A fallback path doubles the code that has to be understood and tested, and it hides configuration mistakes: with one provider, a missing key fails immediately and says why.
The prompt lives in backend/prompts.py, on its own, so it can be read and
argued about rather than buried inside an API call. Four things do the work:
- The system prompt states that the excerpts are the complete set of available information, not a helpful addition to what the model already knows.
- It supplies the exact sentence to emit when the answer is absent. Declining is much easier for a model handed the words to decline with.
- Each excerpt is labelled with its page number, so citing a source is copying a label rather than recalling a fact.
- When retrieval returns nothing, the API is not called at all — the "not enough information" answer is returned directly. An answer already determined should not cost a request.
Measured: 3/3 correct refusals on unanswerable questions, 0/10 false refusals on answerable ones. That is a real number on a small set, not a guarantee.
Given the exchange
Q: What is this research about?
A: Automated retinal disease classification using RetinaNet-DR...
Q: Who proposed it?
the obvious fix — hand the model the chat history — does not work, because the problem is upstream of the model. "Who proposed it?" embeds to a vector about proposing and things; nothing in it points at retinal imaging, so the vector search returns the wrong chunks and no amount of history helps the model reason over passages it was never given.
So the follow-up is rewritten into a standalone question before retrieval: "Who proposed it?" becomes "Who proposed the RetinaNet-DR approach for automated diabetic retinopathy screening?", which the index can actually match.
The rewrite costs an API call, so it is gated behind a cheap regex heuristic — does the question contain a referring word, or is it very short? A self-contained question skips the rewrite entirely and costs nothing extra. The heuristic is deliberately biased towards false positives: rewriting a question that did not need it returns something equivalent and wastes a fraction of a cent, while missing a real follow-up produces a wrong answer.
The rewrite is shown in the UI rather than applied silently, because a system that quietly answers a different question than the one typed is hard to trust.
The browser UI reads /ask/stream, which sends the retrieved passages as the
first event and then the answer in pieces. Retrieval is already finished by the
time generation starts, so there is no reason to make the reader wait for the
whole response before showing them the evidence — most of the perceived
improvement comes from that reordering rather than from the token-by-token text.
EventSource would handle the framing, but it only supports GET and the question
belongs in a request body, so the SSE records are parsed from the fetch stream
directly (~25 lines in app.js). Network chunks split records arbitrarily, so
anything after the last blank-line separator is held back until more arrives.
/ask remains as the non-streaming equivalent — simpler for API consumers, and
it is what the evaluation script calls, since a script has nothing to gain from
partial output.
All the interesting work here is backend and retrieval, and the UI exists to
demonstrate that the pipeline works. A zero-build frontend is servable as static
files by FastAPI itself: no npm toolchain, no build step, no second server or
CORS configuration, and a single-stage Dockerfile instead of a multi-stage one.
That effort went into retrieval quality and evaluation instead.
Single-stage, because there is no frontend build to stage separately.
requirements.txt is copied and installed before the application code so that
editing a .py file reuses the cached PyTorch install instead of re-downloading
it. The embedding model is baked in at build time, so the container needs no
network access to Hugging Face at runtime. The app runs as a non-root user, and
the ChromaDB store lives in a named volume so the index survives
docker compose down.
The image is ~2.75 GB, essentially all of it PyTorch, ChromaDB, and their transitive scientific-Python dependencies. The CPU-only PyTorch wheel already removes about 2 GB of unused CUDA libraries from that figure.
- One document at a time. Uploading a new file replaces the previous index. Supporting several would mean a collection or metadata filter per document plus a document picker — mechanically simple, but out of scope here.
- Scanned PDFs do not work. Extraction reads a PDF's text layer. A scanned page has none, so the upload fails with a message saying exactly that. Handling it would require an OCR pass.
- Chunks do not span page boundaries. A sentence continuing across a page break is split between two chunks — a deliberate trade for exact page attribution that can cost recall on a question whose answer sits on the break.
- The evaluation measures recall at page level, not chunk level. A page number is what gets cited, so that is the unit reported — but it is coarser than the chunk the answer actually needs. Q13 is the proof: semantic-only retrieval scores as a page-level hit there and still produces a false refusal. Read page-level recall as an upper bound.
- The evaluation set is small and single-document. Thirteen questions against one synthetic paper. It is enough to catch regressions and to compare retrieval strategies against each other; it is not enough to claim a general accuracy figure.
- Chunk size, overlap and
kwere chosen by reasoning, not tuned. The harness to tune them exists — change a constant inconfig.pyand re-run the evaluation — but no sweep was run. - No authentication or rate limiting. Anyone who can reach the port can use the API and spend the configured key's credit.
- Conversation memory is process-global and in-memory. There is one active
document and no authentication, so there is exactly one conversation. It holds
the last two exchanges and is cleared on upload or via
POST /api/reset. A multi-user version would key it by session and persist it — a different design, not a bigger version of this one. - The follow-up heuristic is a regex, not a classifier. It catches referring words and very short questions. A follow-up phrased without either ("and the authors of the second paper") would be missed and searched as-is.
- Complex PDF layouts degrade extraction. Multi-column papers, tables, and figure captions come out of the text layer in a reading order that is sometimes wrong, which shows up as slightly incoherent chunks.
- English-centric embeddings.
all-MiniLM-L6-v2is trained predominantly on English; retrieval quality in other languages will be noticeably worse.