Local RAG: A Fully Offline, Privacy-First Document Q&A System

The Problem: RAG Without the Cloud

Most "build your own RAG" tutorials quietly assume an OpenAI API key. The embeddings call out to an API, the generation call goes to an API, and your documents — invoices, contracts, personal notes — end up as tokens on someone else's server.

This project removes that assumption entirely. Every stage of the pipeline — document conversion, chunking, embedding, retrieval, and generation — runs on local hardware. The embedding model is a local sentence-transformer, the search engine is a local OpenSearch instance, and the LLM is served by a local Ollama process. No data leaves the machine, by construction, not by configuration flag.

The goal was a system that's still genuinely useful: hybrid search (not just vector similarity), citation-backed answers, and a usable chat UI — built with the explicit constraint of staying inspectable and dependency-light (no LangChain, no LlamaIndex).

---

Project Overview

Target Users: Anyone who wants to ask questions over their own documents — PDFs, Office files, notebooks, emails — without sending that content to a third party.

Core Capabilities:

---

System Architecture

Two independent flows share the same OpenSearch index: ingestion (Upload page → parse → chunk → embed → index) and retrieval (Chat page → hybrid search → context → Ollama). Both are plain function calls — no orchestration framework, no hidden agent loop.

---

Ingestion Pipeline: From Raw Files to Indexed Chunks

Document Conversion

src/ingestion/parsers.py is the single entry point for turning any of 16 supported file types into Markdown, using markitdown:

• PDFs are special-cased: each page is split into its own single-page PDF with PyMuPDF (fitz), then converted independently. This is what lets every chunk carry a real page number for citations, while still benefiting from markitdown's table and structure extraction.
• DOCX, PPTX, XLSX/XLS, CSV, HTML, JSON, XML, EPUB, IPYNB, ZIP, Outlook MSG have no page concept and are converted whole-document. Each PPTX slide, XLSX sheet, and ZIP entry becomes its own Markdown section, which the chunker later turns into separate heading_path breadcrumbs.
• TXT and MD pass through unchanged — they're already plain text/Markdown.

Every file is SHA-256 hashed (hash_file), and the resulting Markdown is written to data/markdown/{file_hash}.md (single-page docs) or data/markdown/{file_hash}/page_NNN.md (multi-page) — purely for inspection and debugging, so you can see exactly what the model will chunk.

Audio transcription and image captioning — both of which would require a cloud speech/LLM API — are intentionally not enabled, so .wav/.mp3/image formats aren't in the supported list.

Header-Aware Chunking

src/ingestion/chunker.py does two things in sequence:

1. _split_by_headers walks the Markdown line by line, tracking a stack of heading levels (ignoring # characters inside fenced code blocks). Every block of body text is tagged with a heading_path breadcrumb like "Intro > Background". Content before the first heading gets an empty path.
2. chunk_text packs each section's body into chunks of at most chunk_size tokens (512 by default), counted with tiktoken's cl100k_base encoding. It recursively splits on paragraphs, then sentences, and falls back to a hard token split only if a single sentence still exceeds the budget. Each new chunk carries up to chunk_overlap (64) tokens of trailing context forward from the previous one.

For sections with a heading path, the prefix "Section: {heading_path}\n\n" is prepended to every chunk, and the token budget for that section is reduced accordingly (with a _MIN_CHUNK_TOKENS = 64 floor so the prefix can never crowd out all the content).

The result is a list of Chunk objects — chunk_id, file_hash, filename, chunk_idx, page, heading_path, text — that flow straight into embedding and indexing.

Idempotent Indexing

src/ingestion/service.py's ingest_file ties it together:

def ingest_file(path: Path, force: bool = False) -> IngestResult:
    doc = parse_file(path)
    write_markdown(doc, settings.markdown_dir)

    if not force and file_already_indexed(client, settings.opensearch_index, doc.file_hash):
        return IngestResult(filename=doc.filename, file_hash=doc.file_hash,
                             chunks_indexed=0, skipped=True)

    chunks = chunk_document(doc, settings.chunk_size, settings.chunk_overlap)
    embeddings = embed_passages([c.text for c in chunks])
    indexed = bulk_upsert(client, settings.opensearch_index, chunks, embeddings)
    return IngestResult(filename=doc.filename, file_hash=doc.file_hash,
                         chunks_indexed=indexed, skipped=False)

Every chunk gets a deterministic OpenSearch document ID — {file_hash}_{chunk_idx}. Re-running ingestion on an unchanged file is a cheap no-op (skipped=True); re-running with --force simply overwrites the same IDs. There's no separate "delete old chunks" step needed for re-indexing the same file.

---

Retrieval: Hybrid Search with Reciprocal Rank Fusion

src/search/opensearch_client.py creates the index with a mapping that supports both keyword and vector search on the same documents:

"mappings": {
    "properties": {
        "text": {"type": "text"},
        "embedding": {
            "type": "knn_vector",
            "dimension": 384,
            "method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "lucene"},
        },
        "filename": {"type": "keyword"},
        "page": {"type": "integer"},
        "heading_path": {"type": "keyword"},
        "file_hash": {"type": "keyword"},
        "chunk_id": {"type": "keyword"},
        "chunk_idx": {"type": "integer"},
        "ingested_at": {"type": "date"},
    }
}

embed_query vs embed_passages

src/embeddings/embedder.py loads BAAI/bge-small-en-v1.5 once via lru_cache, with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 set before importing sentence_transformers — both a privacy requirement and a fix for a Windows crash when the library checks the Hugging Face Hub after torch has initialized its native thread pools.

Passages (chunks being indexed) are embedded as-is. Queries get the BGE instruction prefix:

_BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: "

def embed_query(text: str) -> list[float]:
    model = get_model()
    vector = model.encode(_BGE_QUERY_PREFIX + text, normalize_embeddings=True, show_progress_bar=False)
    return vector.tolist()

This asymmetric encoding is part of BGE's training recipe — applying it only to queries (never to indexed passages) measurably improves retrieval quality.

Weighted RRF Fusion

src/search/hybrid_search.py runs both searches with candidate_size = max(top_k * 4, 20) and fuses them:

def reciprocal_rank_fusion(ranked_lists, weights=None, k=60) -> list[SearchResult]:
    # score(doc) = sum over lists of weight_i / (k + rank_i), rank is 1-based
    fused_scores: dict[str, float] = {}
    for ranked_list, weight in zip(ranked_lists, weights, strict=True):
        for rank, (chunk_id, source) in enumerate(ranked_list, start=1):
            fused_scores[chunk_id] = fused_scores.get(chunk_id, 0.0) + weight / (k + rank)
    ...

A document that ranks first in both the BM25 and kNN lists wins outright. A document that appears in only one list is still included — it just scores lower. The bm25_weight / knn_weight arguments let the caller bias the fusion toward exact keyword matches or semantic similarity without changing the underlying queries — this is exactly the slider exposed in the Chat sidebar.

---

Generation: Streaming Answers with Citations

src/llm/prompts.py defines the contract the model must follow:

SYSTEM_PROMPT = (
    "You are a helpful assistant that answers questions using ONLY the context "
    "provided below.\n\n"
    "Rules:\n"
    "- Answer strictly using information from the context.\n"
    "- Cite every claim inline as [filename, p.X] (omit the page if none is given).\n"
    "- If the context does not contain enough information to answer, say so "
    "explicitly instead of guessing."
)

build_context concatenates retrieved chunks into a single block, each one labeled with format_source_label — "filename, p.X — heading_path" when both page and heading information are available — and stops adding chunks once context_token_budget (3000 tokens) would be exceeded. build_messages then assembles [system prompt, ...last 6 history turns, user message with context + question].

src/rag/pipeline.py wires retrieval and generation together:

def run_query(client, query, top_k=None, bm25_weight=1.0, knn_weight=1.0,
               filename=None, history=None):
    sources = retrieve(client, query, top_k=top_k, bm25_weight=bm25_weight,
                        knn_weight=knn_weight, filename=filename)
    return sources, generate(query, sources, history)

sources are returned synchronously (for the Sources panel), while generate returns a Python generator that the UI streams token-by-token via st.write_stream.

Ollama Client

src/llm/ollama_client.py wraps ollama.Client().chat(..., stream=True) and converts failures into a single OllamaError with an actionable message:

• Model not found (HTTP 404) → "Model 'llama3.2:3b' not found. Run: ollama pull llama3.2:3b"
• Connection failure → "Could not reach Ollama at http://127.0.0.1:11434. Run: ollama serve (...)"

If OLLAMA_NUM_GPU is set (used as 0 on low-VRAM machines), it's passed through as options={"num_gpu": ...} on every chat call to force CPU-only inference.

---

The Streamlit UI

The app is a standard Streamlit multipage app — Welcome.py plus two pages — sharing cached resources via src/ui/resources.py (@st.cache_resource for the OpenSearch client and the embedding model, so both load once per session).

Welcome.py — Status dashboard

• Checks OpenSearch reachability and cluster status, and shows indexed document/chunk counts via index_stats
• Checks whether the configured Ollama model is available via is_model_available
• Displays the active configuration (embedding model + dims, LLM + endpoint, index name, chunk size/overlap) so it's always visible which settings are live

Upload Documents page

• A multi-file uploader accepting all 16 supported extensions
• Each file is saved to data/uploads/, then passed to ingest_file with a per-file progress bar — showing "indexed N chunks" or "already indexed, skipped"
• A table of all indexed documents (filename, chunk count, ingested-at timestamp) with a delete button per row, backed by delete_by_file_hash

Chat page

• Sidebar controls: a Top-K slider, a single "Keyword ↔ Semantic" slider (0 = pure BM25, 1 = pure kNN, internally mapped to bm25_weight/knn_weight), a document filter dropdown populated from list_documents, and a "Clear conversation" button
• The main panel is a standard st.chat_message loop; each assistant turn streams its answer with st.write_stream and shows an expandable Sources panel listing each chunk's format_source_label, its RRF score (score:.4f), and a 500-character text preview
• Conversation memory: the last 6 user/assistant turns are passed as history into build_messages on every new query

---

Windows-Specific Issues & Fixes

Running a torch/sentence-transformers + OpenSearch + Ollama stack on Windows (this machine: Windows 10, i5-6200U, 16GB RAM, NVIDIA 940M with ~4GB VRAM) surfaced several environment-specific problems that aren't covered by typical tutorials:

These aren't hidden behind silent workarounds — they're documented in ARCHITECTURE.md and encoded directly as defaults/comments in src/config.py, so the reasoning survives if the hardware changes.

---

Testing & Quality

The project enforces mypy --strict on src/, ruff (rule sets E, F, I, UP, B), and a pytest suite that's required to pass before any phase counts as "done":

• test_chunker.py — header-aware chunking and breadcrumb prefixes, paragraph/sentence/overlap behavior
• test_hybrid_search.py — RRF fusion correctness: a document ranked first in both lists wins; a document present in only one list is still returned; weights shift the ranking; empty inputs return empty results
• test_pipeline.py — context token-budgeting, message construction (system prompt + history + question), and run_query wiring with mocked search/chat
• test_parsers.py — markitdown conversion and Markdown persistence to data/markdown/

uv run pytest                          # tests
uv run ruff check .                    # lint
uv run mypy src/                       # strict type check
uv run python scripts/health_check.py  # OpenSearch + Ollama reachability

scripts/health_check.py is a standalone CLI that checks both OpenSearch (/_cluster/health) and Ollama (/api/tags, verifying the configured model is present) and prints actionable OK/FAIL lines — useful before opening the Streamlit app.

scripts/ingest_folder.py bulk-ingests every supported file under a folder recursively, with --force to re-index unchanged files and --reset to drop the OpenSearch index first and rebuild from scratch (needed after a chunking or mapping change).

---

Configuration Reference

Everything is driven by pydantic-settings (src/config.py) and a .env file — no hardcoded values:

---

Remaining / Stretch Work

The build plan's core phases (scaffolding, ingestion, embeddings + index, hybrid search, RAG pipeline, Streamlit UI) are all complete and tested. What's explicitly still open:

• Cross-encoder re-ranker (e.g. bge-reranker-base) as a toggle after RRF fusion, before context building
• Query rewriting for follow-ups — chat history is currently passed raw (last 6 messages); rewriting follow-up questions into standalone queries would improve retrieval in multi-turn conversations
• Eval script — a fixed question set against a known corpus, reporting retrieval hit-rate (precision@k / MRR)
• Documenting expected latency in the UI — given CPU-only generation (~4 tok/s on this hardware), a caption near the chat input would make it clearer the app is working, just slow

---

Key Learnings

Header-aware chunking pays for itself at citation time. Prefixing each chunk with Section: H1 > H2 before embedding doesn't just help the retriever — it makes citations like filename, p.4 — Pricing > Refund Policy meaningfully more useful than filename, p.4 alone.

Deterministic IDs make ingestion boring, which is the goal. {file_hash}_{chunk_idx} means uploading the same file twice, or re-running the bulk ingestion script over a folder, is always safe. No separate "is this already indexed" bookkeeping beyond a single count query on file_hash.

RRF fusion is a cheap, effective default. Two independent ranked lists — one lexical, one semantic — combined with weight / (k + rank) gets most of the benefit of a hybrid retriever without needing a learned reranker. The reranker stays on the "remaining work" list precisely because RRF already performs well enough to ship.

"Fully local" surfaces OS-level problems that cloud APIs hide. The localhost → 127.0.0.1 crash and the GPU/Vulkan crashes only exist because everything — embeddings and generation — runs as native processes on this machine. A cloud-API version of this project would never have hit either bug, and would also never have needed to think about them.

Streaming is what makes CPU-only generation tolerable. At ~4 tokens/second, a blocking call would feel broken. st.write_stream over the Ollama streaming response means the first words appear almost immediately, and the perceived latency is dramatically lower than the actual generation time.

---

Conclusion

This project is a working answer to "do I need a cloud LLM API to build RAG?" — no. A local embedding model, a local hybrid search index, and a local LLM, wired together with explicit, inspectable Python (no LangChain, no LlamaIndex), produce a system that converts sixteen file types, chunks them with heading-aware breadcrumbs, retrieves with BM25 + kNN + RRF, and streams cited answers — all without a single byte leaving the machine.

The constraints that made this hard — Windows networking quirks, a 4GB-VRAM GPU that can't run an 8B model — are also what made the architecture honest: every workaround is a documented default, not a hidden hack, and the test suite (pytest, ruff, mypy --strict) holds the whole thing together as it evolves.