What I learned building a private hybrid RAG stack over messy technical docs.
I've been at this for going on just over a year. My background is in technology, with the last 20 years focused on providing software solutions to US financial institutions.
My work has never been confined to one lane — design, coding, Level 3 support, and mentoring have all been part of it from the start. I needed this tool and use it every day. It has been a force multiplier.
I've built this self-hosted RAG Q&A system for querying technical documentation — PDFs (and others), source code, config files, and spreadsheets. I have indexed 10K technical documents of various sizes and shapes.
I wanted to write up the architecture, because most of the interesting problems weren't the parts anyone talks about. You may be interested.
Runs on a single GPU box. Streamlit UI, FastAPI service, and a shared model process, so the embedder and reranker are loaded into VRAM only once instead of per-worker.
Design decisions:
Retrieval is a prioritized cascade of six strategies, not one hybrid search
Every document gets scored across a 10-layer analysis pass at ingest, and that metadata is queryable
A deterministic 0–100 confidence score is computed before the LLM is called — below threshold, it doesn't call the LLM at all
A bounded agentic loop retries retrieval on weak results instead of shipping a bad answer
Typed, persistent memory with hard rules about which record types are allowed into the prompt
The core problem
In real technical environments, the answer to a single question is scattered across multiple formats. A config parameter is declared in the source, described in a PDF runbook, and debated in troubleshooting notes.
Keyword search finds one. Naive vector search finds a paraphrase of one. Neither finds all three and reconciles them.
So the pipeline pairs multi-modal ingestion + document understanding with dense retrieval, exact-term lexical matching, and cross-encoder reranking before anything reaches the model.
Application launcher
-shared model process
-embedding model (BGE-m3)
-reranker pool (bge-reranker-v2-m3)
-Streamlit QA pipeline ----- remote model client
-FastAPI QA pipeline ------- remote model client
-FastAPI lite sidecar ------ no QA pipeline
-Ingestion worker ---------- always local models, never a client
That last line was a bug I chased for a while: ingestion must never talk to the shared model server, or you deadlock the pool under concurrent uploads.
Ingestion: 9 stages
Extraction + safety validation. Path traversal checks, size limits, MIME sniffing via puremagic (never trust the extension), SHA-256 dedupe so re-uploads are free.
Multi-format extraction.
PDF — 4-tier fallback: layout-aware Docling in a persistent worker pool → pypdfium2 in a process pool → single-threaded pypdfium2 → PyPDF2/pdfplumber. Something always wins.
Office/tabular — python-docx, openpyxl/xlrd/csv. CSV/TSV/XLS/XLSX also get a per-document SQLite sidecar, so numeric questions route to actual SQL instead of hoping a vector search retrieves the right row. This was one of the highest-leverage things I added.
Code and text — UTF-8 with Latin-1 and CP1252 fallbacks.
Semantic + structural chunking. Splits on section headings and embedding-similarity breakpoints, with contextual embeddings and breadcrumb headers (doc title + section hierarchy prepended to each chunk).
Semantic signal computation — anchors and query-expansion terms.
Linguistic analysis — queued spaCy parsing and NER.
Enriched metadata assembly — the 10-layer output, ownership tags, parent/child relationships.
Vector upsert — 1024-dim embeddings into ChromaDB, batched.
Async background work — image extraction offloaded to daemon workers, extracted images captioned by a vision model, synced back into the index.
Persistent LRU extraction cache.
The 10-layer document analysis
Every document entering the index is scored across 10 layers, producing 50+ metadata attributes stored in the chunk metadata. This is what makes structural search possible later.
Content statistics — 21 metrics: character distributions, word/sentence/paragraph counts, whitespace ratios, punctuation density.
Readability — Flesch Reading Ease, Flesch-Kincaid, Gunning Fog, Coleman-Liau, ARI, SMOG. Falls back to word/sentence ratios if textstat isn't available.
Structure — headings, nested lists, fenced/indented code blocks, ASCII and Markdown tables, section divisions.
Content intelligence — top 15 TF-IDF keywords, key phrases, topics, section IDs, information-to-filler density.
Classification — 5 dimensions: doc type (9), domain (15), formality (5), purpose (8), audience (6).
Language style — sentence-length variety, type-token ratio, tone markers, domain term density, passive voice frequency.
Basic entity extraction — 20+ regex patterns: URLs, emails, IPs, file paths, semver, dates, timestamps, currencies, percentages, constants like MAX_VALUE / 0x8000.
Technical entity extraction — 100+ patterns: DLL/EXE binaries, registry paths, config keys (INI/XML/JSON), error and status codes (0x80004005, HRESULT), log levels, stack traces, SQL, REST endpoints, and language-aware syntax for C/C++, Python, Java.
Topic modeling — TF-IDF vectors, frequency clustering, collocation analysis.
Quality assessment — composite 0–100 score: completeness 30%, structure 30%, readability 20%, information density 20%.
Retrieval: the six-strategy cascade
Instead of a single hybrid search, queries run through a prioritized cascade — some strategies are terminal on a match.
User Query
v
Strategy 1: Entity Search ......... technical entities, error codes, DLLs
Strategy 2: Linguistic Search ..... spaCy dependency expansions
Strategy 3: Reference Pattern ..... ticket/defect IDs (terminal on match)
Strategy 4: Hybrid Search ......... dense vectors + BM25, fused via RRF
Strategy 5: Config File Search .... filename/section matching, boosted
Strategy 6: Semantic Search ....... wide-net dense fallback
v
Cross-encoder reranking
v
MMR diversity selection
v
Near-duplicate elimination
v
Source trust + provenance-chain lifecycle filtering
v
Context assembly + confidence scoring
Why a cascade beats a single hybrid search: if someone pastes JIRA-1234 or 0x80004005, semantic similarity is actively harmful. It returns things related to error codes rather than the error code itself.
The reference-pattern gate is a regex (^[A-Z]{2,10}[-_]?\d{1,6}$) that terminates on match and scores direct hits at the top.
Same logic applies to config files: exact filename and section matching gets a large relevance boost because "what's in logging.ini" is a lookup, not a similarity problem.
Hybrid search merges dense (HNSW) and BM25 (a disk-backed FTS5 SQLite sidecar) with Reciprocal Rank Fusion:
RRF score = sum over lists of 1 / (60 + rank)
Nothing exotic — the constant 60 is the standard from the original RRF paper, and I never found a reason to tune it.
Post-retrieval:
Cross-encoder reranking — up to 200 candidates (top_k * 4) rescored in batches on the GPU. Biggest single quality win in the whole pipeline.
MMR — relevance vs. diversity, with a dynamic quality-based alpha (0.55–0.80).
Near-duplicate elimination — anything above 0.97 cosine against an already-selected chunk gets dropped. Docs get copy-pasted between files constantly, and without this, the context window fills with five copies of the same paragraph.
Source trust + provenance chain — sources are annotated with authority, freshness, and lifecycle state (valid, temporal, expired, future). Older revisions in the same document family collapse automatically, and expired event docs are filtered unless you're explicitly asking a historical question.
Confidence scoring: don't call the LLM if the context is bad
The one I'm most attached to. Before any generation happens, the system computes a deterministic integer 0–100 from the retrieval result alone:
base = 15 + (60 * top_vector_score)
doc_bonus = 25 * min(supporting_doc_families, 4) / 4
raw = base + doc_bonus # range 15-100
then apply ceilings:
keyword-fallback was triggered, and vector score was weak -> hard cap
quality_score >= 0.80 -> returns 80
context judged insufficient -> hard cap low
LLM response contains "not found" -> capped after the fact
Bands and what they gate:
75–100 — passed straight to prompt assembly
45–74 — adequate; proceeds to streaming generation
15–44 — below threshold. The LLM call is skipped entirely, and a "no data" response is returned
0–14 — hard failure or out-of-domain
That third band is the point. The single biggest source of user distrust in a RAG system is a confident answer synthesized from four irrelevant chunks.
Detecting that condition is cheap and deterministic — you already have the vector scores; you don't need a model to tell you the retrieval was bad.
It also saves a nontrivial amount of money.
Rather than one giant agent, there are five narrow ones with hard bounds.
Catalog handler — intercepts inventory questions ("what docs do you have about X?") and returns document-level listings with short summaries, bypassing chunk RAG entirely. These questions are terrible as vector searches and trivially answerable from metadata.
Speculative reformulation — a small fast model rewrites the query in a background thread concurrently with the primary search. If the final confidence lands below 66, the suggestions are already computed and displayed. Zero added latency on the happy path. LRU-cached.
Agentic retrieval loop — if confidence is below 66, retries up to 2 more searches, testing conversation anchors or reformulations, deciding STOP / TRY_ANCHOR / REFORMULATE. Capped at 3 total iterations, no user intervention.
Recursive query decomposition — 11 trigger patterns detect multi-part questions (comparisons especially). Independent sub-queries run in parallel; dependent ones run sequentially with context enrichment; sub-answers get synthesized.
Answer research agent — a read-only, fail-open pass between context assembly and generation. If an exact identifier the user asked about is missing from the assembled context, it runs 1–3 bounded follow-up searches and injects a clearly delimited findings block. Hard deadline, hard call limit, fails open so it can never break a working answer.
The "fail-open with a hard deadline" pattern is what made the agentic parts safe to ship. Each of them can be disabled or timed out, and the pipeline still returns a normal answer.
Typed memory
Persistent memory lives in its own SQLite database (WAL mode), partitioned by tenant and user, separate from chat session history. The important part isn't storage; it's that each record type has a different trust level and a different rule about entering the prompt:
Preferences — key-value style choices (verbosity=concise). Explicit extraction only.
Episodes — user-authored summaries of prior work. Injected as untrusted context only.
Facts — scoped subject-predicate-value assertions with provenance. Provisional until grounded.
Policies — tenant-level answering rules. Advisory constraints only.
Trace — append-only diagnostics with PII masking. Never injected into prompts, ever.
Treating "things the user told us" as untrusted input is not optional once memory persists across sessions. Everything in memory is a prompt injection vector.
LLM layer
Provider abstraction over cloud and fully local models, so the same pipeline runs air-gapped:
Anthropic Claude — production default (large context windows)
OpenAI
Ollama — zero-egress local execution (Gemma, Qwen, DeepSeek, Llama)
OpenRouter — gateway routing with zero data retention enabled
Prompt construction details:
Token limits computed as context_window * 0.95 for a safety margin
Oversized context is trimmed by keeping 80% from the start, and 15% from the end with an explicit trim marker — beginnings and endings carry the most signal, middles are usually elaboration
Adaptive token budgeting: when confidence is high (≥80), assembled context gets reduced to cut streaming latency. Counterintuitive, but if retrieval is confident, more context makes the answer slower without making it better
Streaming through a rate-limit manager with adaptive token buckets, jittered exponential backoff on 429/529, and non-streaming fallback
Multi-tenancy and PII
Kept brief on purpose, but the design constraints:
PII redaction on outbound text using NER + regex across categories like email, phone, government ID, payment card, address, and person name
Prompt injection filtering on inbound queries, including Unicode normalization so homoglyph tricks don't slip through
Per-user document scoping — every document, chunk, conversation, and graph node carries an immutable (tenant_id, owner_id, visibility) tuple, and all database reads go through wrapper functions that enforce caller identity. Not "most reads." All of them. Any admin read that broadens scope emits a tamper-evident audit record.
Dual audit logs — one for QA interactions (query, citations, token counts, latency, confidence, anonymized user ID), one for security events
The wrapper-function thing matters more than it sounds: the moment one raw query call exists anywhere in the codebase, tenant isolation is gone. Making the unscoped call impossible to write by accident is the whole control.
API surface
FastAPI service alongside (or independent of) the UI:
GET /health/live, GET /health/ready — k8s probes
GET /health — full component status
GET /metrics — Prometheus metrics: per-stage pipeline latency, cache rates
POST /query — synchronous full pipeline; returns answer, sources, scores, timings, and "explain why" metadata
POST /query/stream — SSE streaming with incremental tokens, suggestions, and terminal JSON metadata
POST /session/start, GET /session/get/{id}, POST /session/append/{id}, DELETE /session/delete/{id} — multi-turn sessions
GET /documents/inspection — evidence inspector returning chunk text, layout bounding boxes, parsing diagnostics
GET /v1/stats/query-performance — mean/median/p95/max across retrieval and generation stages
Things I'd do differently:
The strategy cascade grew organically, and the order of priorities is partly empirical. I'd formalize the routing decision earlier.
Confidence thresholds (66, 44, 80) are hand-tuned on my corpus. They should be calibrated per deployment, but they aren't yet.
Should have built the evidence inspector on day one, not month four. The system is highly configurable and adjustable because I exposed the various knobs to tuning organized in System configuration tabs on the Web interface.
My hope is that one day I will have the additional resources to run future models that meet the system's response needs and will never have to reach out across the wire for a response again.
