Enterprise RAG Knowledge Assistant

A pattern I have built repeatedly: an assistant that answers staff or customer questions from an organisation's own approved documents, cites where each claim came from, respects the access rules those documents already carry, and says so plainly when the answer is not in the corpus.

The problem it solves

Organisations accumulate written knowledge faster than they can make it findable — policy documents, technical manuals, contracts, product specifications, support histories. The information exists; the retrieval cost is the problem. People ask a colleague instead, and a handful of experienced staff spend a meaningful part of each week answering questions that were already documented.

Conventional keyword search underperforms here because people ask in their own words rather than in the document's vocabulary. A general-purpose chatbot fails differently: it answers fluently from training data that has nothing to do with your policies.

How it is built

Ingestion. Source documents are parsed from whatever formats they arrive in — PDFs with tables, scanned files needing OCR, Office documents, CMS content, wikis. Content is cleaned, de-duplicated across near-identical versions, and split with structure-aware chunking so a table or clause keeps the heading that gives it meaning.

Indexing. Chunks are embedded into a vector store and indexed for keyword search in parallel. Access metadata from the source system travels with every chunk, which is what makes permission filtering possible later.

Retrieval. A query runs against both indexes. Dense vector search catches semantic matches; keyword search catches exact identifiers, part numbers and rare terms that embeddings routinely miss. Results are fused and passed through a cross-encoder re-ranker, which reorders candidates considerably more accurately than the initial similarity score.

Generation. The top passages go to the model with instructions to answer only from the supplied context and to cite each source. If nothing clears the relevance threshold, the assistant returns a "not found in the available documents" response rather than improvising.

Evaluation. A test set of real questions with known-good answers runs on every change, so a prompt edit or model upgrade produces a score rather than a feeling.

The parts that are easy to get wrong

• Permissions applied after generation. By then the model has already read restricted content and can leak it through paraphrase. Filtering has to happen at retrieval, scoped to the requesting user.
• Fixed-size chunking. Splitting on a token count severs tables from headers and clauses from their context, and no amount of prompt engineering recovers the meaning.
• Vector search alone. Ask about a specific model number or policy reference and pure semantic search will confidently return something adjacent.
• No refusal path. A system that must always answer will always answer, including when it should not.
• No evaluation set. Without one, every change is a guess and regressions are invisible until a user finds them.

Operating it

After launch the work is monitoring answer quality against the evaluation set, tracking cost per query, reviewing low-confidence and refused queries to find gaps in the source material, and re-indexing as documents change. Questions the assistant handled badly get added to the evaluation set, so real usage feeds back into the quality bar.

Cost is managed through caching, tighter retrieval producing shorter contexts, and routing simple queries to cheaper models. Tracking it per query from day one keeps the monthly figure explainable.