RAG Knowledge Assistants

A retrieval-augmented generation assistant answers questions using your documents rather than whatever the model absorbed during training. Done properly it cites its sources, respects who is allowed to see what, and declines to answer when the answer is not in the corpus. Done carelessly it is a confident liar with your logo on it.

What it is actually for

The pattern earns its keep wherever people repeatedly ask questions whose answers already exist in writing but are slow to find: internal policy and HR handbooks, technical documentation, contracts, product knowledge bases, regulatory or compliance material, research archives, and support histories.

The value is not that a language model is involved. It is that someone gets a correct, sourced answer in ten seconds instead of pinging a colleague and waiting until tomorrow — and that the colleague stops answering the same question twenty times a month.

Retrieval is the hard part, not the model

Teams tend to spend their energy on prompts and model selection. In practice, answer quality is dominated by whether the right passage made it into the context window at all. That is a retrieval engineering problem:

• Ingestion and parsing — PDFs with tables, scanned documents needing OCR, Office files, CMS content, wikis and ticket histories all fail in different ways. This is usually the messiest part of the project.
• Chunking — splitting on a fixed token count severs tables and clauses from their headings. Structure-aware chunking that keeps a section with its context consistently outperforms it.
• Hybrid retrieval — dense vector search finds semantic matches but misses exact identifiers, part numbers and rare terms. Combining it with keyword search (BM25) and fusing the rankings covers both.
• Re-ranking — a cross-encoder pass over the top candidates reorders them far more accurately than the initial vector similarity did.
• Knowing when to stop — if nothing clears the relevance threshold, the correct output is "I could not find this in the available documents", not a fluent guess.

Permissions are designed in, not bolted on

If a document is restricted to the finance team in your CMS or file store, it must stay restricted in the assistant. The reliable way to achieve this is to carry access metadata through ingestion and filter at retrieval time, scoped to the requesting user — so restricted content is never a candidate for their answer in the first place.

Filtering after generation does not work: by then the model has already read the content and can leak it through paraphrase. This is the most common serious flaw I find in pilots built without platform experience, and it is far harder to retrofit than to design in.

Evaluation, or you are guessing

Every engagement includes an evaluation set: real questions from real users, paired with known-good answers and the source passages they should come from. Each change — a prompt edit, a chunking tweak, a model upgrade, a new batch of documents — is scored against it automatically.

Without this you cannot tell improvement from regression, and "it feels better" becomes the release criterion. The set is also what makes a model upgrade a fifteen-minute decision rather than a fortnight of nervous manual spot-checking.

Where it runs

The assistant usually needs to live inside something you already operate — a Drupal or WordPress site, a SaaS application, an intranet, a support tool, or Slack or Teams. Having worked in those codebases for over a decade means the AI layer arrives as a proper module or plugin that respects your caching, queueing, deployment and rollback, rather than a separate service bolted on at the edge. For CMS-hosted assistants specifically, see AI for Drupal & WordPress.

Hosting can be your cloud account or mine, with the model provider chosen to fit your data residency and privacy constraints. If content cannot leave your infrastructure, that constrains the architecture and is worth establishing on day one.

How an engagement usually runs

1. Discovery — the questions people actually ask, where the source material lives, what its access rules are, and what "correct" means.
2. Ingestion pipeline — parsing, cleaning, chunking, embedding, and a repeatable re-index.
3. Retrieval and generation — hybrid search, re-ranking, citation, refusal behaviour and prompt design.
4. Evaluation harness — the question set, scoring, and a regression run wired into CI.
5. Integration and UI — inside your platform, with permissions enforced end to end.
6. Launch and operate — monitoring, cost tracking, trace logging, and a feedback loop from real usage back into the evaluation set.

See this in practice: Enterprise RAG Knowledge Assistant

Common questions