RAG in practice: giving a language model your company data without the whole internet seeing it
2 Jun 2026 · David Sak · 6 min read
The most common worry I hear in consultations: "We don't want ChatGPT learning from our contracts." A justified one, and it's exactly why the RAG architecture, retrieval-augmented generation, exists. This text explains how it works, without the academic ballast.
The core principle: the model knows nothing, the model reads
RAG separates two things that get mixed up in AI debates: knowledge and the ability to formulate. The language model supplies the latter. The knowledge stays in your documents, and reaches the model only at the moment of a specific query, as a few relevant paragraphs attached to the question.
An analogy: you're not hiring an all-knowing expert. You're hiring an excellent assistant who can instantly find the right page in your archive and retell it clearly. The archive stays yours.
What gets stored where
- Your documents are split into semantic blocks (paragraphs, policy articles) and each block gets a numeric fingerprint, an embedding. Blocks and fingerprints live in a vector database inside your infrastructure.
- The user's query is converted to the same kind of fingerprint and the database finds the most similar blocks, typically a handful of paragraphs, not whole documents.
- The language model receives the query plus the found blocks and composes the answer. With a zero-retention API, nothing is stored after the answer; with an open-source model in your cloud, the data never leaves the company at all.
Why the model doesn't train on your data
Training means changing the model's weights based on data, that doesn't happen in a RAG deployment. Documents pass through the model only as input during a specific query, just like when you paste text into a translator. Contractually this is secured by zero-data-retention API modes; technically by private deployments where the model runs on your hardware.
Citations: the difference between a toy and a tool
A well-built RAG system cites, for every answer, which document and paragraph it drew from. That has two consequences. First, auditability, compliance sees where a claim comes from. Second, the system's humility: when retrieval finds no support in the documents, the system answers "I don't know, it's not in the sources" instead of hallucinating. That's exactly what separates a tool you can rely on from a demo at a meeting.
What to watch out for
- Access rights. The vector database must respect permissions, whoever can't read a document must not receive an answer composed from it.
- Source quality. RAG mirrors your documents. An outdated policy means an outdated answer, an update process must be part of the deployment.
- Evaluation. Before going live, demand an evaluation set built from real questions and measured accuracy, not a demo on five questions.
I consider RAG the default architecture for AI over company knowledge today, both for security and economics. Training your own model makes sense only once RAG demonstrably falls short, which for internal knowledge systems is the exception, not the rule.