AI Vocabulary

RAG: does it actually read your documents?

RAG — retrieval-augmented generation — describes an arrangement that fetches relevant documents before formulating an answer, instead of answering solely from what the model has learned. It is currently the most widespread way of making a system work on your own documents.

How it works, in three stages

The question is first turned into a representation allowing it to be compared with documents. The system then selects the closest passages in the corpus it has been given — your contracts, your notes, a case-law database. Finally it drafts its answer relying on those passages, which it has in front of it as it answers.

The difference from a model interrogated alone is that between a closed-book and an open-book examination.

A note on vocabulary, since the acronym circulates unexplained: retrieval means fetching, augmented generation means text production enriched by what has been fetched. The name describes the mechanism exactly, which is rare in this field.

What it changes

The system can cite sources that exist, since it actually retrieved them. It can work on documents it never saw during training, including ones signed yesterday. And it can indicate where its answer comes from, which makes verification practicable.

The system does not know what it failed to find. An absence of results looks like an absence of authority.

What it does not solve

The quality of the answer depends entirely on the quality of retrieval. If the right passages do not come back, the system answers on the wrong ones, with the same assurance. That is the principal defect and it is barely visible: the answer is well formed, it is simply grounded in something else.

The defect is harder still to see because nothing signals it. The system does not say it hesitated between two passages, nor that neither really matched: it answers with what it has.

Document chunking creates a problem of its own. To be compared, texts are cut into pieces; a clause whose scope depends on a definition 40 pages away may be retrieved without it. The system then reads a correct fragment and draws a false conclusion.

It does not remove hallucinations, it shifts their nature. The system invents fewer citations and can still misread those it has found, or fill in by itself what the retrieved passages do not say.

And above all it is not a memory. RAG retrieves documents; it holds neither the decisions taken, nor their reasons, nor what was set aside. To the question why did we accept this clause, it will answer by showing you the clause.

What to ask a provider

Three questions are enough to assess an arrangement of this kind, and they require no technical knowledge.

What corpus does retrieval actually run on: our documents alone, an external database, both?

What happens when nothing relevant is found — does the system say so, or answer anyway?

Can I see the passages the answer relies on, and open them?

The third is the most discriminating. An arrangement that does not show its passages asks you to believe it; one that shows them lets you check in seconds.

Why it matters to a lawyer

Because it is the word you will most often hear in answer to whether the system works on your documents. The answer is yes, and it deserves two further questions: on what corpus exactly, and how will you know retrieval missed something.

The second is the more useful. An arrangement that finds nothing must say so as such, and not leave the impression that nothing exists.