Intelyza Team, October 6, 2026, 4 min read

RAG readiness checklist before you build

A checklist to run before building a retrieval-augmented generation system: sources, permissions, evaluation, failure behavior and ownership.

Retrieval-augmented generation, or RAG, is the usual way to let a language model answer questions from your own documents. The demo is easy: load some PDFs, ask a question, get a fluent answer. The production system is harder, and the difficulty is rarely the model. It is the documents, the permissions, the questions and the people who will own it afterward.

Use this checklist before you commit budget. If you cannot answer an item, that is the first piece of work.

1. Is RAG the right tool?

RAG suits questions whose answers exist in text you hold: policies, contracts, manuals, tickets, product documentation. It suits poorly when:

  • The answer needs calculation across structured data. Use SQL or a BI model, and let the language model write or explain the query if you want a natural-language layer.
  • The corpus is small enough to fit in a single prompt. Skip retrieval and test that first.
  • The user needs an action, not an answer. That is an agent with tools, a different design.

Write down the user, the question types and the decision the answer supports.

2. Do you know your sources?

List every source system, who owns it and how it changes. For each, record:

  • Format: native text, Office files, scanned PDF, email, wiki pages.
  • Freshness: how often content is added or revised, and how deletions are handled.
  • Authority: which source wins when two disagree.
  • Version control: whether superseded documents are marked, because retrieving an old policy is a classic failure.

If three versions of a handbook sit in a shared drive with no labels, retrieval will find all three.

3. Is the content readable by a machine?

Open a sample of fifty or so representative documents and look at what a parser extracts. Scanned pages need OCR. Tables often turn into word soup. Headers, footers and page numbers pollute chunks. Slides lose meaning without speaker notes. Fix extraction before tuning anything else.

Decide how you will chunk. Splitting by headings and sections usually keeps meaning intact better than splitting by fixed length. Keep metadata with each chunk: document title, section, date, owner and source link.

4. Who is allowed to see what?

This is the item teams discover late. If a user could not open a document in its source system, the assistant must not quote it. Decide how permissions are carried into the index: filtered at query time against the user's identity, or separate indexes per audience. Test it with a user who should be denied.

Also decide what happens with personal data. Check retention rules, whether prompts and outputs are logged, where the model runs, and what your vendor contract says about training on your data. If you operate under GDPR or HIPAA, involve your compliance lead before the pilot, not after.

5. Do you have an evaluation set?

You cannot improve what you cannot measure. Before building, collect a set of real questions from the people who will use the system, each with the document and passage that answers it, and a note on what a correct answer looks like. Include awkward ones: questions with no answer in the corpus, questions that need two documents, and questions with ambiguous wording.

Score retrieval and generation separately. Did the right passage come back? Given the right passage, was the answer faithful to it? Rerun the set every time you change chunking, the embedding model, the prompt or the language model.

6. What should it do when it does not know?

Decide the failure behavior up front. Good defaults:

  • Show the source passage and link for every claim.
  • Refuse or ask a clarifying question when retrieval returns nothing relevant.
  • Never present a guess as policy.
  • Give users a visible way to flag a bad answer, and route those flags to a named person.

For high-stakes domains, such as legal, medical or financial advice, keep a human reviewer between the answer and the decision.

7. Have you thought about cost and speed?

Estimate query volume, average context size and the model you plan to use, then calculate cost with your vendor's current pricing. Decide an acceptable response time. Caching common questions, reranking fewer candidates and choosing a smaller model for simple queries are normal levers. Check them against your evaluation set so you do not trade quality away unknowingly.

8. Who owns it after launch?

A RAG system degrades quietly as documents change and users ask new questions. Name an owner for content, an owner for the pipeline and a review rhythm for the evaluation set and flagged answers. Decide how new sources are added and how retired ones are removed.

Quick readiness summary

You are ready to build when you can say yes to each of these:

  • We know the users, the questions and the decision the answer supports.
  • Our sources are listed, owned and extractable.
  • Permissions will carry through to the index.
  • We have a real evaluation set and agreed pass criteria.
  • Failure behavior is specified.
  • Cost, speed and ownership are decided.

If several answers are no, a short readiness assessment or proof of concept on one well-chosen corpus is cheaper than discovering the gaps during a full build.

Related reading

Want help putting this into practice?

Send a few lines about the work. A person reads it and replies within 24 hours.