Most RAG problems are retrieval problems wearing a model costume

When an assistant answers badly, the instinct is to change the model or the prompt. Usually the retrieval was wrong and the model faithfully summarised the wrong thing.

Before touching the prompt, look at what came back for the query. If a human could not answer from those chunks, the model had no chance either — and no amount of instruction will fix a context that does not contain the answer.

This is why evaluation matters more than prompt craft: without it you cannot tell which of the two you just changed.

The check is mechanical. Log what retrieval returned, then read it the way a person would — if the answer is not in there, the prompt was never the problem.

1
2
3
4
5
6
7
8
9

def answerable(question: str, chunks: list[str]) -> bool:
    """Would a careful human answer this from these chunks alone?"""
    context = "\n\n".join(chunks)
    return grade(question, context) >= 0.7  # human-labelled, not model-scored


for case in eval_set:
    if not answerable(case.question, retrieve(case.question)):
        print(f"retrieval miss: {case.id}")

Run it over the whole evaluation set before touching a single instruction. A retrieval miss is a chunking or an embedding fault, and no wording fixes it.

retrieval miss: refund-window-eu
retrieval miss: contract-termination-notice

Two misses out of forty is a chunking problem. Twenty is an embedding problem.