An insurance team we spoke with had built an internal assistant over eight years of policy documents, circulars and claim precedents. Users reported it was confidently wrong about coverage limits roughly one query in five. The instinct was to change the model. The actual cause was that the limits lived in tables inside PDFs, the chunker had split those tables mid row, and the retrieved fragment showed a number with no idea which product it belonged to. The model was reasoning perfectly over garbage.
This is the standard failure. When a retrieval-augmented system gives a wrong answer, the fault is in retrieval far more often than in generation.
Chunking Is a Content Decision
The default advice, 500 tokens with 50 tokens of overlap, is a starting point that suits prose and suits almost nothing else. Contracts, policy schedules, engineering specs and financial statements all carry meaning in structure that a fixed window destroys.
What works better in practice is chunking along the document’s own boundaries. Split on headings and clause numbers. Keep a table whole even if it runs long, and prepend its caption and the section heading to the chunk text so the fragment carries its own context. For long documents, attach a short generated summary of the parent section to each child chunk. That single change tends to move retrieval accuracy more than any embedding model upgrade.
Keep the metadata too. Effective date, document version, business unit, jurisdiction. An answer drawn from a superseded circular is worse than no answer, and only metadata filtering prevents it.
The Retrieval Stack That Actually Performs
Dense vector search alone is not enough for enterprise content, which is full of product codes, policy numbers and named entities that embeddings handle poorly.
- Hybrid search. Combine BM25 keyword matching with vector similarity and fuse the result lists. Exact identifiers get found by the keyword leg, conceptual queries by the vector leg. This is usually worth 10 to 20 points of recall over vectors alone.
- A reranker over the top 50. Retrieve wide, then use a cross encoder to reorder and pass only the best 5 to 10 into context. Reranking is comparatively cheap and reliably lifts precision at the position that matters.
- Query rewriting before search. User questions are terse and full of pronouns. Expanding the query, or generating two or three variants and merging the results, closes the vocabulary gap between how people ask and how documents are written.
- Permission filtering at query time. Filter by the requesting user’s entitlements inside the search call, never after. A retrieval layer that can surface an HR file to the wrong employee is a data incident, and under India’s DPDP Act it is a reportable one.
- Citations on every claim. Each answer should name its source document and section. It makes the system checkable, and it converts user distrust into useful correction reports.
Not one of those five is a model decision. They are all scaffolding, which is the general case rather than a quirk of retrieval — the harness is the agent, and swapping the model underneath a weak retrieval stack changes the tone of the wrong answer and nothing else about it.
Evaluate Retrieval Separately
Judge the retriever on its own before judging the answers. Assemble 100 real questions with the documents that should have been returned, then measure recall at 10 and mean reciprocal rank. If the right chunk is not in the top 10, no amount of prompt work will save the response.
Once retrieval is solid, evaluate generation for faithfulness, meaning whether every claim is supported by the retrieved text, and for appropriate refusal when the corpus simply does not contain the answer. A system that says it does not know is worth considerably more than one that guesses well. It is the same trust economy engineering teams settle into with coding assistants: output you have to verify line by line is worth less than half as much output you can take at face value.
Most of this work is librarianship, which is not what anyone budgeted for. Splitting documents along their own seams, stamping them with effective dates and jurisdictions, deciding who is entitled to see what — none of it demos well, and all of it decides the answer. When an AI agent project of this kind stalls, it has almost always stalled on the corpus rather than the prompt.