The story: a chatbot that quoted a policy that didn’t exist
In November 2022, Jake Moffatt’s grandmother died. He asked Air Canada’s website chatbot about bereavement fares. The bot told him he could buy a full-price ticket and claim the reduced fare afterwards, within 90 days. He bought the ticket.
The real policy said the opposite: the discount must be requested before travel. The chatbot’s own answer even linked to the page that said so. Air Canada refused the refund and argued that he should have trusted the other page.
In February 2024, a British Columbia tribunal disagreed. It held the airline responsible for its chatbot and ordered it to pay $812.02, a partial refund plus interest and fees. The tribunal asked why one part of the website should be trusted over another.
Public reports don’t say how Air Canada built its bot, so I won’t call this a RAG failure. But it is the failure RAG exists to prevent: an answer that sounds official and doesn’t match the approved policy.
Source:
The lesson: the answer was decided before the question
Two pages on one website disagreed, and nothing told the bot which one counted. That points to four things a RAG system must know before it answers:
1. What the documents really say (ingestion). If the extractor misses a tracked change or a hidden sheet, the system answers from text nobody approved.
2. What each document is (status and version). Approved, superseded, draft or informal. A stale or informal page must never beat the approved one.
3. Where each question should go (routing). A price is a fact in a table, not a paragraph to find by similarity.
4. How each answer traces back (clause IDs). If an answer cites refunds.one_off.window, a reviewer can check it in seconds.
All four decisions are made before a single query runs. I call this the pre-retrieval layer, and it is where most RAG quality is won or lost.
Why this is necessary: later layers can’t repair earlier mistakes
Retrieval can only return what ingestion produced, and the model can only answer from what retrieval returned. A better prompt or a smarter model can’t restore a tracked change that was never extracted. Errors made early are carried through every later step, and they arrive looking confident.
There is a cost reason too. Adding a status field to a chunk on day one is cheap. Adding it after the system is live means re-indexing everything and re-testing every answer.
How to read this table. Each row is one decision, what goes wrong if you skip it, and why it is cheaper to fix early.
Ingestion. If a tracked change or hidden sheet is missed, the system never sees it. It then answers confidently from whatever text it did capture, which may be an old draft. Retrieval and the model can only work with what ingestion produced.
Status and version. A filter that removes anything not approved runs every time. A prompt that says “prefer the newest policy” is only a request, and the model can ignore it.
Routing. A price is an exact value in a table. Vector search finds paragraphs that sound similar, so it can quote a nearby but wrong number.
Clause IDs. Without a stable name for each rule, you can’t point to the exact source of an answer. Reviewers, audits and tests then have nothing to check against.
How I built it in the AOP RAG
AOP (Arcaence Online Project) is a support desk for a fictional web platform, Arcaence Online. Its RAG answers policy questions from a deliberately messy corpus of nine files: Markdown, text PDF, Word, Excel, email, a scan and a photo. I made four decisions before writing any retrieval code.
1. Ingestion. Each file type has its own loader. The Word loader reads tracked changes and hidden text. The Excel loader handles hidden sheets and merged cells. The scan and the photo go through OCR. Every file is normalized into one record, duplicates are detected, and each extraction problem is written down.
2. Status. Every document is indexed with a status (approved, superseded, draft or informal), and every chunk carries its status, version and clause ID. Production retrieval returns approved documents only. My tests run with that filter on and off, so I can see which control catches which trap.
3. Routing. Policy text (prose) is searched by meaning in a vector index (pgvector). Prices are looked up exactly in a Postgres table. A price question goes to SQL, and a policy question goes to vectors.
4. Clause IDs. Each policy rule gets a permanent name, like refunds.one_off.window, and a registry file maps every name to its document, version and section. The source documents stay untouched. The names and answer format are frozen and change only with a new version.
How the pieces fit
Files flow down the left. Ingestion, status, routing and the clause registry add what the later steps rely on. The registry gives each chunk its ID as it is labelled. Prose chunks land in the vector index, and prices land in a Postgres table.
At query time
The router decides first. A price question goes to the SQL table and returns an exact value. A policy question goes to vector search, where only approved documents can come back. A cited answer step (Haiku) then writes the reply and names the clause ID it used.
A framework: the Before-Retrieval Check
Before you tune embeddings, chunk sizes or prompts, ask four questions about every document type you plan to serve. Any “no” is a build item, and the cheapest fix is usually a control (a filter, a table, a registry) rather than a prompt change.
How to use it: run the four questions for each document type, record the answers in a decision log, and fix the weakest one first. Then add tests that fail if the control stops working, such as the filter-off run, which should show the trap returning.
How to use this checklist. Run the four questions for every document type you plan to serve, before you tune embeddings, chunk sizes or prompts.
Ask yourself is the question. Pass test is how you prove the answer is yes. Red flag is the sign that you’re relying on hope instead of a control.
Read. Compare what was extracted with what a person sees in the original file. Silent drops, such as tracked changes or scans, are the usual failure.
Label. Check that a superseded document can never come back in production. “The model will prefer the newest policy” means no control exists.
Route. Ask a price question and confirm it returns the table value every time. A number built from paragraph similarity is a warning sign.
Trace. Pick any answer and time how long a reviewer takes to verify it. If the answer cites page titles instead of stable IDs, they’ll struggle.
Closing
Read, label, route and trace. Each of the four is a decision made before a single query runs. Get them right and retrieval and the model have good material to work with. Get them wrong and no later step can fully repair it.







