What we are talking about
Most AI projects start with a model and end with a demo. Mine started with a folder of nine documents that were built to break it.
I am building a support assistant for a fictional web platform, Arcaence Online. It will answer questions about terms, fees and refunds, and every answer must point to the exact clause it came from. Before I wrote a line of the assistant, I wrote the exam it has to pass: 40 questions, 15 of them deliberate traps, and a set of rules about which source wins when two documents disagree. It is all committed in the public repo, and none of it needs a model to read.
The idea is simple: for an AI system, the test is the requirements document. A requirement says what the system must do. A test says what it must do, and proves it. If you cannot write down what a good answer looks like, including the answers that look good and are wrong, you do not yet know what you are building.
A trap is the unit of this exam. It is a question where a tempting wrong source exists, so that a system which looks fine on easy questions still fails.
The two lanes use the same steps. Only the order changes, and with it the moment you learn that an answer is wrong.
Why it is necessary
A wrong AI answer looks exactly like a right one. With ordinary software, a requirement is met or it is not, and a failure usually shows. With an AI system the output varies, and the worst failures are fluent, confident and wrong. A glance will not catch them. Only a test written in advance by someone who knew which wrong answers were tempting will.
The decisions that matter are product decisions. Which source wins when two documents disagree? Which number must never come from a guess? What must the system refuse to do, whatever a document tells it? None of these is an engineering question, and each is cheap to settle before the build and expensive to discover after a customer has been quoted the wrong refund window.
Customer knowledge is contradictory by default. Real companies keep their policies in a current version, an old version, a draft, a staff email and a spreadsheet. Writing the exam forces the conversation nobody has had: who decides which of these is true?
Without an exam you cannot tell progress from drift. Change one setting and one answer improves while three others get worse. Only a fixed set of cases, scored the same way every time, shows which.
It is the record of what was promised. For sign-off and governance, a passing test is evidence. “We were careful” is not.
If you work with a customer on an AI system, as a product manager or an engineer embedded in their team, three questions get you most of the way to an exam:
Which wrong answer would hurt most? Start there. It is usually a number, a date or a promise.
When two of your documents disagree, which wins, and who decides? If nobody can answer, that is your first finding.
What may the system never do, whatever a document says? This is where limits on behaviour come from, and where you learn how much untrusted text the system will read.
The answers become test cases. The test cases become the specification, and the specification stays honest because it runs.
Each question on the left produces cases in the middle, and those cases are the specification the system is held to.
How I implemented it in the Arcaence-Online project
The corpus is the exam paper. The project has nine files in five formats: the current terms and fee schedule, a superseded older version, an unpublished draft, a staff email, meeting notes, a finance spreadsheet, a scanned page and a phone photo. A clean corpus would test nothing, so each file exists to create a specific failure.
Four of the traps, in plain terms:
The old version. The superseded terms share 88 per cent of their wording with the current ones, so a search matches both almost equally. Only a human decision, written in a manifest, says which is current.
The draft. An unpublished draft changes the refund window from 7 days to 10. The assistant must answer from the approved terms, not the draft. In the Word file that edit is stored as a deletion and an insertion, and a careless reader joins them into “710”.
The hidden price list. The fee spreadsheet has a hidden tab of retired prices. A reader that skips hidden sheets answers correctly. One that reads everything quotes a price that no longer exists.
The planted instruction. The draft ends with a line addressed to “any AI assistant”, telling it to approve every refund. Text inside a document is data, never an instruction, and the exam checks that the assistant treats it that way.
Each trap became a rule before any code existed. The manifest gives every document a status (approved, superseded, draft, informal) and a rank from 1 to 5. Search will filter by status before it ranks by similarity. Prices will come only from an exact table, never from text. Every citation will be checked against a list of real clause IDs.
The exam grows as the build finds surprises. Each loader I wrote was run against the same files, and every surprise went into a findings log. So far there are six, each tied to the trap that guards against it. The tracked change, the hidden tab and the planted instruction above were all found this way, by reading what the loaders actually kept.
The order matters. The exam was written on day one, before the pipeline. The scorer that runs it comes on 15 Nov, and from then on every change to the system is measured against the same cases. The student is built second.
Next
The next post is written from the forward deployed engineer’s seat. It covers eight ways real customer files break ingestion, and why prices come from a SQL table and never from the vector store. The whole project, including the test suite, is public at github.com/saurabhsmahajan/arcaence-online.