How to Stop AI Agents from Making Things Up: A Practical Guide
You own what your AI says. In 2024 a Canadian tribunal held an airline liable for its chatbot’s wrong answer about a refund policy. Before launch, ask for three things: answers that cite their sources, a path for the agent to say “I don’t know”, and a measured accuracy score on your own questions.
The short answer: a hallucination is a confident answer that isn’t backed by facts. You can’t remove it from the model, so AI hallucination prevention means designing the system around it. Give the agent current, clean knowledge; make it answer only from what it retrieved, with citations; fetch exact facts such as prices and policies from systems, not memory; check every claim before it goes out; let it say “I don’t know”; and measure all of this on a test set and in production.
The cost of a made-up answer is real. In Moffatt v. Air Canada, a chatbot told a grieving customer he could claim a bereavement discount after flying, which the airline’s policy did not allow. The airline argued it couldn’t be held liable for its chatbot; the tribunal disagreed, finding it hadn’t taken reasonable care to make the chatbot accurate. Among teams building agents, quality is the top barrier to production in LangChain’s 2026 survey, and large companies named hallucinations as their biggest quality problem. Our Agent Assurance team tests and monitors agents for exactly this risk.
Five words explained
- Hallucination: an answer the AI states as fact that its sources don’t support.
- Grounding: making the AI answer from specific, retrieved facts instead of its general memory.
- Citation: a pointer from each claim to the passage or record it came from.
- Abstaining: the AI choosing to say “I don’t know” or hand over, instead of guessing.
- Groundedness score: the share of claims in an answer that its sources actually support.
Why do AI agents hallucinate?
AI agents hallucinate when a wrong answer starts at any of five points: stale or conflicting knowledge, weak search, a vague question, the model filling gaps, or misread tool results. Most teams blame the model, but it is only one of the five.
Each step hands its output to the next, so an error early in the chain becomes a confident answer at the end. Hallucination fixes should therefore start with knowledge and search, not with the model.
The Air Canada case shows the first point: the airline’s own website held a correct policy page, while the chatbot gave a different answer. Two sources of truth, and the customer trusted the one that was wrong.
How do you stop AI agents from hallucinating?
Six layers of defence stop most hallucinations: fix the knowledge, retrieve well, answer only from cited sources, use tools for exact facts, check before sending, and abstain when evidence is thin.
Each layer catches what the one above missed. The last layer is the safety net: when the evidence is thin, a person gets the case instead of the customer getting a guess.
1. Fix the knowledge
Keep one current source for every fact the agent may state, and retire old versions. If documents are long or messy, structure them first; see AI document extraction for large documents.
2. Retrieve well
Search by exact words and by meaning, then filter by date, product and version, as in hybrid RAG explained simply. If the right passage never reaches the model, no prompt can save the answer.
3. Answer only from sources, with citations
Instruct the agent to use only the passages it was given and to cite one for every claim. An uncited claim is a warning sign you can catch automatically.
4. Use tools for exact facts
Prices, balances, dates, stock levels and policy terms should come from a system or database through a tool, not from the model’s memory. Ask for structured output (fixed fields) when the answer feeds another system.
5. Check before sending
A final step compares each claim and number in the draft with its cited source. Unsupported sentences are removed, or the whole answer is held back. For sensitive topics such as refunds, fees or medical and legal matters, answer from approved wording.
6. Abstain or hand off
Make “I don’t know, here is who can help” a good answer, not a failure. When evidence is thin, the agent passes the case to a person with what it found; see human in the loop AI.
How do you measure AI hallucinations?
Measure AI hallucinations with a golden set of real questions with agreed answers. Include questions the agent should not answer. Track four numbers on every change: groundedness, correctness, the share of questions it rightly declines, and the share it wrongly declines. Then score a sample of live conversations every day, because documents, users and model versions all change. How to test AI agents covers the full method.
How do you stop an AI agent from taking a wrong action?
A made-up sentence is embarrassing; a made-up action is costly. Before an agent issues a refund or updates a record, it should confirm every value against the system of record, stay within strict limits, and ask a person for anything risky. The permissions side of this is covered in AI agent permissions.
Common mistakes
- Relying on “do not make things up” in the prompt as the only control.
- Upgrading to a bigger model instead of fixing knowledge and retrieval.
- No way for the agent to say “I don’t know”.
- Judging accuracy from a demo instead of a test set.
- Two places that state the same policy differently.
How SHVX can help
If you can’t say how often your agent is wrong, you can’t say whether it is safe to launch. Our Agent Assurance team builds your golden set, adds grounding and claim checks, and monitors accuracy after launch.
Learn more about Agent Assurance.
FAQ
Why do AI models hallucinate?
They are built to produce likely-sounding text. When the facts they need are missing or unclear, they fill the gap with something plausible.
Can hallucinations be eliminated?
Not completely. They can be made rare, caught before they reach a customer, and measured over time.
Does RAG stop hallucinations?
It reduces them by giving the model the right facts, but you still need citations, claim checks and a way to abstain.
How do we measure our hallucination rate?
Score answers on a golden set for groundedness: the share of claims backed by a cited source.
Case details and survey figures checked in November 2026.