How to Test AI Agents: A QA Framework, Process and Toolkit
The short answer: test an AI agent in layers. Check the plain code parts with ordinary unit tests, score the agent’s answers against a set of real examples, check that it uses its tools in the right order, attack it on purpose, and run a few full end-to-end scenarios. Run every test several times, because agents don’t behave the same way twice. Nothing ships unless it passes agreed targets, and every failure found later becomes a new test.
Why testing agents is different
An AI agent is software that uses a language model to decide what to do next: search, call a system, fill a form, send a message. Traditional QA assumes the same input always gives the same output. Agents break that assumption in four ways.
- They vary. Ask the same question twice and you may get two different, both reasonable, answers. A single pass tells you little.
- There’s no single right answer. “Summarise this complaint” has many good outputs. You score against a rubric, not an exact match.
- They take actions. A wrong answer is bad; a wrong refund, email or database update is worse. Testing must check side effects, not just words.
- They change without a code change. A new model version, an edited prompt or new documents can shift behaviour overnight.
The framework: five layers of tests
1. Component tests. The tools an agent calls (APIs, parsers, database queries, prompt templates) are normal code. Test them with normal, fast, repeatable unit tests. Most bugs should die here.
2. Answer and retrieval evals. An eval is a test that scores an AI output. Build a golden set: a few hundred real questions or tasks with agreed good answers. Score each run with three kinds of graders: simple rules (is the format right, is the number correct), an AI judge (a second model grading against a written rubric), and people for a sample. For assistants that search documents, also score whether the right source was found.
3. Behaviour (trajectory) tests. A trajectory is the path the agent took: which tools it called, in what order, with what inputs. Check it used the right tool, didn’t skip a required step such as a permission check, stayed within its step limit, and recovered sensibly when a tool failed.
4. Safety and security tests. Try to break it on purpose. Hide instructions inside documents (prompt injection), ask for data the user shouldn’t see, push it outside its job. This is often called red teaming.
5. End-to-end scenarios. A small number of realistic, multi-turn tasks run in a safe copy of your systems, often with a second AI playing the customer. These are slow and costly, so keep them few and meaningful.
Read it like a classic test pyramid: many cheap, automatic checks at the bottom, a few expensive, realistic ones at the top. The highlighted behaviour layer is the one most teams skip, and the one that catches agents doing the right thing the wrong way.
What to measure
| Metric | Plain meaning |
|---|---|
| Task success rate | Share of tasks the agent completed correctly |
| Groundedness | Share of claims backed by a retrieved source |
| Tool-use accuracy | Right tool, right inputs, right order |
| Consistency | Pass rate when the same test runs several times |
| Safety violations | Attacks that got through, which should be zero for critical ones |
| Cost and time per task | Tokens, money and seconds to finish one task |
Set a target for each before you start, agreed with the business owner. “Good enough” for a draft-writing assistant is very different from “good enough” for an agent that moves money.
The process: from change to production
- Define success and risk. List the tasks the agent must do, what a good result looks like, and which actions are high-risk.
- Build and version the golden set. Mix common cases, edge cases and attacks. Store it with the code so every version of the agent has a matching test set.
- Run evals on every change. Prompts, model versions, tools and source documents are all “code” for an agent. Any change triggers the suite automatically.
- Gate the release. If any metric misses its target, the change goes back. For high-risk agents, a named person signs off too.
- Release in stages. First run the new version in shadow (it sees real traffic but its answers aren’t used), then as a canary for a small share of users.
- Monitor and feed back. Score a sample of live conversations every day. Every real failure becomes a new test, so the same mistake can’t return.
The diamond is the gate. A change that misses its targets loops back to be fixed; one that passes moves through sign-off and a staged release. The bottom row closes the loop: live failures grow the golden set that tests the next change.
The technology
You don’t need one big platform. A typical stack has:
- A test runner you already use (such as pytest), plus Playwright for any web interface the agent drives.
- An eval framework to run golden sets and graders. Open-source options include promptfoo, DeepEval and Ragas (for search-based assistants).
- Tracing, so every step, tool call and token is recorded. Langfuse, Arize Phoenix and LangSmith are common, and many accept OpenTelemetry data.
- Sandboxes and mocks: safe copies of the systems the agent acts on, so tests can’t send a real email or change a real record.
- Red-team tools such as garak or PyRIT to automate common attacks.
One caution on AI judges: check them against human grades on a sample before you trust them, and re-check when you change the judge model.
Common mistakes
- Testing once and calling it passed.
- Only checking the final answer, not the steps taken.
- No sandbox, so tests touch real systems.
- A golden set written once and never updated.
- Upgrading the model without re-running the evals.
How SHVX can help
Our Agent Assurance team builds golden sets, eval pipelines and release gates for agentic solutions, and runs them as an ongoing service.
Learn more about Agent Assurance.
FAQ
What is an AI eval?
A repeatable test that scores an AI output against a rule, a rubric or a reference answer.
How many test cases do I need?
Start with 100–200 real examples for the main tasks plus a set of attacks, then grow it from production failures.
Can an AI grade another AI?
Yes, for many tasks, if you write a clear rubric and regularly check its grades against people’s.
How often should we re-test?
On every change to prompts, models, tools or source data, and on a schedule even when nothing changed, because vendors update models.