When every test is green and the AI is still wrong

Moamen Elmasry, product owner. Definition of Trustworthy, version 1.0.

My distrust of AI answers comes down to one thing: ask an AI the same question twice and you will not get the same answer twice. Classic software is deterministic; it does exactly what it was told, every time. Generative AI is probabilistic. Each time, it picks one answer among many possible ones, and nothing guarantees that the next one will be as good as the one I just read.

When I started testing AI products as a product owner, that distrust became a work problem. I had to say whether a feature could go out to users, and my usual tools were no help. The Definition of Done, which every agile team knows, assumes that a feature eventually gets done. An AI answer never quite does, since it will never be exactly the one each user expects. So I asked myself different questions. What would the AI have to show me before I trusted it with my users? From what level of quality? And what had to be added to the Definition of Done to cover what it will never cover?

The Definition of Trustworthy grew out of those questions. Its starting point has not changed since: replacing a feeling of confidence with evidence.

To see what can go wrong without it, picture a team about to ship an assistant that writes meeting minutes from a transcript. Before release, they do things properly. Twenty-five test meetings, each with reference minutes written by someone from the business, and a score that measures whether the important information made it into the output. The result comes in at 0.96, against a target of 0.85. Everyone is happy.

Except that one of the twenty-five reports says a budget was approved in the meeting. Nobody approved it. The text still contains every piece of expected information, so it gets the top score and vanishes into the average. If it reaches a client or a steering committee, the mistake will cost far more than the eleven points gained above target.

What the Definition of Done cannot do

The Definition of Done sets what a feature must meet to count as finished. It works very well for anything with an exact answer. A user without the right permissions either sees the client record or doesn't; a link either expires after thirty minutes or it doesn't. You write the test, run it, get a yes or a no, and you will get the same yes tomorrow.

A generated answer does not fit into that yes or no. There are a dozen good ways to summarise a meeting, and the assistant will never write the same one twice. Comparing word for word would reject good summaries. Reading through a few examples would mean signing off on a feeling, which is exactly what I wanted to avoid.

So I split the two questions apart. The Definition of Done still covers whatever is exact. The Definition of Trustworthy, DoT for short, covers the rest. Its job is to prove that a generated result reaches the quality needed for its intended use, at a level of risk someone has agreed to accept. The name is deliberate: nobody asks whether an AI answer is finished. The question is whether you can trust it.

Knowing which side you are on

The hard part is not measuring, it is sorting. A single product almost always mixes both kinds of behaviour. The assistant checks the user's permissions, fetches the right data, sticks to a format, and then writes. So I never classify a whole product, only each of its behaviours, and I ask two questions.

The first is simple: can the right result be written down in advance and compared exactly? If several answers are acceptable, you are on the DoT side. The second is less obvious, yet it is the one that settles the hard cases: where does the right result come from? If it follows from a written rule (a schema, a decision table, a formula), it belongs to the Definition of Done. If someone had to look at each case to say what was correct, it belongs to the DoT.

Take a feature that sorts requests into three categories. If the sorting follows a table the team wrote, you can test it case by case, and it belongs to the Definition of Done. If a trained model does the sorting, nobody wrote the rule down: all you can do is compare its choices with annotated examples, and it belongs to the DoT, even with only three categories. You then have to accept that it will sometimes be wrong, because that is precisely what a threshold means. When the business will not accept any error at all, the answer is not a 100% threshold. It is to write the rule down so it can be tested.

An experiment rather than a test

In practice, a DoT evaluation looks more like an experiment than a unit test. You freeze a version of the product, run it over a versioned set of cases, then score every result with evaluators chosen in advance. That can be a script, a rubric, a person, sometimes an AI acting as judge, provided it has first been checked against human scores. Then come the acceptance rules.

Everything hinges on when those rules are set: before the first run. The temptation is familiar. You look at a demo score, 4.2 out of 5, and set the target at 4. The test is bound to pass, and it no longer proves anything.

Box: ten questions to settle before running an evaluation

• Which capability are we evaluating? One at a time.
• On which cases, and which version of them?
• What counts as an acceptable answer: an annotated reference, a rubric?
• With which metric?
• Which threshold, and who approved it?
• Which subsets must pass on their own (a language, a type of customer, a sensitive category)?
• Which mistake blocks everything, whatever the score?
• Which model, instructions, parameters and number of repetitions?
• What do we keep as evidence?
• Who approves the protocol, and who makes the final call?

If a single answer is missing, the evaluation is not ready to run.

Why a good average is never enough

Back to our meeting minutes. With a threshold alone, the 0.96 went through. Stopping it takes two more tools. The first is the veto: a rule saying that an invented decision blocks acceptance, even once, whatever the score. The second is the critical slice: you check the cases that matter most separately (say, meetings where money is committed) so that a good overall average cannot hide a group that is doing badly. In our example, raising the threshold to 1 would not have helped. The faulty report had the top score, and only a veto could stop it.

The verdict itself can take three values. "Pass" if everything was measured and every rule holds, "Fail" if one of them does not. And "Not Ready" when one of the ten answers in the box is missing. That third value is often treated as a formality; it is actually the one that protects you most. A team under pressure always wants to fill in the missing threshold, "just to move forward, we'll get it signed off later". In the report, that hastily typed number looks exactly like an approved one, and nobody will ever tell them apart.

Someone still has to decide

For a product that mixes both kinds of behaviour, which is the case for most assistants, each part has to pass on its own. First the exact tests, then the DoT evaluation, then testing with users whenever the product is exposed to them. I refuse to blend it all into a single score: an excellent writing score does not make up for a user seeing data they are not allowed to see.

That leaves the question of who decides. Not the person running the tests, and not the tool. Thresholds and vetoes belong to the product owner and to the business teams involved, because they are the ones carrying the risk. Running an evaluation can be automated, and that is what I built the Testing Agent for. But when a threshold is missing, the tool must flag it, not guess it.

What the method does not promise

The DoT is not a standard. It is a method written from my own practice and from the test levels that have been taught for a long time. It also has a cost: gathering cases annotated by people from the business takes time, and that is often where projects stall. I have not found a shortcut. Without those cases you can still produce numbers, but you do not know what they are worth, and you are back to the confidence-by-feeling this all started from.

This page is the reference for the method. It will be dated whenever something of substance changes.

To go further: the Testing Agent applies the method directly inside your AI assistant, and the AI Product Testing course walks through it step by step, with exercises you run in your browser.