Evaluation Is the Product

A conventional system fails loudly. It throws an exception, returns nothing, times out, crashes. The failure announces itself, and monitoring catches it before a user does.

A language model fails quietly and in complete sentences. It produces a fluent, well structured, confident answer that happens to be wrong. Nothing crashes. No alert fires. The output looks exactly like the outputs that were right.

That single property changes what delivery means. The programs I have seen ship understood it early. The ones that stalled treated evaluation as a quality gate somewhere near the end.

In a regulated environment the conclusion is sharper. The evaluation framework, the audit trail and the human oversight model are not scaffolding around the deliverable. They are the deliverable. The model is a component.

An accuracy number is not an evaluation

Pilots report a single figure. Ninety four percent accurate. The number is usually true and almost never useful, because it is an average across a test set somebody assembled, and risk does not live in the average.

The questions that give a number meaning: accurate on which categories of input, measured against what ground truth, established by whom, on data drawn from when. A system that runs at 94% overall and at 60% on the eight percent of cases carrying legal consequence is not a 94% system. It is a system with a specific, locatable hole, and the aggregate is what hides it.

Production evaluation is stratified. You define the segments where being wrong is expensive and you measure those separately, permanently, with their own thresholds. A model that is wrong three percent of the time is not a three percent problem. It is a problem in whichever three percent nobody thought to look at.

Somebody has to write down what unacceptable means

Every program I have joined that could not ship had the same gap. The team could describe a good output in detail. Nobody had written down what an unacceptable one was.

This is harder than it sounds and it is not a technical task. It requires business, legal, and clinical or compliance stakeholders to agree, in writing, on the categories of wrong. A confident fabrication. An answer that is correct but discloses something it should not. A refusal where the system should have answered. An answer that is right and unusable because it cannot be traced.

Until those categories exist there is nothing to measure against, no threshold to breach, and no basis on which anyone can sign. Teams skip it because it is uncomfortable: writing down what failure looks like means naming who owns each kind of failure. That discomfort is the work.

An evaluation without a name is a document

Approval has to belong to a person. Not a committee, not a function. A name that appears next to the decision that this system is fit to run.

The same applies downstream. When the system produces something unacceptable in production, and it will, three things have to exist already: who finds out, how fast, and who is empowered to turn it off. If turning it off requires a meeting, the system does not have an off switch.

This is where most AI governance material quietly fails. It describes principles instead of assigning accountability. A principle cannot be paged at two in the morning.

An audit trail is a design decision, not a log

Logging captures what the system did. An audit trail answers a different question, asked later, by someone who was not there: why did this system produce this output, for this person, on this date, and was that within the approved bounds at the time.

Answering that a year later requires things that have to be designed in from the start. Which version of the model. Which version of the prompt. What was retrieved and from where. What the human reviewer saw, and what they did with it. How long all of it is kept, and who is allowed to read it.

None of it retrofits cheaply. The architecture that was fastest to demo almost never has it, and adding it afterwards usually costs more than rebuilding. That is one of the reasons pilots die at month fourteen instead of month four.

Human in the loop is a control or it is a disclaimer

"A human reviews the output" appears in nearly every AI program plan. In some of them it is a real control. In the rest it is a sentence.

It is a control when the reviewer has explicit criteria, enough time per item to apply them, the authority to reject, and a record of what was rejected. It is a disclaimer when the reviewer is handed forty fluent, plausible outputs an hour, with no criteria and an implicit expectation to approve.

The second arrangement is worse than no review, because it manufactures the appearance of oversight. Automation bias is well documented: reviewers agree with confident machine output at high rates, and the more polished the output, the higher the agreement. A generative system produces the most polished output most reviewers have ever been asked to check.

So the design question is not whether a human is in the loop. It is what that human is measured on, and whether the process is capable of producing a rejection at all.

What it looks like in the plan

All of the above is delivery work. It has scope, effort, owners and a place on the critical path. If it does not appear in the plan as such it will not happen, because nothing in the pilot phase forces it.

Concretely. The evaluation framework is a named deliverable with a named owner. The taxonomy of unacceptable outputs is signed by business and compliance before the build, not after. Audit requirements are architectural inputs on day one. The reviewer model is staffed, and its rejection rate is reported like any other metric. The thresholds that would stop a rollout are agreed while everyone is still calm.

This is slower to start and dramatically faster to finish, which is the only speed that has ever mattered.

The part nobody wants to hear

Teams resist this because it feels like bureaucracy bolted onto something exciting. It is the opposite. The evaluation apparatus is what allows the system to be trusted with real work, and the trust is the entire value.

A model nobody is willing to rely on unsupervised has not automated anything. It has added a review queue.

The organisations getting generative AI into production in regulated environments are not the ones with better models. Everyone has access to roughly the same models. They are the ones that built the apparatus that makes a probabilistic system safe to depend on.

That apparatus is the product. The model is a component.