AI quality

Enterprise AI needs a proof language

AI delivery is not only about building the feature. In enterprise systems, the hard work is often agreeing what proof different stakeholders will trust.

2026-06-14 Adapted from a LinkedIn post
Back to writing

The feature was not hard only because it used AI.

It was hard because we had to prove what good enough meant to several very different audiences.

We recently reached the point where an AI feature could move from integrated code to customer-visible rollout. The code was already in production. The capability was still hidden until the approval and compliance work caught up.

My first reaction was not pure celebration. Part of me thought: with today's models, and with what we know now, we might build parts of it differently.

That is the strange emotional tax of long AI delivery cycles. By the time you are finally allowed to ship, you have already learned enough to see the earlier version with sharper eyes.

But I am still proud of the work. Not because the implementation was perfect. It was not.

I am proud because the team had to create a proof language around something probabilistic.

A normal feature can often be tested through stable contracts: input, output, expected behavior, regression.

AI features break that comfort. The output can be different and still be equally good.

That one sentence caused more friction than I expected.

As QA, I was happy when hundreds of prompt executions became stable enough against our evaluation set. We had corner cases. We had repeated runs. We had metrics that showed the behavior was not drifting wildly.

Then I explained the same result to a data scientist, and the reaction was almost the opposite. If the test set succeeds that often, is it really representative?

That was a useful correction.

Our test data was synthetic by design. It was built to protect known risks and edge cases. It was not pretending to be the natural distribution of all future customer behavior.

Both perspectives were valid. QA wanted regression confidence. Data science wanted distribution honesty. The product needed both.

Compliance added another layer. We had to be careful not to describe a person's internal emotional state as fact. Safer wording was closer to: the text suggests frustration, not the person is frustrated.

That sounds like a small wording change. It is not. It changes what the system claims to know.

This is where many AI speed narratives feel incomplete to me.

AI can make implementation faster. It can also multiply the number of proof layers you need before responsible deployment: quality, drift, evaluation data, model suitability, customer impact, compliance language, and approval evidence.

For engineering managers, the work is not just how do we make the team faster?

It is: what kind of evidence will each stakeholder trust? Where are their definitions of quality different? Which differences are real disagreement, and which are just different professional vocabularies?

The model may generate the output. The organization still has to agree what good means.

Where have you seen an AI project slowed down less by implementation, and more by proving it was safe enough to ship?