Traditional testing asks whether the same input produces the same output. A probabilistic system passes every test your organisation runs today and can still be wrong a third of the time. This is the one genuine gap across sixteen marketed AI platform components — and the software is a few hundred lines. What it costs is expert time and a decision nobody has made.
Second of four groups in a component-by-component teardown. The overview is in The Enterprise AI Platform: What’s Left When You Unpack It, and the first group covered the model call itself.
Every component so far has dissolved into an existing system. This one does not.
Four Outcomes, Not Two
Traditional testing has pass and fail. Extraction has four, and collapsing them hides everything.
| The system said | The truth was | Outcome |
|---|---|---|
| A value | The same value | Correct |
| A value | A different value | Wrong |
| “I could not read it” | It was unreadable | Correct abstention |
| “I could not read it” | It was readable | Over-abstention |
| A value | It was unreadable | Fabrication |
Fabrication and over-abstention have opposite fixes. One needs a stricter instruction, the other a looser one. A single accuracy percentage tells you neither.
So you set three gates, not one. A system at 96% accuracy that never abstains is worse than one at 92% that abstains cleanly — the first fails silently, the second routes to a human.
The key point:
A system that never says “I don’t know” is not accurate. It is confident.
“We Already Have the Data” Is 80% Right
Every enterprise has historic records. Referrals keyed by staff, invoices posted to the ERP. Those give you correct values for free, and they are why this component is cheap.
But they do not tell you where the value came from. The clerk who entered that member ID may have read it off the fax, or pulled it from the patient record after matching, or picked up the phone. All three produce an identical row.
So when your system correctly reports that a field was illegible, the historic record scores it as an error. Tune against that and you are training toward confident invention — the failure you least want.
The fix is two days. Take 200 records and have someone open the source document and mark each field present, absent, or illegible. Not re-keying values, which are already right. Just answering where they came from.
Then it becomes self-sustaining: a reviewer who fills in a field the system marked illegible has just told you it was readable.
A Human in the Loop Does Not Remove the Need
The intuitive objection: if a person confirms every output, the system is just typing with a head start. Why measure it?
Reviewers stop verifying and start confirming. Give someone pre-filled fields and they approve what looks plausible. Well documented in clinical decision support. Approval rates stay high while catch rates quietly fall.
Confidence-gating is the only path to value, and it needs a number. Reviewing everything costs what the manual process cost. The return comes from auto-processing the clean cases, and you cannot responsibly set that threshold without measured per-field accuracy.
Corrections are free labels. Every field a reviewer fixes is a labelled case at zero marginal cost. The review queue is a labelling pipeline that is already running.
And measure the reviewers themselves. Seed a small share of known-bad cases and track the catch rate. If reviewers miss planted errors at 40%, the human control is decorative and your auto-processing threshold is built on sand. Almost nobody does this, and it is cheap.
Sometimes the Business Process Is the Evaluation
Look for this before building anything.
Invoice intake has continuous free ground truth. Three-way match and payment reconciliation tell you, without any labelling, which extractions were wrong. Every invoice that fails to reconcile is a labelled error.
Referrals have a weaker version: eligibility check failures and denials coded to intake data.
Approval is an opinion at the moment of review. The denial is the fact. Where a downstream system will later reveal whether the output was right, name it and carry the key. That signal is bigger, fresher, and cheaper than any dataset you maintain.
When There Is No Correct Answer
Extraction has a right answer. A great deal of enterprise AI does not.
A care task recommendation — suggesting what a clinician should do next for a complex patient — has no ground truth to score. Neither does a summary, or a prioritised worklist. This is the shape most assistant use cases take, and the entire evaluation literature is about the other shape.
The answer is not better measurement. It is constraining the output until measurement becomes possible.
Stop generating. Select instead.
The system does not write tasks. It chooses from a governed catalogue of approved tasks, each with defined wording and rationale, owned and versioned by the clinical function that owns the protocol.
| Free generation | Selection from a catalogue | |
|---|---|---|
| Is there a right answer? | No | Yes — which tasks should fire |
| Measurable | Not really | Precision and recall per task |
| Worst failure | An invented task, unbounded | A wrong or missing selection, bounded |
| Audit trail | “The system said this” | Task identifier plus the facts that triggered it |
| Fixing a bad output | Case by case | Retire the catalogue entry once |
And the dangerous failure inverts. In extraction, invention is what hurts you. In recommendation, omission is — a missing task is invisible to someone skimming a plausible list, while a wrong task gets noticed and removed. Measure omission explicitly, weighted by harm.
The key point:
A schema constrains the shape of the output. A governed catalogue constrains its meaning. Both turn “the system said something” into “the system chose from what we approved.”
Buying It
The discipline does not change when the AI arrives inside a purchased product. Only the lever does.
Your vendor can change the model underneath you on a Tuesday with no release note, and you will find out from a billing variance three weeks later.
Run your labelled cases against the vendor during the pilot — not their curated demonstration, but your documents, including the bad ones. Then you are negotiating against a number instead of an impression, and a bake-off between two vendors scores both on the same set.
Then put it in the contract: notification before model changes, an accuracy floor on your acceptance set retested at renewal, the right to evaluate, and export of AI-generated fields with some indication of provenance. Most vendors will not offer these. Some concede several during procurement, and almost none afterward.
The same 200 cases gate your custom code, score vendors in a bake-off, and regression-test the purchased product every quarter. It is the one artefact that survives a build-versus-buy reversal. Enterprises version code and data — almost none version their ground truth.
What Generalises
Not the runner. Your teams work in Python, Java, .NET, and SaaS configuration, and a shared library serves one of them while the rest bypass it.
What generalises is two schemas and a place to send them.
- The report — point in time, before release. Capability, dataset version, accuracy, fabrication rate, abstention rate, the gates, pass or fail.
- The measurement stream — continuous, after release. Which capability, which record, which field, what the model said, what the person did, why, and the key that joins to a downstream outcome.
Send the fact of a correction, not its content. Old and new values are sensitive and stay in the application’s own store. The shared tier holds counts and categories, which is what makes it cheap to approve and safe to query. In most enterprises this is a table in the data platform rather than a service.
A table nobody reads is worse than no table. Four patterns are worth looking for, and only a portfolio view catches them:
- Agreement rate declining on a capability — usually the input mix shifted, or a vendor changed a model
- Auto-processing rate rising without a threshold review — someone crossed from assisted to autonomous by tuning a number
- The same field failing across several capabilities — often one shared cause
- Correction rate near zero — not a success, but reviewers who stopped verifying
That last one a single team cannot see; its own approval rate looks like quality.
Monthly for the portfolio, quarterly for thresholds with the business owners who signed them. Not a live dashboard — nobody watches dashboards.
Same pattern as test coverage. Nobody shares a test runner across Java and .NET. Everyone reports a coverage number into the same quality dashboard.
Final Thoughts
This is the only component in sixteen that does not dissolve into something the enterprise already runs. And even here the build is small: a scoring script, two schemas, and a table.
What is expensive is the expert time to say what “correct” means, and the seniority to say a number is not good enough yet. Neither is a technology problem, which is why no platform purchase solves it.
Enterprises buy evaluation platforms and then discover they have nothing to evaluate against.
Measure what the system produces.
Then decide who signs.
Tech Canvas 