Usage Is Not Quality, and Neither Is Value

Which of your AI solutions would you switch off tomorrow? Most leadership teams cannot say, and rarely for lack of reporting. A typical quarterly report lists 14 solutions in production, 9,000 weekly users and high satisfaction. It cannot say which of the 14 are right often enough to trust, which save time once someone has checked their output, or which cost more to run than they return. Every number on it measures usage. None of them measures quality or value.

Four questions decide whether an ML and AI team runs a portfolio or a collection of pilots:

1. How do we get an overview of quality, usage and value across our AI solutions?
2. How do we make sure each solution lives up to what users and the business expect?
3. How do we handle errors, problems and continuous improvement?
4. Where should the team focus first?

The answers share one idea. Quality, usage and value are three different measurements, they move independently, and almost every misleading AI report comes from letting one stand in for another.

Three numbers per solution, one page for the portfolio #

Define the three precisely, because the loose versions are the ones that mislead:

• Quality is how often the output is right for the task. Measure it on a fixed set of real cases graded by people who know the domain, and on a regular sample of production output. A vendor’s benchmark score says how the model does on someone else’s task.
• Usage is how much of the work the solution was built for actually goes through it: the share of eligible cases, and how often people accept, edit or override what it produces. Logins and prompt counts measure curiosity.
• Value is the change in a business outcome against a baseline: handling time, error rate, cost per case, compliance findings, revenue. Without a baseline it is a claim.

Each can move while the others stand still. High usage with low quality is a risk that grows every week. High quality with low usage is a fit or adoption problem. Neither says anything about value, and value is the number that justifies the budget.

Put every solution on one page with the same columns. The columns matter less for what they contain than for what they refuse to accept in its place:

Mark every value figure with how it was measured: a controlled comparison, a before-and-after, or an estimate. A leadership team will read them as equally solid unless the page says otherwise. The same page is also the inventory that any AI governance work starts from, whether that is ISO/IEC 42001 or a risk classification under the EU AI Act.

Here is that page for four of the 14 solutions, trimmed to the three numbers and the decision. The decision column holds one of four states, defined further down.

Read across the rows, not down the columns. The reply drafts are the only solution where all three numbers agree, so they get scaled. The policy Q&A tool is the most used solution in the portfolio and the least accurate: a report that counts usage puts it at the top, and it is the largest risk on the page. Contract extraction is accurate and avoided, so more model work would not help, and the first job is to ask the lawyers why. The meeting summaries have users and a survey and nothing else, which is what a solution that should be stopped looks like.

Expectations are written down before anything is built #

A solution can only meet expectations someone wrote down. Before building, or now for anything already live, the business owner, a few real users and the team agree on five things:

1. The task and who it serves.
2. What good looks like, shown with real cases rather than adjectives.
3. Which errors matter. Some are cosmetic, some are costly, some are unacceptable at any rate, and each class gets a severity.
4. The thresholds: the quality needed to launch, and the level at which the solution is pulled.
5. The value hypothesis, with its baseline measured before launch, because it cannot be measured afterwards.

The cases become a golden set: a few hundred real examples graded by domain experts, covering the common cases, the hard ones and the ones that must never go wrong. It is the release gate. No model, prompt or data change ships if the pass rate drops or a severe failure appears.

Grading a few hundred cases on every change is where language models are tempted to grade themselves. That can work, with care. Zheng et al. found that a strong model used as a judge agreed with human preferences over 80% of the time, the same level at which humans agree with each other, and also documented position, verbosity and self-enhancement biases. So keep a human-graded slice, measure how often the judge agrees with it for each failure class, and avoid letting a model judge output from its own family.

In production, users show their expectations through what they do rather than what they rate. Thumbs-up buttons collect few answers, mostly from people with strong feelings. Acceptance and edit rates, overrides, escalations, repeated questions and abandoned sessions arrive on every case.

The business expectation, value, needs a comparison, and self-reports are not one. In METR’s randomized trial in early 2025, 16 experienced open-source developers took 19% longer to finish real tasks when they could use AI tools. They had expected a 24% speedup, and afterwards they still believed AI had made them 20% faster. METR said in 2026 that developers are likely faster with today’s tools, and repeated that self-reported speedups “can be quite unreliable”. The lesson for anyone measuring value holds either way: ask people how much time a tool saves and you measure their impression. Measure the time instead, on comparable work, with and without.

That rarely needs a research team. Rolling a solution out to teams in waves gives a comparison group for free. And count the review: a draft that saves ten minutes of writing and needs eight minutes of checking saves two.

Every error becomes a test #

Errors come from four places: user reports, overrides, sampled reviews and incidents. Route all of them into one queue, and give each confirmed error three labels: the failure class it belongs to, its severity, and its root cause. The root cause decides who fixes it:

Then add the case to the golden set. That one habit turns error handling into a regression suite that grows with every mistake, and it is what makes improvement continuous rather than episodic: a problem fixed once cannot quietly come back with the next model update.

FIG · The improvement loopDIAGRAM

Two more practices keep the loop honest.

Error budgets. Borrowed from site reliability engineering: for each solution, agree in advance the error rate per severity that is acceptable. While a solution stays inside its budget, the team ships improvements. When it goes over, new features stop until it is back. The argument about whether something is good enough happens once, before launch, instead of after every incident.

Changes are releases. A new model version, a rewritten prompt and a batch of new documents in the search index all change behaviour, and each runs the golden set before it ships. Model providers update and retire models on their own schedule, so a model upgrade belongs in the release process, not in maintenance.

Underneath both sits a weekly hour in which domain experts read a sample of real production output for each solution, weighted towards high-risk cases. Nothing else surfaces the errors nobody reported, and it is the first habit to go when a team gets busy.

Fix what is live and used before building more #

The pressure on an ML and AI team usually points at the next pilot. The order that pays is different:

1. Measure what is live. Build the one-page scorecard for the solutions already in production. Until it exists, every priority call is an opinion.
2. Fix the heavily used solutions with quality gaps. Volume multiplies every improvement, and every error.
3. Stop what is not used or cannot show value. Every live solution costs run money, review time and risk. Retiring one frees capacity for the next item.
4. Build the shared machinery once: logging, golden-set tooling, the evaluation harness and the release gate. Every new solution then starts measured, and the team stops rebuilding the same plumbing.
5. Then choose new use cases, on criteria that predict whether they will pay.

Of these, the cost of checking is the criterion most often skipped and the one that decides the most. AI pays off where a person can verify a result in a fraction of the time it takes to produce it: a reply a caseworker can read in thirty seconds, fields a clerk can confirm at a glance. Where checking takes as long as doing, the time saved goes into review, and the value case quietly disappears.

Every solution on the scorecard then carries one of four decisions, reviewed each quarter with leadership:

A first quarter can do all of this. In weeks one to four, list every solution with its owner, task, risk and usage from the logs. In weeks five to eight, build golden sets and baselines for the three most-used solutions and start the weekly review. In weeks nine to twelve, hold the first portfolio review: a decision for every solution, and a shortlist of new use cases scored on the criteria above.

Five questions for every solution #

The whole approach fits in one line: three numbers per solution, one release gate, one improvement loop and four decisions. Leadership should be able to answer five questions about every solution without asking the team:

• Who owns it, and what work does it support?
• How often is it wrong, and how do we know?
• How much of the eligible work goes through it?
• What changed in the outcome, compared with what?
• When it is wrong, who finds out, and how fast?

Those five answers also settle the question at the top. You switch off the solution whose answers say it costs more than it returns, and you can only do that once every solution has them.