Your AI Doesn’t Need To Be Perfect. It Just Needs To Beat the Baseline.
A large US mortgage lender now runs a generative model across more than 1.5 million documents every month. The system classifies forms and extracts over 90% of the fields the underwriting team needs, which lets them shrink manual review and push far more loans through the pipe without scaling headcount linearly.
No one waited for perfect extraction. They shipped because it was obviously better than the manual process they had before.
That is the pattern that separates AI teams who are in production from teams who are still stuck in pilots. The bar in most real environments is not “100% accuracy.” The bar is “better than the current baseline, with clear guardrails and a way to measure the lift.”
The 100% accuracy myth
In traditional software, it is natural to want hard guarantees. A feature either works or it does not. The instinct is to carry that over to AI and demand 99–100% accuracy from any model that touches a customer, a dollar amount, or a regulated decision.
You see the result in requirements documents and RFPs. People ask for “near-perfect” accuracy without being able to say compared to what, or at what cost, or with what human oversight.
There are two problems with that posture.
First, for most interesting problems the “perfect” system does not exist. Even in credit underwriting, where the literature is mature, serious work compares machine learning models to incumbent scorecards and looks at relative gains in predictive power and cost, not some abstract idea of perfection. In practice you trade off incremental accuracy against complexity, latency, explainability, and regulatory comfort.
Second, an aggregate accuracy score tells you very little about whether a system is safe or useful in context. Teams that work on evaluation at scale are blunt about this. They argue that reliability, error profiles, and cost often matter more than headline accuracy in production, and that accuracy-only optimization can make systems 4–11x more expensive without meaningful business benefit.
Regulators take a similar stance. Data protection authorities explicitly say that AI outputs are “statistically informed guesses” rather than facts, and that the principle is to label them as such and control how they are used, not to demand 100% statistical accuracy.
When you insist on perfection, you do not get safer systems. You get no systems, or you get unmeasured shadow use where no one is watching the error modes.
Define “better than what?”
The right question for a production AI system is almost always “better than what?” rather than “how close to 100%?”
There are three baselines that matter.
1. Manual baseline
How does a human team perform today on the same task. For underwriting this might be loans per underwriter per day and downstream error rate. For engineering this might be PRs per engineer per week and change‑failure rate. For customer support it could be average handle time, CSAT, and tickets per agent.
2. Rule-based baseline
Many workflows already have deterministic rules or heuristics. A fraud engine that flags anything over $X, a routing rule that sends any ticket with the word “refund” to a specific queue. A fair comparison tests whether the AI system beats that simple rules engine on the real metrics you care about.
3. Naive model baseline
In model-heavy environments there is often an incumbent logistic regression or gradient boosted tree. Again, the question is whether a more complex model or agent system is meaningfully better on business outcomes, not whether it wins a benchmark by a few points.
This is exactly how serious benchmark work is structured. The better guides insist on including a “do-nothing” or trivial baseline, reporting cost and latency alongside success rates, and checking that your system actually outperforms simple alternatives.
If your AI agent barely beats “call the underlying model a few times” or “run the legacy rules engine,” you do not have a product. You have research.
Measure before you touch anything
Once you are clear about the baseline, you need to actually measure it.
In our work at Lazer, we see AI projects fail more from missing instrumentation than from bad models. Teams launch something that “looks good in a demo” and realize months later they never captured clean before/after data.
A practical pattern is:
- Pick 2–3 primary metrics per use case.
Examples:
- Mortgage: cycle time per application, manual touch time per file, downstream delinquency rate.
- Customer service: CSAT, first‑contact resolution, average handle time, tickets per agent.
- Sales: meetings per rep per week, win rate, time from first contact to close.
- Collect a baseline window of data.
Four to six weeks of production data is usually enough to capture normal variability.
- Build eval sets from production failures.
Good evaluation harnesses are seeded with real tickets, real documents, real edge cases, not synthetic prompts that only exist in the lab. You want to know how the system behaves where your current process actually breaks.
- Track more than one metric.
Modern evaluation work consistently recommends multi‑metric views: task success, latency, cost, and reliability, not just accuracy. That is how you avoid “improvements” that win a leaderboard but lose you money.
This sounds basic. It is basic. It is also what most AI pilots skip.
How accurate is “enough”?
“Good enough” is not a single number. It depends on three things:
1. Impact of a wrong decision
If you are recommending a product, the cost of a bad suggestion is annoyance or a missed upsell. If you are approving a loan or suggesting a clinical action, the stakes are orders of magnitude higher. In regulated domains, serious teams treat the model as a decision aid and keep the final decision with humans.
2. Whether a human will review the output
A model that drafts emails for an agent who edits them can run with lower standalone accuracy than a model that auto‑sends. A summarization system that feeds into a doctor’s workflow can tolerate more ambiguity than one that triggers an automated prescription.
3. How well you can detect uncertainty and route it
If you have strong confidence scores, fallbacks, and escalation paths, you can let the system run on the easy 70–80% of cases and route the rest. If you cannot tell when it is about to be wrong, you are stuck holding it to a much higher accuracy threshold.
Our internal AI delivery guide is explicit about making this contract visible. Teams are encouraged to agree in writing on target accuracy ranges and the cost of chasing the last few points.
External work in reliability pushes the same idea. In healthcare deployments, practitioners care less about peak benchmark scores and more about whether the system is predictable, auditable, and easy to overrule when it is wrong. Those lessons translate directly to finance, government, and any enterprise environment.
So you rarely need “95%+ on every metric” as a blanket requirement.
You need:
- A clear threshold for when the AI operates autonomously.
- A clear rule for when it hands off to a human.
- Evidence that, under those rules, the overall system beats the baseline.
Deterministic software still does the heavy lifting
None of this works if you treat an LLM or an agent as the whole system.
The production pattern we see again and again is simple:
- Models sit inside deterministic rails.
- Typed schemas around model outputs.
- Validation layers that reject out‑of‑policy answers.
- Workflow engines that decide what happens next.
Business rules stay explicit:
- Eligibility criteria for loans live in code.
- Compliance checks live in code.
- SLAs and escalation thresholds live in code.
Data flows are boring on purpose:
- ETL pipelines, vector indexes, feature stores, and caches that feed AI components are built with normal engineering discipline.
- Logs, traces, and evals are wired in so you can see what the system is doing in production.
This is why “AI engineering” for enterprises looks a lot like regular engineering with some extra pieces. We still spend time on data schemas, CI/CD, observability, and rollback strategies. We just add eval harnesses, tool interfaces for agents, and routing logic between models.
The payoff is that you can safely run less‑than‑perfect models, because the rest of the system constrains the blast radius.
Humans in the loop where it matters
Humans are not a temporary crutch that you remove later. They are part of the design.
There are three places they show up in systems that last.
1. During development
Teams run eval cycles every sprint. They look at real examples of good and bad outputs, not just aggregate scores, and adjust prompts, tools, and workflows accordingly. This is closer to continuous QA than to a one‑time model test.
2. At decision points
- Underwriting: models pre‑score applications and propose decisions; underwriters review a subset based on risk and confidence thresholds.
- Claims: models propose payouts or flags, but any case above a certain amount or ambiguity score goes to a human queue.
- Customer service: AI triages, drafts responses, and resolves simple tickets; agents own the escalations and complex accounts.
3. In monitoring and improvement loops
Good deployments give operators a way to mark outputs as wrong, flag new edge cases, and feed that data back into the eval harness. Without that, you get gradual degradation as traffic shifts and models drift.
The effect is the same across domains. You are not asking the AI to be perfect. You are asking it to do the cheap, repetitive pattern‑matching at scale, and you are designing the system so that humans see and correct the errors that actually matter.
What “better than baseline” looks like in practice
Three patterns make this concrete.
Mortgage operations
Beyond the document processing example that auto‑extracts 90%+ of key fields, some lenders are now running AI risk models alongside traditional scorecards. In one public case, a regional US bank used AI‑based models to improve mortgage default prediction accuracy by more than 15% compared to their legacy methods.
That did not mean the AI had perfect foresight. It meant:
- Fewer false negatives in risky segments.
- Better early warning on emerging delinquencies.
- More targeted follow‑up and restructuring.
The system earned its place by improving portfolio performance relative to the incumbent baseline, while still routing edge decisions to human credit officers.
Customer service
A European consumer fintech reported that its AI assistant handled over 2.3 million customer requests in its first month, took over roughly two‑thirds of incoming chats, cut resolution times from about 11 minutes to under 2, and reduced repeat inquiries by around 25%. The company estimated about $40 million in profit improvement, even after keeping human agents in the loop for complex edge cases.
Telecom and commerce providers have similar numbers. One large telecom operator’s virtual agent now resolves about 70% of incoming inquiries on its own and cut cost‑per‑chat by roughly two‑thirds. In e‑commerce, a major marketplace reports that AI chatbots handle about three‑quarters of online queries, saving on the order of $150 million per year in support costs while lifting customer satisfaction.
None of these deployments waited for “human‑level” accuracy on every possible question. They picked a set of intents where AI could confidently outperform the queue and escalation system they already had, instrumented the outcomes, and expanded from there.
Voice and claims
In insurance and voice‑driven ordering, a pattern we see often is that teams start with a cost‑saving goal and discover unexpected upside in throughput or revenue.
One insurance leader put it bluntly in a public interview: “I am okay if AI gets 90% of claims right. But how do I know which 10% it got wrong?” Their solution was not to hold out for a future 99.9% model. It was to:
- Use AI for straightforward claims.
- Design clear uncertainty thresholds.
- Route anything above those thresholds to humans.
In parallel, a large quick‑service brand found that customers ordered more through an AI drive‑through agent than with a human on the headset. The project started as a labor and throughput experiment. The main long‑term value was an increase in average ticket size per visit.
In both cases, the systems earned their place by beating the baseline on concrete business metrics, within a workflow that made it easy to see and correct the 10–20% of cases where the AI struggled.
A practical playbook: beating the baseline in 90 days
If you want to deploy AI into production without getting stuck in perfectionism, a simple loop works well.
1. Pick one narrow workflow
Choose something with clear start and end states, not “make everything more efficient.” Examples: triage incoming support emails, pre‑classify mortgage docs, generate first‑pass contract summaries.
2. Instrument the current process for 4–6 weeks
Capture volumes, handle times, error or rework rates, and any revenue or satisfaction signal you can get.
3. Design the rails before you pick a model
- Data pipelines and schemas.
- Where the model plugs in.
- What counts as a safe action.
- Where humans review or approve.
4. Build the AI‑assisted version and an eval harness in parallel
Use real production data to assemble eval sets. Include easy cases and the failures your current process trips over.
5. Agree on “go / no‑go” thresholds up front
Examples:
- “We ship if first‑contact resolution is +10% with no drop in CSAT.”
- “We ship if underwriting cycle time drops 25% while delinquency rates hold steady.”
- “We ship if agents can handle 30% more tickets with the same headcount, with change‑failure rate flat.”
6. Run a limited rollout against the baseline
A/B test or pilot with one region, one queue, or one product line. Watch your metrics, but also read raw examples weekly with the operators who live in the workflow.
7. Iterate or kill quickly
If the AI‑assisted flow beats the baseline and operators trust it, expand its scope. If it does not, treat that as a signal to adjust the problem framing, not as evidence that AI “doesn’t work here.”
This is essentially how we run AI builds and FDE‑style engagements at Lazer. Short discovery to understand workflows and metrics, separation between experimentation and production engineering, evals wired in from day one, and progressive rollout rather than big‑bang go‑lives.
It is not glamorous. It is also how you turn a probabilistic model into a system that reliably beats the baseline.
The Real AI Advantage Is Measured, Not Perfect
The teams that win with AI are not the ones waiting for flawless models; they are the ones disciplined enough to define the baseline, measure the current process, build deterministic guardrails, and let humans own the decisions that matter most. Perfection sounds safe, but in practice it often delays value and pushes experimentation into the shadows. A better standard is simpler and more useful: prove the AI-assisted workflow performs better than today’s process on the metrics the business actually cares about, then expand only where the evidence supports it. That is how probabilistic models become reliable operating systems for real work—not by being perfect, but by consistently beating the baseline.
