Why your AI pilots never reach production

What are the three reasons, in order of frequency?

This is measured, not anecdotal. RAND, interviewing 65 data scientists and engineers with five or more years building production models, reports that by some estimates more than 80% of AI projects fail – about twice the rate of IT projects without AI. MIT NANDA found 95% of enterprise generative AI pilots returning nothing measurable.

Both point at the same cause, and it is not the model. RAND found the problem being misunderstood or miscommunicated, and systems optimised for the wrong metric or built outside the real workflow. MIT concluded the divide is "not driven by model quality" but by approach. That matches the pattern in the systems I run, and the specific technology in use rarely appears in it.

1. The use case was chosen because it was interesting, not because it was expensive. Somebody demonstrated something impressive, and the organisation worked backwards from the demonstration to a justification. The result works and nobody misses it when it stops.
2. The data could not support it. Not missing data – data that exists but is inconsistent, ambiguously defined, or means different things in different systems. This is discovered late because everyone assumes their own data is fine.
3. Nobody owned the decision to go live. The pilot succeeded. Then the question of who accepts the risk of running it in front of customers had no answer, so it sat in a pilot state indefinitely, which is a decision by default.

How do you tell which one you have?

Why does choosing a different model rarely help?

The most common response to a stalled project is to try a better model. It is satisfying because it is tractable, and it almost never addresses the actual constraint.

If the use case was not worth doing, a better model produces a better solution to a problem nobody had. If the data is inconsistent, a better model learns the inconsistency more accurately. If nobody will own the decision to go live, model quality was never what was blocking it.

The exception is genuine capability limits, where the task was simply beyond what was available when you built it. That does happen, and it is worth re-testing periodically. It is just far rarer than it gets blamed for.

Why do organisations keep choosing the wrong use case, given how well documented this is?

The failure mode described above is not obscure. It has been measured repeatedly, and it keeps happening anyway, which suggests the cause is not ignorance.

Dataiku and the Harris Poll, surveying 900 CEOs across eight countries, found 80% saying their job is at risk if AI fails, and 87% saying they would stake their job on delivering results from their AI initiatives. That is a specific, personal incentive to show something working quickly and visibly, and a demonstration is a much faster way to produce that feeling than a boring process that takes months to show its value in the numbers.

The honest version of this section is that the pressure is real and understandable, and it still produces the wrong selection. A visible demo satisfies the pressure in the short term and does nothing for the actual return, which is exactly the gap the RAND and MIT findings describe.

What does "the data could not support it" actually look like?

Abstractly this sounds like a data-quality problem you would notice. In practice it is quieter than that, and it is usually discovered only after something has already gone live.

• The same field means different things in different systems. A "closed" deal in the CRM and a "closed" case in the support desk are not the same kind of closed, and a system trained across both learns a blended, wrong definition.
• Free-text fields are filled inconsistently. One region writes a reason code as a sentence, another as an abbreviation, a third leaves it blank. Anything trained on that field learns the inconsistency as if it were signal.
• Historical records reflect a process that no longer exists. The data is accurate for how the company worked two reorganisations ago, and nobody updated it because nothing downstream depended on it being current, until now.

None of these show up in a spreadsheet of missing values. They show up when the system is wrong in a way nobody can immediately explain, which is usually well after launch.

What does working ownership actually look like, in contrast?

It is easier to describe the failure than the working version, so it is worth being specific about what "someone owns it" means in practice.

• One named person, not a committee, who can say yes to putting the system in front of customers.
• A defined escalation path for when the system is wrong, that a frontline person actually knows about.
• A rollback that has been run at least once before launch, not just documented.
• A standing slot, weekly or monthly, where that person reviews what the system actually did, not just whether it is technically running.

None of this is expensive. It is the difference between a decision someone made and a decision that happened by default because no one made it.

What if more than one of the three causes is present at once?

Often all three are, and the instinct is to fix whichever feels most tractable, which is usually the wrong order.

Fix selection first, because it determines whether the other two are worth doing at all. There is no point hardening the data or naming an owner for a use case that was not worth building. Fix data second, because an owner cannot make a good go-live decision on numbers they cannot trust. Naming an owner last still works, because the decision they are being asked to own is now a real one rather than a guess.

Is starting over ever the right call?

Sometimes, and it is worth saying plainly rather than implying every stalled project can be salvaged with the right sequence of fixes.

If the use case was wrong, restart the selection rather than trying to rescue the build. If the data problem is structural, such as a source system being replaced anyway, it can be cheaper to wait than to engineer around data that is about to change. A full post-mortem before deciding which applies is worth doing properly. See our AI pilot failed, what now for the seven-question version of that post-mortem.

What does unblocking actually involve?

1. Re-select the use case against cost and frequencyThe right first project is boring, expensive and repetitive. If it is exciting, be suspicious.
2. Test the data before designing anythingNot whether it exists. Whether the same field means the same thing everywhere it appears.
3. Name who accepts the riskBefore the build, not after. The absence of this name is why pilots become permanent.
4. Define what "working" means, numericallyAgreed in advance, or evaluation becomes an argument about vibes that nobody can win.
5. Ship the unglamorous one firstProve the pipeline end to end on something low-stakes before attempting the ambitious thing.

Frequently asked questions

Related