Why most AI pilots never reach production
The gap between an impressive demo and a dependable system is evaluation, escalation and ownership — not model capability.
Demos optimise for the happy path
A pilot is usually built against clean inputs and reviewed by the person who built it. Production has malformed data, ambiguous requests, systems that time out, and users who will not read a disclaimer.
The capability was never the constraint. The constraint is what happens on the twelve percent of cases the demo never touched.
Three things production systems have
An evaluation set: a fixed collection of real cases with known good outcomes, run on every change, scored. Without it, quality drift is invisible until a customer finds it.
An escalation path: a confidence threshold beneath which the workflow hands to a person, with the context attached. Systems that cannot say 'I don't know' fail loudly and expensively.
An owner: a named person accountable for the output, the same as any other operational process.
Choose boring problems first
The strongest early candidates are repetitive, high-volume, tolerant of a short delay, and cheap to verify. Lead triage, research summarisation, reporting assembly, data normalisation.
Start where being wrong is recoverable. Earn the right to automate the work where it is not.