A demo shows the model handling a case well. Production asks a harder question: what happens across ten thousand cases, including the ones nobody anticipated, when a real decision depends on the answer?
Pilots that stall usually stall at exactly that boundary. The technology worked. What was missing was everything around it.
Start from a task with a number attached
The most common cause of a stalled pilot is that it was never aimed at anything specific. "Explore what AI could do for us" produces an interesting demo and no decision, because there is no criterion for success.
Pick a task that already has a measurable cost: hours per week, error rate, backlog age, time to first response. Write the current number down before starting. That number is what makes the eventual conversation about whether to roll it out a factual one rather than a matter of impression.
Ground the model in your own data
A general-purpose model knows a great deal about the world and nothing at all about your business. Asked a question about your operations, it will produce something fluent and unreliable.
Retrieval changes the shape of the problem. Instead of asking the model to recall, you retrieve the relevant records and documents first and ask it to answer from those. This does three useful things at once: accuracy improves, answers can cite their source, and updating knowledge becomes a matter of updating documents rather than retraining anything.
Design for being wrong
This is the part that separates pilots from systems.
The model will be wrong sometimes. That is a property of the technology, not a defect to be engineered away, and a design that assumes otherwise will fail in production in ways that damage trust quickly.
What works:
- Confidence thresholds. Handle the clear cases automatically; route the uncertain ones to a person. The threshold is a business decision about risk tolerance, and it should be adjustable without a deployment.
- Visible sources. When an answer cites the record it came from, a reviewer can check it in seconds instead of re-doing the work.
- A real escalation path. Not a dead end that says the assistant cannot help, but a handover that carries the context across.
- Output logging. You cannot improve what you did not record. Log what the model produced, what the human did with it, and where the two diverged.
Keep the provider replaceable
Model capability and pricing are both moving fast. Wiring one vendor's API directly into a hundred call sites is a decision you will want back within a year.
Put provider access behind an interface of your own. It costs very little at the start and preserves the ability to switch, run an evaluation across two providers, or move a workload to a cheaper model once you know how hard the task actually is.
Measure, then decide
At the end of the pilot, compare against the number you wrote down at the beginning. Include the cases the system handed back to a person — those are part of the cost, not an exception to it.
Sometimes the answer is that the automation handles seventy per cent of volume and the remaining thirty still needs a person. That is often an excellent result. It is only disappointing against a promise nobody should have made.