How to run an AI pilot project that earns a real decision

Three things get called a pilot and only one is. A demo shows a capability on curated inputs, usually to build internal appetite, and it is genuinely useful for that. A proof of concept answers a technical feasibility question: can this model, on our data, hit acceptable quality at all. A pilot answers a commercial one: run against real volume, with real users, does this beat the current process by enough to justify operating it. Confusing the three is the single most common way six weeks of budget produces nothing anyone can act on.

The tell is what happens at the end. If the output is a meeting where people say the results look promising, you ran a demo. If the output is a number next to a threshold agreed in advance, and a person whose job it is to say yes or no, you ran a pilot. The second version can end in a no and still have been worth every hour, because a documented no stops a much larger spend.

This framing also fixes the budgeting question. You are not buying software during a pilot. You are buying the information required to decide whether to buy software. That reframes what counts as expensive: a pilot that costs a fraction of the eventual build and reliably prevents a bad build is cheap, and one that costs less but produces an ambiguous result is a waste at any price.

A pilot needs a task with edges. The practical test is whether you can name who does it today, how many times a week it happens, how long one instance takes, and where the output goes next. If any of those four is vague, the scope is still a theme and the pilot will spend its first three weeks discovering what it was supposed to be.

Beyond having edges, the workflow needs four properties. Volume, because a task that runs eleven times a month cannot generate a signal in six weeks. Tolerance for error, because the first version will be wrong sometimes and you want that landing somewhere recoverable rather than in front of a customer or a regulator. A measurable status quo, because improvement is meaningless without a baseline you already trust. And existing data, because a pilot that starts with a data collection project is a data project wearing a pilot's name.

There is also a political property that matters more than teams expect. Somebody in the business has to actually want it. A workflow chosen because it looked good in a strategy deck, handed to a team that never complained about it, gets polite cooperation and no adoption, and the pilot measures the enthusiasm rather than the technology. Pick the task that a specific person is visibly tired of doing.

The strongest candidates tend to be unglamorous: extraction from documents that arrive in inconsistent formats, first-draft responses in a queue where a human still approves, triage and routing, reconciliation between two systems that disagree. They share a shape, which is high volume, low individual stakes, a human checkpoint, and a number that already exists in a dashboard somewhere.

A usable success criterion has four parts: the metric, the baseline it is measured against, the threshold that counts as a win, and the person who owns the measurement. Something like: first-response drafting currently takes an agent nine minutes on average across roughly 400 tickets a week, measured by the support lead from the helpdesk export. The pilot succeeds if median handling time drops below five minutes with approval-rate quality holding at or above the current level. That sentence is boring, and it is the entire difference between a decision and an argument.

Write the failure condition with the same care, because it is the part everyone skips. Name in advance what result would mean stop, and get the decision-maker to agree to it while nobody is invested yet. Without that, a marginal outcome always gets read generously, and a project that should have ended at week six drifts into a second phase on the strength of enthusiasm alone. Most of the abandonment in the S&P Global numbers is that drift eventually hitting a wall, later and more expensively than it needed to.

Two things belong next to the threshold. A cost ceiling, expressed per unit of work rather than as a monthly total, because per-unit is what tells you whether the economics survive ten times the volume. And a quality floor that is separate from the speed target, since almost any latency or cost number can be hit by degrading output, and a pilot that optimizes the metric you wrote while quietly wrecking the one you did not is a familiar and avoidable outcome.

This is also the honest gate on whether to run a pilot at all. Being unable to write the baseline usually means the process is not instrumented, and the correct first project is instrumentation. Being unable to name the decision-maker usually means no one has actually agreed to change how the work is done, which no pilot result will fix.

Production conditions means the real data with its real gaps, the actual users doing their actual jobs, and the integration points that will exist afterwards. The reason is not rigor for its own sake. It is that the interesting failures live exactly there: the malformed records, the edge case that represents four percent of volume and eighty percent of the difficulty, the handoff that turns out to need a permission nobody has. A pilot on clean data finds none of them and reports a number you cannot rely on.

Four things need to exist by the end, and building them during the pilot is the point rather than overhead. An evaluation harness that runs in CI, so quality is a test rather than an impression, and so the number can be reproduced after a model or prompt change. Guardrails on input and output. A hard cost ceiling, so a runaway loop cannot turn into a surprise invoice. And a documented fallback to the old process, which is what makes it safe to put the thing in front of real users at all.

Six weeks is the shape we have settled on for one production feature: long enough to hit real edge cases and get a stable measurement, short enough that the decision stays in front of the people who authorized it. Longer engagements tend to lose their decision-maker to a reorganization or a budget cycle, which is its own quiet way for a pilot to fail.

Keep the code in your repositories from the first commit. It costs nothing during the pilot and decides what a yes is worth: if the work sits in a vendor's environment until an end-of-engagement handover, a positive result buys you a dependency rather than a system, and the handover event fails far more often than a handover process does.

At the end you should be able to state four things in a paragraph: what the baseline was, what the system achieved against it, what it costs per unit at real volume, and what breaks if you triple that volume. A pilot that produces those four sentences has done its job whether the verdict is build, buy or stop.

A clean no is a good outcome and should be treated as one internally, because the alternative was finding out after the full build. What you keep from a negative pilot is real: an instrumented baseline you did not have before, an evaluation harness that outlives the attempt, and a documented reason this particular workflow resists automation, which stops the same idea being re-proposed every two quarters.

The failure mode to watch for is neither yes nor no. It is a result everyone can interpret in their own favour, which happens whenever the threshold was written after the fact or never written at all. That ambiguity is what turns into the abandonment statistics: not a project that failed, but one that was never able to tell anyone it had succeeded.