AI Pilots and Maturity: Picking the First Project, 90-Day Metrics, and Why Pilots Fail

An AI pilot is a purchase of information: you are paying 90 days of attention to learn whether a system deserves production investment. So run it like one — pick a workflow with a measurable baseline, agree the metrics and the decision rule before day one, and end with an explicit scale, iterate, or kill call. Pilots rarely fail on model quality; they fail on decision discipline.

1. By the numbers: adoption is broad but shallow

Read those four numbers together and the strategic picture is clear. AI adoption in US business is real but shallow: most firms that use AI at all confine it to a handful of functions, and among very large firms in knowledge-intensive sectors — where use rates reach 50%–60% of firms in Information, Professional Services, and Finance — the differentiator is no longer whether you run pilots but whether pilots convert into scaled, instrumented functions.^[1] If your organization is still choosing its first serious pilot, you are not behind the market. You are, however, about to make a decision that sets the pattern for every AI investment that follows.

2. What a pilot buys you: information for a go/no-go

The most useful framing available comes from NIST's AI Risk Management Framework. Its MAP function "establishes the context to frame risks related to an AI system," and NIST is explicit about what that context work is for: after completing MAP, "Framework users should have sufficient contextual knowledge about AI system impacts to inform an initial go/no-go decision about whether to design, develop, or deploy an AI system."^[4] That is exactly what a pilot is — a structured way to acquire the contextual knowledge that a go/no-go decision needs. If your pilot plan cannot name the decision it will inform, it is not a pilot; it is a demo with a budget line.

This reframing changes what counts as success. A pilot that reveals your data cannot support the use case, or that unit costs at production volume would be prohibitive, has succeeded — it bought you a cheap no before you paid for an expensive one. The distinction between a demo-driven pilot and a decision-driven pilot is the central tension of this entire topic.

3. Picking the first project

Three filters do most of the work of selecting a first pilot. First, the workflow must already produce a per-unit metric — issues resolved per hour, invoices processed per day, days from claim to decision. Second, the data the system needs must be accessible today, not after a data-platform project. Third, the workflow must tolerate error gracefully: a human should review or absorb the model's mistakes without customer or regulatory harm. A candidate that fails any of the three is a second or third pilot, not a first.

The first filter is why customer support became the early proving ground for generative AI. In the Brynjolfsson, Li, and Raymond field study of 5,172 customer support agents, access to a generative AI assistant raised productivity by 15% on average, measured as issues resolved per hour — a number that exists only because support operations log per-unit throughput natively.^[3] The same study found the gains concentrated among less-experienced agents, while the most experienced workers saw small speed gains with small quality declines — a distribution worth remembering when you choose which team pilots first and how you set expectations with your best performers.^[3]

It is also worth knowing where your peers actually deploy. Among US firms using AI, the most common business functions are sales and marketing, strategy and business development, and IT.^[1] That concentration is informative in both directions: these functions have abundant text-heavy tasks and measurable outputs, which makes them natural first pilots — and it also means vendor tooling and reference patterns there are the most mature, which lowers your integration effort.

With candidates that pass the three filters, prioritize on two axes: business impact and implementation effort. The quadrant logic is familiar, but the discipline is in scoring honestly — implementation effort must include data preparation, integration, security review, and change management, not just model work, because those are the line items that pilots systematically underestimate.

4. The 90-day measurement plan

Ninety days is long enough to observe a metric trend and short enough to keep attention. But direct financial ROI rarely materializes inside the window — the honest posture is to measure leading indicators now and project financial returns from them, explicitly, with the assumptions written down. For the full treatment of how pilot indicators roll up into a defensible ROI model, see /guides/measuring-ai-roi-guide; this section covers what to instrument inside the pilot itself.

• Operational metrics — the per-unit throughput or quality measure the workflow already logs: cycle time, error or rework rate, units handled per person-hour. These are your primary evidence because they move within 90 days.
• Adoption metrics — share of eligible users active weekly, share of AI outputs accepted versus overridden, time to first productive use. A pilot with strong operational numbers and collapsing adoption is telling you the scale-up will fail.
• Cost and effort metrics — actual spend per unit of work (inference, licenses, review labor) and the integration hours consumed. These feed the production unit-economics projection, which is where scale decisions are won or lost — the full framework lives at /insights/enterprise-ai-tco-guide.

Baseline before day one, or the pilot is unmeasurable. Capture at least four weeks of the primary metric under current-state operations, from the same logs the pilot will report against. Then run stage gates at 30, 60, and 90 days with pre-assigned questions — the point of interim gates is to kill or fix early, while sunk costs are small.

Attribution deserves more care than it usually gets. If the pilot runs while a process change, a seasonal peak, or a reorg is also in flight, the metric movement is contaminated. Where volume allows, hold out a comparison group — a queue, region, or team that keeps current-state operations — and compare trends rather than absolute levels. Where it does not, at minimum document the concurrent changes in the decision memo so the readout is honest about confidence. And police scope: every mid-pilot feature addition resets your ability to say what caused the result.

5. Why pilots fail: twelve causes, four clusters

Catalogs of pilot failure causes run long — unclear objectives, weak sponsorship, dirty data, over-scoping, talent gaps, thin budgets, no change plan, undefined metrics, vendor dependence, ignored compliance, unscalable architecture, no iteration loop. Useful as a checklist, but the twelve collapse into four root clusters, and each cluster has one owner who can prevent it.

The adoption gap deserves particular respect because the national data says augmentation is the dominant mode: 66% of AI-using firms rely on AI solely to augment tasks, and AI-related employment decreases occur in only 2% of firms.^[1] Augmentation means your return arrives only through changed human behavior — the model can be flawless and the pilot still fails if the humans route around it. That is a change-management problem with a known playbook; we cover it in depth at /guides/ai-change-management-adoption, and your pilot plan should budget for it from day one rather than discovering it at day 60.

6. A working maturity model: ad hoc to transformative

Maturity models attract deserved skepticism, so let us be precise about what this one is: a sequencing tool of our own construction, not an industry standard or a certification ladder. Its only job is to answer one question — given where the organization actually is, what is the next right move? Used that way, four stages are enough.

Two honest observations about using it. First, the national picture suggests most adopters sit at stages one and two: when 57% of AI-using firms confine AI to three or fewer business functions and 65% limit worker use to three or fewer tasks, integrated enterprise-wide AI is the exception, not the norm.^[1] Second, stages are per-domain, not per-company — your support organization can be at stage three while finance is at stage one, and pretending the enterprise has a single maturity score obscures exactly the sequencing information the model exists to provide.

7. Who approves what: the stakeholder map

More pilot calendar time is lost to unsequenced approvals than to model problems. AI initiatives cut across data, infrastructure, legal, risk, and the business, and each constituency holds a genuine veto over some part of the work. Map them before kickoff, not when the first blocker surfaces.

• Executive sponsor — approves scope, funding, and the strategic fit; signs the go/no-go memo.
• Data owners — approve data access and use; the single most common source of hidden schedule risk, so start this approval first.
• IT and platform teams — approve infrastructure, security posture, and integration points into production systems.
• Legal and compliance — approve regulatory exposure, vendor contract terms, and data-protection obligations.
• Risk and model validation — approve model performance evidence, bias testing, and human-oversight design for consequential decisions.
• Operational leads and end users — approve workflow fit; without their sign-off, expect the adoption gap to do its work.

Put the map in a RACI matrix in the pilot charter, run the slow approvals (data access, security review) in parallel with pilot setup, and keep the same matrix alive for the scale decision — the approvers who felt consulted during the pilot are the ones who move quickly when you ask for production sign-off. NIST's MAP guidance points the same direction: incorporating perspectives from a diverse internal team and from actors outside the deploying team is what makes the context-mapping — and therefore the go/no-go — trustworthy.^[4]

8. Honest objections

"Pilots are theater — just deploy and iterate." For low-risk assistive tooling, this is substantially right: a coding assistant or a meeting summarizer does not need a 90-day charter, and over-governing cheap experiments is its own failure mode. The pilot discipline in this guide is for systems that embed in workflows, touch customer or regulated outcomes, or carry production-scale cost commitments — where the information a pilot buys is genuinely expensive to acquire any other way. Match the governance weight to the decision weight.

"Ninety days is too short to prove ROI." Correct — which is an argument for honest metric design, not longer pilots. Leading indicators move in 90 days; dollars follow over quarters. The failure is claiming dollar ROI a pilot cannot show, then losing credibility when finance audits it. Project financial returns from measured operational deltas, show the assumptions, and let the projection be challenged — that discipline is exactly what separates a fundable scale case from an optimistic slide.

"Maturity models are consultant-ware." Often true: a maturity model used to produce a benchmark score and a transformation program is decoration. The defensible use is narrower — it stops two specific, expensive errors: attempting a stage-three deployment on stage-one foundations (no governance, no data readiness), and buying stage-four platform commitments while the organization is still learning to run one pilot well. If the model is not changing your next investment decision, discard it.

9. The read

Treat pilots as decision instruments and maturity as sequencing, and the operating pattern follows. Run a small portfolio — two or three decision-driven pilots, not one bet-the-year project and not a dozen ungoverned demos. Pre-commit the gates and the decision rule so the day-90 memo writes itself. Fund scaling only where three tests pass together: the operational metric moved against a clean baseline, adoption held without executive pressure, and projected production unit economics survive contact with the TCO model. And bank the losses properly — a killed pilot with a written memo is reusable organizational knowledge; a quietly abandoned one is pure cost.

The prize for getting this right is not the first pilot's ROI, which is usually modest. It is the repeatable capability — charter, baseline, gates, memo — that turns every subsequent AI investment decision from a faith-based debate into an evidence-based one. That capability is what moves an organization from stage one to stage two, and it costs far less than the pilots it will save you from.

10. How to apply this