Evaluating AI Vendors and Models: Selection Roadmap, Benchmarks, Model Cards, and Lock-In

Evaluating an AI vendor is not one decision — it is four intertwined ones: whether the vendor fits your requirements, whether the model's measured capability transfers to your workload, whether the vendor's documentation and disclosures hold up to scrutiny, and whether you can leave when the model you bought is retired. This guide walks through all four, in the order a procurement actually runs.

1. By the numbers

2. The central tension: scoreboard capability versus workload fit

Every AI vendor evaluation is pulled between two forces. On one side sits public evidence — benchmark scores, analyst positioning, model cards — cheap to consume and easy to defend in a steering committee. On the other side sits private evidence — proof-of-concept results on your data, your latency and cost envelope, your compliance posture — expensive to produce and impossible to outsource. Teams that over-weight the public evidence buy the model that wins the leaderboard; teams that run the process below buy the model that wins their workload. Those are frequently not the same model.

3. The selection roadmap: six stages, in order

A defensible AI vendor selection runs as a staged funnel, and the stages exist to keep expensive activities (pilots, legal review) from being spent on vendors that cheap activities (requirements screening) would have eliminated.

Two of these stages routinely get shortchanged. Stage 1 gets skipped because a vendor demo arrives before the requirements do, and the demo silently becomes the requirement. Stage 5 gets skipped because by the time due diligence starts, the organization has emotionally committed. The fix for both is procedural: no vendor conversation before the rubric is written, and no contract before the deprecation and exit questions have written answers. The commercial half of stage 6 — pricing structures, SLAs, indemnification — is covered in depth in /guides/ai-vendor-contracts-and-pricing; this guide focuses on the evaluation evidence that should feed it.

4. Reading analyst reports without outsourcing the decision

Most enterprise selections start from the big analyst firms' quadrant- and wave-style market reports. Used correctly they compress weeks of market scanning; used incorrectly they become the decision. Four reading rules keep them in their lane. First, read the inclusion criteria before the graphic: these reports typically require minimum revenue, customer counts, or feature scope, so a vendor that fits your use case perfectly may be absent for reasons unrelated to quality. Second, understand what the axes reward. Positioning frameworks blend execution measures (market presence, viability, delivery record) with strategy measures (vision, roadmap), and the weightings are the analyst's, not yours — a vendor placed modestly overall may lead on the one capability dimension your workload depends on.

Third, check the date. AI vendor positions shift with each model generation, acquisition, and partnership, and reports refreshed annually can lag the market they describe by most of a product cycle. Fourth, treat the report as a shortlisting input, never a verdict: it cannot see your integration constraints, your data, your team's skills, or your total cost at volume. The honest use of an analyst report is to generate the long list in stage 2 and to mine its capability criteria for your own rubric — then let your RFP and PoC, not the chart, pick the winner.

5. Benchmarks: what the numbers measure, and what they leak

Benchmark scores are the most quoted and least understood evidence in AI procurement, so it pays to know what the flagship benchmarks actually test. MMLU (Massive Multitask Language Understanding) tests a model's multitask accuracy — its world knowledge and problem solving — across 57 tasks including elementary mathematics, US history, computer science, and law.^[1] When it was introduced in 2020, the largest GPT-3 model beat random chance by almost 20 percentage points on average while remaining near-random on some socially important subjects such as law and morality — and, notably, the authors found models did not know when they were wrong.^[1] Frontier models now cluster near the top of the scale, which means MMLU today functions less as a differentiator and more as a floor check.

HumanEval, introduced alongside OpenAI's Codex model, measures functional correctness: the model writes Python from a docstring and passes or fails real unit tests. In the original 2021 paper, Codex solved 28.8% of problems where GPT-3 solved 0% and GPT-J solved 11.4% — and with 100 samples per problem, repeated sampling lifted Codex to 70.2%.^[7] That last number carries the buried lesson for buyers: the sampling and prompting protocol can move a score by more than the gap between competing vendors. A reported benchmark number without its evaluation protocol — model version, prompt strategy, number of attempts — is not comparable to anything.

Newer benchmarks push toward realism. SWE-bench evaluates whether a model can resolve 2,294 real GitHub issues drawn from 12 popular Python repositories — editing an actual codebase, often across multiple files, until the repository's tests pass. At publication in 2023, the best-performing model, Claude 2, resolved just 1.96% of issues.^[2] Scores have risen dramatically since, which illustrates the second thing buyers must track: benchmark saturation. A benchmark discriminates between vendors only during the window when scores are climbing; once the frontier clusters at the top, differences are noise, and once the test set has circulated for years, training-data contamination can quietly inflate results. Static, public benchmarks are, by construction, leaking into the models being tested on them.

Two institutional efforts partially correct for single-number thinking. Stanford's HELM (Holistic Evaluation of Language Models) project evaluates models multi-metrically — accuracy alongside calibration, robustness, fairness, bias, toxicity, and efficiency; before HELM, models had been evaluated on an average of just 17.9% of its core scenarios, a figure the project raised to 96.0%, exposing tradeoffs that headline accuracy hides.^[8] For the infrastructure layer, MLCommons' MLPerf Inference suite benchmarks how fast systems can process inputs and produce results using a trained model^[9] — the relevant scoreboard when you are buying serving hardware or a hosting platform rather than a model. And independent aggregators now re-measure vendor claims: Artificial Analysis runs its own evaluations across hundreds of models and providers, publishing an Intelligence Index (version 4.1.1 incorporates 9 separate evaluations) alongside the numbers procurement actually needs — cost per task, output tokens per second, and time per task.^[10]

Finally, note what no public benchmark measures: hallucination behavior on your domain, resistance to prompt injection, output stability across versions, and cost and latency at your traffic shape. Those properties determine production success, and they only fall out of the stage-4 proof of concept — which is why the PoC, not the leaderboard, should carry the most weight in the rubric.

6. Model cards and system cards: reading the vendor's own disclosure

Model cards were proposed by Mitchell and colleagues in 2018 as short documents accompanying released models — motivated by the observation that trained models were increasingly performing high-impact tasks in areas such as law enforcement, medicine, education, and employment, and that documentation should clarify intended use cases and 'minimize their usage in contexts for which they are not well suited.'^[6] The framework's distinctive demand is disaggregated evaluation: performance reported not as one number but across different cultural, demographic, or phenotypic groups and intersections of them.^[6] The genre has since grown into vendor system cards covering safety evaluations, red-teaming results, and alignment assessments — Anthropic, for example, publishes per-model system cards through a transparency hub, alongside a Responsible Scaling Policy evaluation process 'designed to systematically assess our models' capabilities in areas where they could pose catastrophic risks' before release.^[11]

For an enterprise team, a model or system card is due diligence raw material, and six sections deserve a structured read. Intended use and limitations: where the vendor itself says the model should not be used — deploying into an explicitly out-of-scope context converts the vendor's disclaimer into your liability. Training data disclosure: provenance signals for IP exposure and personal-data risk, usually the card's thinnest section — thinness is itself a finding. Disaggregated performance: whether metrics are broken out by domain, language, and demographic slice, or reported as one flattering aggregate.^[6] Safety evaluations: known failure modes and mitigations, which feed your risk assessment and contract safety clauses. License and usage terms: commercial-use scope, redistribution, attribution. Versioning and update policy: the bridge between documentation and the deprecation problem below.

Read cards with the obvious caveat in force: they are authored by the vendor, describe a snapshot in time, and understandably emphasize strengths. Formats vary widely across vendors, so a side-by-side read requires internal expertise rather than a template. The right posture is the one auditors take with management representations — a valuable, structured disclosure to be tested, not accepted. Request an updated card for every major version you adopt, and treat the gap between the card's claims and your PoC's measurements as a data point about the vendor's candor. How those disclosures feed a formal validation and monitoring regime is the subject of /guides/model-risk-management-guide.

7. Deprecation and lock-in: the part of the evaluation most teams skip

Every frontier vendor now retires models on a schedule, and the schedules are short enough to matter to procurement. Anthropic runs an explicit lifecycle — Active, Legacy, Deprecated, Retired — commits to at least 60 days' notice before retiring publicly released models, and fails requests to retired models outright; its public table shows the cadence in practice, with the March 2024 version of Claude 3 Haiku retired on April 20, 2026.^[4] OpenAI commits to at least 6 months' notice for generally available models and 3 months for specialized variants, warning that preview models may get as little as 2 weeks; its shutdown list currently includes gpt-4-turbo and gpt-3.5-turbo variants dated October 23, 2026.^[5] Microsoft's Foundry platform (which serves Azure OpenAI models) is the most programmatic: a generally available model gets an 18-month lifecycle set at launch, closes to new customers at 12 months, and returns 410 Gone at retirement — with at least 60 days' active notice, no retirement-date extensions, and a shorter 12-month lifecycle for partner models from Anthropic, DeepSeek, Fireworks, and Mistral AI.^[3]

Read those policies as an evaluator and three consequences follow. First, model choice is perishable: any capability comparison you run is a snapshot whose shelf life is set by the vendor's lifecycle clock, so the durable asset is your evaluation harness, not the winner it picked. Second, notice windows define your required migration speed — an organization that needs two quarters to re-validate a model in a regulated workflow cannot safely depend on a 60-day floor without contractual extensions or a platform whose dates are published years ahead. Third, deprecation risk compounds with behavioral drift: a forced migration is not a like-for-like swap, because successor models differ in tone, formatting, refusal behavior, and failure modes even when their benchmark scores are higher. Budget re-validation, not just re-pointing.

The architectural hedge is an abstraction layer: a gateway that exposes one internal API and routes to providers behind it, normalizing request and response formats, centralizing credentials, logging, and rate limits, and enabling fallback routing when a provider degrades. It is also how multi-vendor leverage becomes real — switching costs drop from a rewrite to a routing change, which changes the tone of every renewal negotiation. But price it honestly. A gateway is software you now own: it must track every provider's API changes, it flattens vendor-specific capabilities toward a lowest common denominator, it adds a latency hop, and fallback across models with different behavior can silently change output quality mid-incident unless you validate responses at the gateway. In regulated environments, routing also has to respect data-residency boundaries — failover to a provider in the wrong jurisdiction is an incident, not resilience.

8. Mapping vendors to the business function

One structural note keeps shortlists sane: the AI vendor landscape is organized less by technology than by business function, and evaluation criteria should shift with the function. Sales and marketing tools live or die on CRM integration depth and data hygiene; customer-support platforms on containment quality and escalation behavior; finance applications on auditability and explainability; supply-chain tools on forecast accuracy against your demand history; HR tools on bias testing and employment-law exposure. The six-stage roadmap applies everywhere, but the mandatory bar in stage 1 and the KPIs in stage 4 are function-specific — a single enterprise evaluation template usually produces a rubric that fits nobody. Write one rubric skeleton, then let each function's risk profile set the weights.

9. Honest objections

The strongest counterargument to everything above is that process has a price. A six-stage funnel with a private eval set and an abstraction layer can consume two quarters and real engineering headcount while competitors ship — and in a market where model capability improves this quickly, deferred value is a genuine cost, not a rounding error. There is also a serious case for deliberate single-vendor concentration: deep adoption of one provider's stack buys better support, earlier feature access, volume pricing, and none of the gateway's lowest-common-denominator tax. Many successful AI programs are, in practice, happily locked in.

Both objections deserve a real answer, and the answer is proportionality: the depth of this process should scale with the blast radius of the decision. A team piloting an internal drafting assistant does not need an RFP and a gateway; a bank routing customer-facing decisions through a model does. The non-negotiable core is small — written requirements before demos, a private eval set, a read of the deprecation policy, a tested exit path — and none of those four takes a quarter. Concentration, likewise, is a legitimate strategy when chosen with the exit cost measured, and a trap when discovered at renewal. The process is not about avoiding commitment; it is about knowing the price of the commitment before the vendor does.

10. The read

Treat AI vendor evaluation as the construction of three durable assets: a requirements rubric that outlives any one selection, a private evaluation harness that outlives any one model, and an exit path that outlives any one contract. Public benchmarks and analyst charts shortlist; model cards and system cards direct your scrutiny; your own PoC decides; and the deprecation clock — 60 days to 18 months depending on provider^[4][3] — guarantees you will run this loop again, so build it to be rerun. And because the model's output failures ultimately land on the buyer, the liability dimension of that choice — covered in /insights/ai-output-risk-and-liability — belongs in the same steering conversation as capability and cost.