AI Output Risk and Liability: Hallucination Exposure, Indemnification, and Intellectual Property

When an AI system's output is wrong, the loss almost always lands on the enterprise that deployed it, not the vendor that built the model. Vendor terms place output evaluation on the customer, the indemnities vendors do offer cover intellectual property claims rather than wrongness, and copyright may not protect what the model wrote. Managing that exposure is a design problem before it is a contract problem.

This piece walks the four questions a CIO, general counsel, and procurement lead need to answer together: where in your business a wrong output actually hurts, what the major vendors' own published terms allocate to whom, why benchmark scores are not warranties, and what happens to the IP status of content your models produce. The short answer running through all four: the contract narrows your exposure at the edges, but the middle of the risk — a confidently wrong output acted on by your business — stays with you, and only operational controls reduce it.

A risk-tier framing: annoying, actionable, regulated

Start with vocabulary. NIST's Generative AI Profile names the failure mode confabulation: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived"^[1]. The framing matters because it treats wrong output as an inherent property of the technology class, not a defect a patch will remove. A language model is a probabilistic text generator; some residual error rate survives every mitigation. So the governing question is not "how do we get to zero" but "where does a wrong output land, and what happens next."

Three questions sort any AI surface into a risk tier. First, who acts on the output — a trained reviewer who treats it as a draft, or a downstream system or customer who treats it as fact? Second, is the resulting action reversible — can an editor fix the copy before it ships, or has a payment moved, a diagnosis been influenced, a filing been submitted? Third, is the surface regulated — does a sector regulator, a disclosure regime, or a professional duty of care attach to statements made there? Each "yes" moves the surface up a tier, and the tiers demand different control postures, not just different levels of the same one.

The practical consequence is that hallucination risk is a property of the deployment surface, not of the model. The same model generating marketing variants and answering benefits questions for employees carries two different liability profiles. That argues for risk-tiering your surfaces and routing accordingly: permissive review for tier one, engineered validation for tier two, and mandatory human sign-off with logged provenance for tier three. The sign-off pattern is a discipline of its own — see /guides/human-in-the-loop-agents for the design patterns and /guides/agent-governance-guide for how to make the tiering an auditable policy rather than a slide.

What the vendors' own terms actually say

The liability allocation question — who pays when the model is wrong — has a clearer answer than most procurement teams expect, because the major vendors have published it. The pattern across published terms is consistent: responsibility for output accuracy sits with the customer, while the vendor's indemnity, where one exists, covers third-party intellectual property claims — a different risk entirely.

Anthropic's commercial terms state it directly: "It is Customer's responsibility to evaluate whether Outputs are appropriate for Customer's use case, including where human review is appropriate, before using or sharing Outputs"^[4]. Customers must also notify their users that factual assertions in outputs should not be relied on without independent checking. What Anthropic does take on is IP defense: it "will defend Customer" against claims "alleging that Customer's paid use of the Services … or Outputs generated through such authorized use violates any third-party intellectual property right"^[4]. The exclusions are where negotiation attention belongs: the indemnity does not apply to the extent a claim arises from customer modifications to outputs, combination with non-Anthropic technology or content, customer-provided inputs, use the customer knew or should have known was infringing, practice of a patented invention contained in an output, or trademark use of an output in commerce^[4].

Google Cloud publishes a two-pronged structure. The training-data indemnity "covers any allegations that Google's use of training data to create any of our generative models utilized by a generative AI service, infringes a third party's intellectual property right"^[2]. The generated-output indemnity is a "second layer of protection" — notable because, as Google's own announcement puts it, "the generated output is created by our customers in response to prompts or other inputs that they provide to our services"^[2]. And it is conditional: it "only applies if you didn't try to intentionally create or use generated output to infringe the rights of others" and expects customers to be "using existing and emerging tools, for example to cite sources to help use generated output responsibly"^[2].

Microsoft makes the conditionality most explicit of all. Its Customer Copyright Commitment "describes Microsoft's obligation to defend customers against certain third-party intellectual property claims relating to Output Content" — but for Azure OpenAI, coverage holds only if the customer "must have implemented all mitigations required by the Azure OpenAI documentation"^[5]. Those required mitigations are specific and technical: a metaprompt "directing the model to prevent copyright infringement in its output," a retained testing and evaluation report from red-teaming or systematic measurement designed to detect reproduction of third-party content, and — per use case — protected-material filters configured on in filter mode for text generation and code scenarios^[5]. If a customer tenders a claim, it must demonstrate compliance. An indemnity your platform team has silently invalidated by turning a filter off is not an indemnity.

Benchmark scores are not warranties

Vendor sales decks increasingly lead with hallucination benchmarks, and buyers increasingly quote them back in requirements documents. Both moves misunderstand what a benchmark score is. Public factuality benchmarks measure narrow, specific constructions — adversarial general-knowledge questions, grounding against a supplied document — under fixed prompting conditions, and scores are sensitive to dataset version, prompt template, and scoring method. Two impressive numbers from two vendors are usually not measuring the same thing, and neither is measuring your workload. We cover the major factuality benchmarks, what each actually measures, and the current sourced numbers in the benchmark section of /guides/hallucination-control-guide — the short version is that even frontier models fail a meaningful fraction of grounded-response tasks, which is exactly why the vendor terms above read the way they do.

The procurement translation: treat a benchmark score as a screening signal, never as a contractual performance level. If you want hallucination performance in the contract, three moves make it enforceable. Name the public benchmark, dataset version, and scoring method rather than accepting a vendor-run internal number. Require disclosure of evaluation conditions — prompting, sampling, any retrieval augmentation — so a claimed score is reproducible. And anchor acceptance on a pilot against your own domain data with your own annotators, because your risk tier is defined by your surfaces, and no public benchmark samples them. A vendor that resists all three is telling you the marketing number is not load-bearing — which is useful information.

The IP status of what your models produce

IP risk from generative AI runs in two directions. The inbound direction — does the output infringe someone else's rights — is the one the vendor indemnities above address. The outbound direction gets far less attention and is arguably more strategically important: can you protect what the model produces? In the United States, the Copyright Office's registration guidance answers with a human-authorship requirement. Its March 2023 guidance is unambiguous that registration turns on whether the "traditional elements of authorship" were conceived and executed by a human^[6].

The guidance treats a bare prompt like "instructions to a commissioned artist": the human identifies what to depict, but the machine "determines how those instructions are implemented in its output" — and when the technology determines the expressive elements, "the generated material is not the product of human authorship"^[6]. Purely AI-generated material must be disclaimed in a registration application. The analysis is case-by-case, and works combining AI-generated material with sufficient human selection, arrangement, or modification can still support a claim — but the burden of showing the human contribution is on you^[6].

The enterprise consequence is concrete. Content your pipeline produces with no meaningful human authorship — a fully automated product-description generator, wholly machine-written marketing pages — may sit outside copyright protection, meaning a competitor who copies it verbatim may face no infringement claim. For assets where exclusivity matters, that argues for a documented human-authorship step (editing, selection, arrangement) in the workflow, and for records that can evidence it. For code, the same logic makes provenance tracking of generated versus human-written portions worth the tooling cost. This is a place where the governance control and the asset-value argument point the same direction.

In the EU, the pressure arrives as transparency rather than authorship doctrine. Under the AI Act's transparency framework as the European Commission describes it, people must be made aware when they are interacting with a machine, providers must ensure "AI-generated content is identifiable," and certain content — deep fakes, and AI-generated text published to inform the public on matters of public interest — must be clearly and visibly labeled, with these rules coming into effect in August 2026^[3]. If your content pipeline publishes into the EU, output labeling and content marking stop being a brand choice and become a compliance surface, which means your generation pipeline needs to know — and record — which artifacts are machine-generated. That record, usefully, is the same provenance log the liability controls below require.

Contractual risk transfer, and the insurance question

What about transferring the residual risk — the wrong-output losses the vendor terms leave with you? Contractually, the levers are the classical ones: negotiated liability caps and carve-outs, service-level commitments tied to defined and measurable failure events, audit rights, and remediation obligations. The honest difficulty is definitional: an enforceable clause needs a "hallucination event" defined tightly enough to adjudicate — which output, judged against what reference, by whom — and most drafting attempts collapse back into the evaluation-methodology problem from the benchmark section. Insurance faces the same actuarial obstacle from the other side: pricing a risk requires a stable, measurable event definition and loss history, and you should treat any policy language here with the same scrutiny as a vendor indemnity — what event triggers coverage, measured how, excluded when. We make no claims about what products exist at what prices; the portable point is that risk transfer instruments follow measurement maturity, and until your organization can define and count its own wrong-output events, neither a contract clause nor a policy can be evaluated, let alone relied on. Build the measurement first; it is also the thing that reduces the underlying risk.

Honest objections

"Vendors are being reasonable — nobody can warrant a probabilistic system's truthfulness." Largely true, and worth conceding. A model vendor cannot control your prompts, your retrieval corpus, or your deployment surface, and an unconditional accuracy warranty would be either priced prohibitively or refused. The critique is not that output-responsibility clauses exist; it is that many buyers read "indemnification" in a term sheet and believe the wrongness risk moved when only the IP risk did. The fix is reading comprehension in procurement, not outrage at the terms.

"The indemnities are marketing-shaped — the exclusions swallow the grant." Partly fair. Anthropic's exclusion list, Google's intentionality condition, and Microsoft's compliance-on-claim mitigation regime each narrow real-world coverage substantially, and Microsoft's version means your entitlement depends on configuration state your platform team controls day to day^[5]. But narrowed coverage is not zero coverage, and the mitigation requirements — filters, metaprompts, retained evaluation reports — are controls you should want anyway. Read cynically, the conditions are cost-shifting; read practically, they are a vendor-subsidized checklist for the controls that reduce the inbound IP risk itself.

"There's no settled case law — isn't structuring for this premature?" The doctrine is genuinely unsettled, and specific predictions about how courts will allocate hallucination losses would be guesswork. But the uncertainty argument cuts the other way: where courts have not spoken, the contract and your documented controls are most of what will matter, and both are things you set now. Waiting for precedent means entering the period that produces the precedent with the default allocation — which, per the terms above, is you.

The read

The decision this supports: allocate your negotiation effort and your engineering effort to different risks. Negotiate the IP boundary — indemnity scope, exclusions, and the configuration conditions that keep coverage alive — because that is where vendor terms actually move. Engineer the wrongness boundary — risk-tiered surfaces, output logging with provenance, human sign-off on regulated surfaces, domain-specific evaluation — because no published term moves it, and the vendors' own terms say so in plain language. A quarterly review that checks (a) which surfaces sit in which tier, (b) whether indemnity-preserving configurations are still in place, and (c) what your logged wrong-output rate is doing, covers the whole exposure with three questions.