Protecting the Model Itself: Supply Chain Attacks, Theft, Extraction, and Scanning

Treat every model in your stack as three assets in one file: an executable artifact that can carry malicious code, a piece of intellectual property that can be stolen through its own API, and a compressed copy of its training data that can leak. Each identity has its own attack class — and its own control: scan-at-ingest, rate-limited serving, and output minimization, all anchored to signed provenance.

This is the model-as-asset threat surface, and it is deliberately distinct from application-layer AI security. Prompt injection, jailbreaks, insecure output handling, and agent tool abuse are attacks on what the model *does* in your application; this guide is about attacks on what the model *is* — the checkpoint file you download, the weights you serve, and the data those weights memorized. The two surfaces need different owners, different tools, and different budget lines, and conflating them is why so many enterprise AI security programs have a red team for prompts and nothing at all guarding the artifact pipeline.

1. Four ways to lose a model

Before the controls, the map. Four attack classes target the model as an asset, and they differ in where the attacker enters, what they walk away with, and which control actually stops them. Mixing them up produces mismatched defenses — a team proud of its API rate limits while loading unscanned pickle checkpoints from a public hub has secured the second threat and left the first wide open.

2. Anchor on the NIST taxonomy, not vendor category names

The vocabulary problem is real: vendors describe the same attack as "model tampering," "AI supply chain risk," or "MLSecOps exposure" depending on what they sell. The authoritative neutral reference is NIST AI 100-2e2025, *Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations*, published March 24, 2025. NIST states that "the taxonomy is arranged in a conceptual hierarchy that includes key types of ML methods, life cycle stages of attack, and attacker goals, objectives, capabilities, and knowledge," and the report "identifies current challenges in the life cycle of AI systems and describes corresponding methods for mitigating and managing the consequences of those attacks."^[5]

Use it two ways. First, as the shared language for security reviews and vendor evaluations: when a scanning vendor claims coverage, ask them to map their detections to the NIST attack categories — evasion, poisoning, privacy — and watch which cells stay empty. Second, as the bridge to governance: the taxonomy plugs into the NIST AI Risk Management Framework, the voluntary framework released January 26, 2023 that most US enterprises already use to structure AI risk ownership.^[6] Model-asset security should be a named risk family inside your existing RMF-style program, not a parallel initiative competing for the same budget.

3. Supply-chain attacks: the model file is a program

The pickle problem

The single most concrete model-security fact an engineering leader should internalize: a PyTorch checkpoint saved in the default format is a Python pickle, and pickle is not a passive data format. Hugging Face's own security documentation is blunt about it: "There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file."^[3] Deserializing a pickle can import modules and invoke functions — including `exec` — which means `torch.load()` on an untrusted checkpoint is functionally equivalent to running an untrusted script with the full privileges of your training or serving environment. That environment typically holds cloud credentials, data-lake access, and network reach into the rest of your stack.

This is why model supply-chain attacks are not hypothetical exotica. An attacker does not need to poison your training data or craft adversarial inputs; they need one engineer to download one convenient checkpoint from a public hub and load it. The blast radius is whatever that process can touch.

Safetensors and safe formats

The structural fix is a serialization format that cannot carry code. Hugging Face's safetensors documentation describes it as "a new simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy),"^[7] and the format is now used across major open-source ML projects including transformers, diffusers, and llama.cpp.^[7] A safetensors file is a header plus raw tensor bytes — there is no instruction stream to execute. Your policy consequence is simple: prefer `.safetensors` (or other data-only formats) for every model you ingest, and treat any pickle-based artifact — `.bin`, `.pt`, `.ckpt`, pickled sklearn models — as untrusted code requiring a scan and a sandboxed load.

Registries, scanning, and the dependency layer

Public model hubs have accepted that they are software-distribution infrastructure and now behave like it. Hugging Face runs a security scanner over every file pushed to the Hub: ClamAV antivirus scans plus a pickle-import scan that extracts and displays the list of imports referenced in any pickled file, flagging suspicious ones — using `pickletools` to read the opcodes without executing them.^[3] That is genuinely useful, and genuinely partial: the same page carries the disclaimer that the approach "is not 100% foolproof" and that safe/unsafe import lists are maintained on a best-effort basis.^[3] Hub-side scanning is a first filter, not a substitute for scanning at your own ingestion boundary.

The same discipline extends below the model to the library layer. AI pipelines pull deep dependency trees from public package indexes, and the standard software-supply-chain playbook applies unchanged: pinned and hash-verified dependencies, a private mirror or artifact proxy in front of public indexes, typosquatting-aware review for new packages, and SBOM generation for the serving image. None of this is AI-specific — which is exactly the point. Your AppSec team already owns these controls; the change is scoping them to cover training and inference environments, which often grew up outside the paved road.

4. Model theft: extraction through the front door

The second identity of the model is expensive IP, and the uncomfortable finding from a decade of research is that serving a model publicly is partially publishing it. The foundational paper is Tramèr et al.'s 2016 "Stealing Machine Learning Models via Prediction APIs," which opens from the enterprise premise: "Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications." The authors then demonstrated "simple, efficient attacks that extract target ML models with near-perfect fidelity" against production ML-as-a-service platforms, found that attacks remained possible even when confidence values were withheld, and concluded that "careful ML model deployment and new model extraction countermeasures" are necessary.^[2]

For modern LLMs the exact replication attack is harder — you cannot clone a frontier model with a few thousand queries — but the economic version of the attack scales fine: distillation. An attacker (or an unscrupulous competitor) queries your model at volume, collects input-output pairs, and fine-tunes a cheaper model on your model's behavior. They get a serviceable approximation of the capability you paid to build, at the price of your inference bill. Every major model provider's terms of service now prohibits training on outputs precisely because contracts are one of the few controls that reach this attack.

The technical control stack has three layers, and each is honest about what it does. Rate limiting and query monitoring raise the cost and duration of extraction: per-key and per-tenant quotas, anomaly detection for systematic input patterns (grid-like probing, decision-boundary exploration, high-volume paraphrase sweeps), and progressive throttling. This does not make extraction impossible; it makes it slow, expensive, and visible — which for a monitored API is usually enough to detect and terminate the account. Watermarking provides forensic attribution rather than prevention: Adi et al. demonstrated "an approach for watermarking Deep Neural Networks in a black-box way" using backdoor-style trigger inputs, showing experimentally that "such a watermark has no noticeable impact on the primary task."^[8] If your model surfaces somewhere it should not, a watermark turns suspicion into evidence. Weight access control addresses the blunt version of theft — someone copying the checkpoint files — with the boring, effective controls: encrypted storage, least-privilege access to artifact stores, and audit logs on model registry reads. For most enterprises the insider-copies-the-bucket scenario is more probable than the API distillation one.

5. Training-data extraction and membership inference

The third identity is the one general counsel cares about: the model as a lossy copy of its training data. Carlini et al.'s "Extracting Training Data from Large Language Models" demonstrated that "an adversary can perform a training data extraction attack to recover individual training examples by querying the language model," and against GPT-2 extracted hundreds of verbatim sequences including "(public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs." Critically, the attack worked "even though each of the above sequences are included in just one document in the training data."^[1]

That scaling result is the strategic point. Memorization is not a defect of small, overfit models that bigger ones outgrow — the exposure grows with capability. Any enterprise fine-tuning a large model on customer records, support transcripts, contracts, or clinical text should assume some of that text is recoverable by a sufficiently motivated querier, and design the deployment around that assumption.

The adjacent attack is membership inference, formalized by Shokri et al.: "given a data record and black-box access to a model, determine if the record was in the model's training dataset."^[4] The attacker recovers no content — only the fact of membership — but in regulated contexts that fact is itself a disclosure. Confirming that a person's record was in the training set of a readmission-risk model reveals they were a patient. The study demonstrated the attack against models trained on commercial ML platforms, including on sensitive hospital discharge data.^[4]

Defenses stack from cheap to expensive. Cheapest: train on less sensitive data — aggressive deduplication, PII scrubbing, and retrieval architectures (RAG) that keep sensitive records in an access-controlled store instead of baking them into weights. Cheap: minimize outputs — return labels or ranked results rather than raw probabilities where the product allows, since rich confidence outputs are the signal membership-inference attacks feed on.^[4] Moderate: the same API rate limits and query auditing you deployed against extraction, because these attacks also require many queries. Expensive: differentially private training, which bounds any single record's influence mathematically at a real cost in accuracy and training complexity — justified for genuinely sensitive training sets, over-engineering for most others.

6. Scanning tools: static artifact analysis vs. behavioral probing

The open-source tool landscape splits along a line that mirrors the threat map: tools that scan the model *file* and tools that probe the model's *behavior*. You need one of each; neither substitutes for the other.

For static artifact scanning, Protect AI's ModelScan is the reference open-source implementation: "an open source project from Protect AI that scans models to determine if they contain unsafe code," supporting multiple serialization formats — pickle and pickle-derived formats, H5, and TensorFlow SavedModel — covering PyTorch, TensorFlow, Keras, sklearn, and XGBoost artifacts, under an Apache 2.0 license. It targets what the project calls Model Serialization Attacks: malicious code injected into a model at save time, before distribution.^[9] This is the tool class that belongs in your ingestion pipeline — every third-party artifact scanned before it can reach a training or serving environment.

For behavioral probing, NVIDIA's garak — the "Generative AI Red-teaming & Assessment Kit" — "checks if an LLM can be made to fail in a way we don't want." Its README positions it in the lineage of nmap and Metasploit for LLMs: it "probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses," against targets including Hugging Face, OpenAI, and AWS Bedrock endpoints, also under Apache 2.0.^[10] Note the scope: garak's probes largely test the *deployed behavior* surface — including data leakage, which is squarely this guide's territory, but also prompt injection and toxicity, which belong to the application-security cluster. That overlap is fine; run it in both programs.

A commercial layer exists above these open-source tools — managed scanning, registry integration, and compliance reporting. Evaluate it with the taxonomy in hand: which NIST AI 100-2 attack categories does the product actually detect, at which lifecycle stage, and what does it add over the Apache-licensed baseline you can deploy today?^[5] A vendor that cannot map its detections to that taxonomy is selling a category name.

7. Honest objections

"Nobody is actually stealing our model." Probably true for a fine-tuned support classifier; the full-fidelity extraction attacks were demonstrated against simpler model classes,^[2] and cloning a large model through its API is costly. But the objection proves too little: the supply-chain and data-extraction threats do not depend on your model being worth stealing — they depend on your pipeline loading third-party artifacts (nearly universal) and your training data containing sensitive text (very common). Prioritize accordingly: scan-at-ingest and output minimization first, anti-distillation defenses only where the model is genuinely differentiating IP.

"Rate limiting punishes legitimate heavy users." Also fair — crude global limits do. The workable version is tiered: authenticated per-tenant quotas with negotiated ceilings for real high-volume customers, plus pattern-based anomaly detection rather than raw volume caps. The goal is not to cap usage but to make systematic harvesting distinguishable from production traffic.

"Watermarks can be removed." Robustness against removal by fine-tuning or pruning is an active research question, and Adi et al. themselves frame their contribution as evaluating robustness "against a multitude of practical attacks" rather than claiming immunity.^[8] Treat watermarking as it is: inexpensive forensic insurance that converts a suspected theft into a demonstrable one often enough to matter, not a lock.

"Scanning is security theater — the scanner disclaims completeness." The Hub's own disclaimer concedes the point that import-list scanning is best-effort.^[3] But the alternative to an imperfect filter is no filter, and the format policy underneath it is not best-effort: a safetensors-only ingestion rule structurally eliminates the load-time execution class rather than detecting it.^[7] Scanning covers the legacy formats you cannot yet ban; the ban is the real control.

8. The read: a control set you can deploy this quarter

The decision this guide supports: stand up a model-asset security lane inside your existing security organization, distinct from (and cheaper than) your LLM application-security effort. It is mostly familiar supply-chain discipline pointed at a new artifact type, which is why it can move fast. The core moves: a private model registry as the single source models deploy from; signed provenance and a safe-format policy at that registry's front door; static scanning for every legacy-format artifact; rate limits and query anomaly monitoring on every model-serving endpoint; output minimization as a design default; and behavioral probing before release. Governance-wise, register the whole lane as a risk family in your NIST AI RMF program so it inherits ownership, review cadence, and audit expectations instead of inventing them.^[6]

What this guide deliberately does not cover is the application-layer cluster — prompt injection, insecure output handling, and agent tool abuse — which has its own control set and its own reference documents. For the runtime governance of agentic systems that consume these models, the controls compose with the identity, approval, and audit framework covered in the agent governance guide below.