Where to Run Your Models: GPUs, Hosting Options, and Edge vs. Cloud

Where you run your models is a portfolio decision, not a single choice. The default answer — a managed API — is right for most workloads, most of the time. But the moment sustained volume, a residency requirement, or a latency floor enters the picture, you need a position on three questions: which GPUs, which hosting tier, and what (if anything) moves to the edge.

By the numbers

The hosting ladder: five tiers, one variable that matters most

Strip away the vendor branding and there are five places a model can run: behind a vendor's API, on a hyperscaler's managed AI platform, inside your own VPC, in your own data center, or on devices at the edge. The tiers differ in who carries the operational burden and where your data physically travels — but the variable that decides between them is almost always utilization. A managed API charges per token, so idle demand costs nothing. A GPU — rented or owned — charges per hour, so every idle minute is waste. Until your traffic is high and steady enough to keep a fleet busy, climbing the ladder buys you cost *risk*, not cost savings.

Notice that the ladder is no longer a clean tradeoff of "convenience versus privacy." Managed platforms now reach into your network: Amazon Bedrock, for example, can be accessed through VPC interface endpoints so that instances without public IP addresses call the model over AWS PrivateLink rather than the internet.^[5] And the API tier now prices residency explicitly — Anthropic charges a 1.1x multiplier for US-only inference routing and a 10% premium for regional over global endpoints on Bedrock and Google Cloud.^[4] Control used to be a reason to self-host; increasingly it is a line item you can simply buy.

The GPU decision: A100 vs. H100 vs. L40S

If you do descend to self-hosted GPUs, the hardware question comes first, and it is less about raw compute than most spec sheets suggest. LLM inference has two phases with different bottlenecks: prompt processing is compute-heavy, but token generation — where most wall-clock time goes in chat and agent workloads — is dominated by how fast the GPU can stream weights and KV cache out of memory. That makes memory capacity and memory bandwidth the first two numbers to check, ahead of core counts.

Capacity sets a hard floor. A 70-billion-parameter model stored at two bytes per weight needs roughly 140 GB before you allocate a single token of KV cache — that is two 80 GB cards minimum, and more in practice. Quantizing to one byte per weight halves the footprint and is now routine for serving, but the arithmetic still rules out a single 48 GB L40S for models of that class. The L40S earns its place in a different bracket: small and mid-size models — the 7B-to-30B range that /guides/model-right-sizing-guide argues most enterprise tasks should target anyway — served at meaningfully lower power (350 W) and acquisition cost.^[8]

Between A100 and H100, bandwidth is the honest headline: NVIDIA's Ampere tuning guide lists the A100-40GB at 1,550 GB/s,^[6] its DGX A100 documentation confirms the 80 GB-per-GPU configuration (640 GB across eight GPUs),^[7] and the Hopper tuning guide puts the H100's HBM3 at up to 3 TB/s — a 93% increase over the A100-40GB.^[3] For a memory-bound decode loop, that gap translates directly into tokens per second per dollar — which is why the H100 can be the cheaper card per token even at nearly four times the hourly rate, provided you keep it busy. The A100 remains a rational choice when availability, existing fleet compatibility, or price beats peak throughput.

One more axis matters at the top of the range: models that exceed a single card's memory force multi-GPU serving, and at that point the interconnect between GPUs becomes part of the purchase. Tensor-parallel serving shuttles activations between cards on every forward pass, so eight loosely coupled PCIe cards and eight NVLink-connected cards in a DGX-class chassis are not interchangeable, even when the spec-sheet totals match. This is also the quiet argument for right-sizing before hardware selection: if a distilled or quantized model fits on one card, you escape the interconnect problem — and its price tag — entirely.

What GPU capacity costs today

Cloud GPU pricing is unusually transparent right now because AWS publishes reservation rates for its Capacity Blocks for ML program — short-term reserved GPU capacity with prices the company says are updated regularly based on supply and demand trends.^[1] As of this writing, an 8× A100 p4d.24xlarge runs $11.80 per instance-hour across US regions ($1.475 per GPU-hour); an 8× H100 p5.48xlarge is $41.528 in US East ($5.191 per GPU-hour); an 8× B200 p6-b200.48xlarge is $98.84 ($12.355 per GPU-hour); and the newest 8× B300 p6-b300.48xlarge lists at $112.32 ($14.04 per GPU-hour).^[1]

Two readings of that chart matter for planning. First, each GPU generation launches at a steep premium and then deflates as the next one ships — the A100 at $1.475 per GPU-hour is now priced like a commodity, and AWS's own note that reservation prices track supply and demand tells you the H100's rate will follow the same curve.^[1] Locking multi-year commitments at generation-peak prices is how GPU budgets end up underwater. Second, the premium buys throughput, so the per-token comparison is much flatter than the per-hour one; a faster card that finishes the same queue in a third of the time can justify triple the rate.

On-premises hardware changes the shape of the cost, not just the amount: capital outlay up front, then depreciation, power, cooling, floor space, and the staff to run it, all amortized over a three-to-five-year lifecycle. The break-even against cloud rates depends almost entirely on sustained utilization — an owned GPU only beats a rented one if it stays busy for years. That full total-cost model, including the ancillary charges (egress, storage, licensing) that quietly dominate some deployments, is worked through in /insights/enterprise-ai-tco-guide.

Spot capacity: the discount and the failure drill

All three hyperscalers sell reclaimable capacity at deep discounts, and the numbers are worth taking seriously: AWS advertises savings of up to 90% versus On-Demand prices,^[2] Google Cloud states up to 91% off for many machine types, GPUs, and TPUs on Spot VMs,^[9] and Azure claims up to 90% versus pay-as-you-go for Spot Virtual Machines.^[10] The catch is uniform: the provider can take the capacity back, and the warning you get is short.

For LLM workloads, the fit splits cleanly by phase. Batch inference — evaluation runs, document backfills, embedding generation — is the natural spot workload: requests can be checkpointed, retried, and rescheduled, so a reclaimed node costs minutes, not customers. Interactive serving on spot is harder but not off-limits: the working patterns are hybrid pools (an on-demand baseline sized for the latency SLA, with spot nodes absorbing the peak), request-level retry against redundant endpoints, and autoscalers wired to the interruption signal so replacement capacity starts spinning up the moment a reclaim notice lands. What does not work is treating spot as cheap on-demand and hoping.

API vs. self-hosted: control now has a rate card

The strongest argument for staying at the top of the ladder is that the API tier keeps repricing itself downward and sideways. On the Claude API, Sonnet 5 lists at $2 per million input tokens and $10 per million output tokens, with Haiku 4.5 at $1 and $5 for lighter tasks — and the Batch API halves both numbers for anything asynchronous.^[4] Prompt caching cuts repeated context to a tenth of the base input price.^[4] A workload that engineers those levers well can see its effective per-token rate drop severalfold without touching infrastructure — which resets the volume threshold at which self-hosting begins to pay.

The residency objection is also weaker than it was. Beyond Bedrock's PrivateLink path,^[5] Anthropic now sells geographic control directly: regional and multi-region endpoints on Bedrock and Google Cloud carry a 10% premium over global endpoints, and first-party US-only routing bills at a 1.1x multiplier.^[4] That is the market telling you what residency actually costs — roughly a tenth on top of your token bill. If your compliance requirement can be satisfied by contractual routing guarantees at +10%, standing up a GPU fleet to get the same assurance is an expensive way to buy the same sentence in your audit report. Self-hosting earns its keep when the requirement is stricter than routing — model weights you own, air-gapped networks, or inference over data that cannot leave your tenancy under any contract.

There is a procurement dimension to the tier choice, too. Staying at the API tier concentrates your exposure in one vendor's rate card and deprecation schedule; the mitigation is contractual and architectural — an abstraction layer that keeps a second provider one configuration change away — rather than infrastructural. Descending to self-hosted open-weight models trades that concentration risk for a different one: you now own the upgrade treadmill, and the vendor risk moves from your model provider to your GPU supplier and serving-stack maintainers. Neither position is risk-free; the mistake is holding one while pricing your alternatives as if you held the other.

When you do cross over, cross deliberately: the serving stack, autoscaling behavior, and batch scheduling that make a GPU fleet cost-competitive are their own discipline, covered in depth in /guides/llm-inference-at-scale. The economics only work if you bring the operational maturity with you.

Edge vs. cloud: the fourth location

Edge inference — running models on devices, gateways, or site-local servers — is a different decision from the hosting ladder, because its drivers are physical rather than financial. Three of them are decisive when they apply. Latency: removing the network round trip is the only way to get response times below what physics allows for a data-center round trip, which matters for machine-vision quality gates on production lines and point-of-sale flows. Locality of data: video, biometrics, and proprietary sensor streams that never leave the site never appear in a cloud data-processing agreement — privacy by architecture rather than by contract. Autonomy: a factory cell or a retail store that must keep working through a WAN outage cannot depend on a regional endpoint.

The costs are equally structural. Edge hardware is sized in watts, not racks, so it runs small, quantized models — which pushes you toward the aggressive right-sizing and distillation practices in /guides/model-right-sizing-guide, and means frontier-model quality is generally not on the menu at the edge. And every device is a deployment target: model updates, telemetry, security patching, and hardware refresh across hundreds of sites is a fleet-management program, not a Terraform module. The pattern that works in practice is hybrid: latency-critical and privacy-critical inference at the edge, with batched uploads feeding cloud-side retraining, evaluation, and the heavyweight models that need data-center hardware.

Honest objections

"We'll never reach the utilization to justify GPUs — this whole exercise is academic." For most organizations, correct — and that is a conclusion, not a failure. The decision framework's most common valid output is "stay on the API and revisit in two quarters." The exercise still pays for itself, because knowing your crossover volume turns vendor negotiations and budget reviews from vibes into arithmetic.

"GPU prices move too fast for any of these numbers to matter." The specific figures will drift — AWS says outright that its reservation prices are updated regularly with supply and demand.^[1] But the structure is durable: each generation premium deflates, spot stays roughly an order of magnitude below reserved, and bandwidth-per-dollar keeps improving. Build the decision on the structure and refresh the numbers quarterly, rather than refusing to decide because the numbers move.

"Spot for inference is a trap; the interruptions will eat the savings." It can be, if spot is treated as a drop-in substitute. But the failure modes are bounded and documented — two minutes of warning at best, none at worst^[11][9] — and the mitigations are standard distributed-systems hygiene, not research problems. For batch workloads the objection barely applies at all; for interactive serving, the hybrid-pool pattern lets you take the discount only on the traffic that can tolerate it.

The read

Treat placement as a portfolio with a default and documented exceptions. The default is the managed API — batched, cached, and right-sized — because it converts fixed infrastructure risk into a variable cost you can optimize weekly. The first exception is sustained, predictable volume on a model you control, which justifies self-hosted GPUs: H100-class for large models where bandwidth is the bottleneck, L40S-class for the small and mid-size models most enterprise tasks actually need, spot capacity for everything that can be interrupted. The second exception is a data or latency boundary that no contract can satisfy — which sends specific workloads on-premises or to the edge, narrowly, while everything else stays on the ladder above.