LLM Inference at Scale: Serving Stacks, Batching, Autoscaling, and Serverless

The first inference-at-scale decision is not vLLM versus Triton. It is whether to run inference infrastructure at all. Managed APIs — with batch endpoints at half price^[1][2] — absorb variable demand better than a half-utilized GPU cluster ever will. This guide covers the whole ladder: when to buy, when to self-host, and how the serving stack actually works when you do.

The reason this decision deserves an engineering guide rather than a paragraph is that the self-hosted stack has matured fast. Open-source serving engines now ship the batching and memory-management techniques that were research papers three years ago, cloud autoscalers can key off queue depth instead of misleading GPU-utilization gauges, and serverless GPU runtimes can scale a model to zero between bursts. Each layer changes the economics — and therefore changes where the buy-versus-host line sits for your workload.

1. By the numbers

2. The decision before the stack: buy the API or run the GPUs

Self-hosting LLM inference is a commitment to owning a utilization problem. A managed API charges per token, so idle demand costs nothing; a GPU cluster charges per hour, so every idle minute is pure waste. The economics only invert when sustained volume keeps the fleet busy — and the volume math belongs in your FinOps model, not in an infrastructure preference (the companion piece at /guides/llm-finops-guide works through unit economics; /guides/reasoning-models-enterprise-guide covers why token volumes are rising faster than request counts).

Three forces legitimately push an enterprise down the self-hosting path. Sustained, predictable volume at which per-token pricing exceeds the fully loaded cost of a well-utilized fleet. Data control and residency requirements that rule out shipping prompts to a third party — though managed offerings have narrowed this gap with regional routing and zero-retention options. And model control: fine-tuned or open-weight models that no API vendor serves, or latency budgets that require owning the serving loop end to end. If none of the three applies, the honest answer is that you are not a self-hosting candidate yet, and the rest of this guide is due diligence rather than a build plan.

3. The serving stack landscape: vLLM, TGI, and Triton/TensorRT-LLM

If you do self-host, the serving engine is the highest-leverage choice, because it determines your throughput per GPU dollar. The modern landscape consolidated around a handful of open-source engines, and the defining innovation came out of UC Berkeley's Sky Computing Lab: vLLM, built on the PagedAttention algorithm described in the 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention."^[3] The insight is that the key-value cache — the per-request memory that grows with every generated token — was being allocated in contiguous chunks and wasted through fragmentation. PagedAttention manages it the way an operating system manages virtual memory, in non-contiguous pages, which lets far more concurrent requests share a GPU.

Today's vLLM pairs PagedAttention with continuous batching of incoming requests, chunked prefill, and prefix caching, exposes an OpenAI-compatible API server (plus Anthropic Messages API and gRPC support), supports quantization formats from FP8 and INT4 through GPTQ, AWQ, and GGUF, and runs on NVIDIA, AMD, and Intel GPUs as well as CPUs and TPUs.^[6] That breadth — one engine, an API surface your applications already speak, most hardware you might buy — is why it has become the default open-source choice.

Hugging Face's Text Generation Inference (TGI) was the other early production toolkit: a Rust/Python server with a simple launcher, tensor parallelism across GPUs, token streaming over server-sent events, continuous batching, and production plumbing such as OpenTelemetry tracing and Prometheus metrics.^[7] But its own documentation now leads with a caution that changes the selection calculus: "text-generation-inference is now in maintenance mode," accepting only minor bug fixes and lightweight maintenance, and it explicitly recommends vLLM and SGLang as the engines to use going forward.^[7]

NVIDIA's pairing of Triton Inference Server and TensorRT-LLM plays a different game. Triton is a general-purpose inference server — it streamlines serving across TensorRT, PyTorch, ONNX, OpenVINO, and other backends, with dynamic batching, concurrent model execution, model ensembles, and GPU-utilization and latency metrics built in, over HTTP/REST and gRPC per the KServe standard.^[8] TensorRT-LLM supplies the LLM-specific speed: a Python API that compiles models into optimized TensorRT engines, with token streaming, in-flight batching (NVIDIA's term for continuous batching), paged attention, and quantization.^[9] The combination can be the fastest path on NVIDIA hardware, but it buys that speed with an engine-compilation workflow and NVIDIA lock-in that a Python-native engine like vLLM does not impose. It earns its complexity when you run a heterogeneous fleet — LLMs next to vision, speech, and classical ML models — under one serving standard.

4. Continuous batching: the dial between throughput and latency

Batching is where GPU economics are won or lost, because a GPU runs a batch of thirty sequences for nearly the price of one. The naive approach — static batching, where the server collects requests, runs the whole batch to completion, and only then admits new work — fails for LLMs because generation lengths vary wildly: a batch is held hostage by its longest sequence while finished slots sit idle. The fix that vLLM's paper benchmarks against, pioneered by the Orca system it cites as state of the art, is iteration-level scheduling: the batch is recomposed at every token step, so a completed sequence's slot is immediately refilled from the queue.^[3] Every modern engine ships a version of this — continuous batching in vLLM and TGI, in-flight batching in TensorRT-LLM.^[6][7][9]

Continuous batching does not abolish the throughput-versus-latency tradeoff; it just gives you a better dial. Admitting more concurrent sequences raises tokens per second per GPU but slows each individual stream and stretches time-to-first-token, because prefill work for new arrivals competes with decode work for running ones. So tune against the two service-level objectives that actually matter to users — time-to-first-token for interactivity, inter-token latency for streaming read speed — and treat aggregate tokens per second as the cost metric, not the user-experience metric. In practice that means: cap queue wait with an admission timeout, split latency-sensitive and throughput-oriented traffic onto separate pools rather than sharing one compromise configuration, and load-test with production-shaped prompt and output length distributions, because batching behavior is dominated by length variance that synthetic uniform tests hide.

5. Speculative decoding: buying back latency

Autoregressive generation is memory-bandwidth-bound: each token requires a full forward pass, and the accelerator spends most of that pass moving weights, not computing. Speculative decoding exploits the slack. A small, fast draft model proposes several tokens ahead; the large target model then scores the whole proposed run in a single parallel pass and accepts the longest prefix consistent with its own distribution, falling back to normal sampling where the draft diverges. Crucially, the acceptance rule is constructed so outputs match the target model's distribution — this is an exactness-preserving speedup, not a quality trade.

The two founding papers put verbatim numbers on it. Leviathan, Kalman, and Matias (Google) report, on T5-XXL, "a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs."^[10] Chen et al. (DeepMind) benchmark speculative sampling with Chinchilla, a 70-billion-parameter model, "achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself."^[4]

The operational realities: the speedup is proportional to how often the draft model guesses what the target would have said, so it varies by domain — a draft aligned with your traffic (same family, or distilled from the target) matters more than raw draft speed. Rejected drafts are wasted compute, so a poorly matched pair can cost money to go slower. And because the technique spends spare compute capacity to reduce latency, it shines on latency-bound, lightly batched serving, while a server already saturated by continuous batching has less spare capacity for it to exploit. Treat it as a latency tool first and a cost tool second: benchmark acceptance rates on your own traffic before crediting it with savings.

6. Autoscaling GPU inference: scale on the queue, not the GPU gauge

GPU-backed inference breaks the assumptions general-purpose autoscalers were built on. Replicas take minutes to become useful (provision node, pull a multi-gigabyte image, load weights into VRAM), capacity comes in coarse whole-accelerator increments, and the obvious metric lies: Google's GKE guidance for scaling LLM inference warns that GPU utilization does not measure how much work is being done while the GPU is active, making it hard to map latency or throughput targets onto a utilization threshold — a decoding LLM can pin the gauge near its ceiling while serving a fraction of its possible load.^[11]

The same guidance says what to use instead: model-server metrics. Queue size — requests waiting for admission to the batch — is its recommendation for throughput- and cost-oriented scaling, because queue depth correlates directly with request latency and reacts sharply to load spikes; batch size is the sharper signal when latency targets are strict enough that queue-based scaling reacts too slowly.^[11] The pattern generalizes beyond Kubernetes. SageMaker's asynchronous inference endpoints are the fully managed version: AWS recommends a target-tracking policy on an ApproximateBacklogSizePerInstance metric — queue depth per replica — and, unlike its other hosting modes, lets the endpoint scale to zero instances (MinCapacity of 0), with arriving requests queued until a HasBacklogWithoutCapacity alarm scales it back up.^[12]

Whatever the platform, the architecture is the same three parts: a queue in front of the model server, a metric exporting that queue's depth per replica, and a scaler consuming it (Kubernetes teams typically wire this with an event-driven autoscaler such as KEDA plus Prometheus metrics from the serving engine — vLLM, TGI, and Triton all export them^[6][7][8]). Two tuning rules follow from GPU physics. Scale out early and in late — asymmetric thresholds — because a scale-out takes minutes to help while a premature scale-in hurts immediately. And keep node pools segregated by model and GPU class, so the autoscaler reasons about one homogeneous replica type instead of bin-packing mixed workloads.

7. Serverless inference and the cold-start tax

Serverless is the right shape for the spiky, low-duty-cycle workloads where dedicated GPUs would idle — but the platforms differ sharply in what they can hold. AWS Lambda's resource model is memory-first and CPU-only: functions get 128 MB to 10,240 MB of memory with CPU allocated proportionally (one vCPU at 1,769 MB), a 900-second timeout, container images up to 10 GB uncompressed, and up to 10,240 MB of ephemeral storage.^[13] That makes Lambda a fine host for small quantized models, embedding workloads, and — most commonly — the orchestration layer that calls a managed LLM API, but not a serving platform for multi-billion-parameter models.

Google Cloud Run crossed the GPU line: a service can attach one GPU per instance — an NVIDIA L4 with 24 GB of VRAM (minimum 4 CPU and 16 GiB of memory) or an RTX PRO 6000 Blackwell with 96 GB — and still scale down to zero when idle, with GPU instances starting in approximately 5 seconds thanks to pre-installed drivers.^[5] That combination — scale-to-zero economics with real VRAM — is what makes serverless self-hosting genuinely viable for intermittent workloads. A category of AI-native serverless GPU platforms offers the same pattern with a developer experience built around Python functions and snapshotting; evaluate them on the same three questions you would ask any of these platforms: how fast is a cold start with *your* model, what does per-second GPU pricing cost at your duty cycle, and where does your data transit.

Mitigating cold starts

The advertised instance start is the small term. The dominant term for an LLM is everything after: pulling a multi-gigabyte image, reading model weights, and loading them into VRAM before the first token can be produced. Four mitigations, in order of preference:

1. Shrink what must load. Quantized weights load faster and fit smaller (and cheaper) GPUs; every gigabyte you remove is paid back on every cold start. vLLM's supported quantization formats (FP8, INT8, INT4, GPTQ, AWQ, GGUF) make this a configuration choice rather than a research project.^[6]
2. Separate weights from the image. Keep the container image minimal and stream weights from fast object storage or a mounted volume, so image pull and weight load are not serialized through one oversized artifact.
3. Pay for a warm floor. Minimum-instance settings and provisioned-concurrency features hold initialized replicas ready. This converts cold-start latency into a fixed cost — which is exactly the dedicated-capacity tradeoff creeping back in, so size the floor from traffic data, not fear.
4. Go hybrid. Serve the predictable baseline on always-on autoscaled capacity and let serverless absorb only the burst above it, so cold starts land on the marginal request rather than the median one.

8. Batch scheduling: the cheapest tokens are the ones that can wait

A large share of enterprise inference — document processing, classification backfills, embedding refreshes, nightly summarization, evaluation runs — does not need an answer in seconds. Both major API vendors price that patience explicitly. Anthropic's Message Batches API processes asynchronous request sets "with most batches finishing in less than 1 hour while reducing costs by 50% and increasing throughput."^[1] OpenAI's Batch API offers a "50% cost discount compared to synchronous APIs," with each batch completing within 24 hours and often more quickly.^[2] Routing every workload that tolerates hours of latency through these endpoints is the single easiest inference cost win available, and it requires no infrastructure at all — which is why it sits below self-hosting on the escalation ladder.

Self-hosters face the same tradeoff without a menu price: batch work scheduled onto your own fleet during demand troughs raises utilization — attacking the idle-capacity problem that makes GPU ownership expensive — at the cost of prediction freshness. The scheduling policy is the dial. Fixed-interval runs (nightly, hourly) are operationally simple but let staleness grow to the full interval and burn capacity when inputs have not changed. Event-driven triggers (run when new data lands) align compute with actual change, at the cost of orchestration complexity and less predictable completion times. Adaptive scheduling — modulating frequency by data velocity or downstream value — is the efficient frontier but demands real orchestration maturity. Choose by asking one question per pipeline: what does an hour of staleness cost this consumer? If the answer is "nothing," it belongs in the cheapest, most deferrable tier you have.

9. Multi-region: latency, residency, and the cost of being everywhere

Multi-region inference serves two masters that pull in opposite directions: latency (put compute near users, route freely) and data residency (constrain where requests may be processed). The managed platforms now expose that tension as a product switch. Amazon Bedrock's cross-region inference routes requests through inference profiles — a geographic profile keeps processing within a stated boundary such as the US or EU for compliance, while a global profile lets Bedrock select any supported commercial region and prices approximately 10% lower.^[14] There is no additional routing charge, and AWS states that cross-region traffic stays on its network, encrypted in transit, with the processing region logged in CloudTrail.^[14] That is the general shape to demand from any vendor: residency as an explicit, auditable configuration, not an assumption.

Self-hosted multi-region is a different magnitude of commitment: duplicated GPU fleets that each need enough headroom to absorb failover, model weights and rollouts synchronized across regions, global load balancing with health-aware routing, and observability that can correlate incidents across all of it. Because inference is stateless per request, it avoids the hardest distributed-data problems — but the cost structure is brutal, since every region multiplies the idle-capacity problem you already had in one. So sequence it: deploy where your users concentrate first, measure whether network round-trip is actually the dominant term in your end-to-end latency (for long generations, decode time usually dwarfs it — streaming the first token early often improves perceived latency more than a new region does), and expand only when measurement or a residency mandate says so. For most teams, the honest multi-region strategy is a managed API's global endpoint for reach, plus self-hosted capacity only in the one or two regions where volume or sovereignty demands it.

10. Honest objections

"The buy-first framing understates the case for owning inference." Sometimes true. At sustained high volume with flat demand, a well-utilized fleet serving a quantized open-weight model can beat per-token pricing, and the gap widens as open-weight quality closes on frontier APIs. Regulated workloads may make the data-control argument decisive regardless of cost. And owning the loop unlocks optimizations no API exposes — custom speculative-decoding pairs, aggressive prefix caching against your own traffic, latency engineering to a hard SLO. The counter is not that these advantages are unreal; it is that they are all conditional on utilization and on engineering capacity you must staff permanently. The 2–4× throughput leaps documented in the serving literature^[3] cut both ways: the stack improves fast, and a self-hosted deployment nobody re-tunes is quietly overpaying within a year.

"Serving-stack choice is temporary — engines are converging, so the analysis is overkill." Partly right: continuous batching, paged KV caches, and quantization are now table stakes everywhere, and OpenAI-compatible APIs make the data plane feel swappable. But the operational surfaces are not converging — TGI's move to maintenance mode stranded exactly the teams who assumed engine choice was permanent^[7], and Triton's engine-compilation workflow is a genuinely different operating model from vLLM's.^[8] The durable decision is not the engine logo; it is standardizing on an open API surface and portable model formats so the engine underneath stays replaceable.

"Batch discounts and queue-depth autoscaling are tactics, not strategy." The numbers say otherwise. A 50% price cut on every workload that can wait^[1][2] is larger than most teams will ever recover through serving-stack tuning, and it is available today with zero infrastructure. Tactics that halve the bill are strategy.

11. The read: climb the ladder, don't leap it

Treat inference infrastructure as an escalation ladder, where each rung is earned by evidence from the rung below — utilization data, latency measurements, or a compliance mandate — rather than by ambition. Most organizations should live on the first two rungs indefinitely; the point of understanding the upper rungs is knowing precisely what would justify the climb.

12. How to apply this