Serverless LLM inference: AWS Lambda, Cloud Run, and Modal

Large language models continue to drive AI initiatives, creating variable and unpredictable inference demand. Serverless platforms promise scalable, pay-per-use deployments to address these fluctuating workloads without maintaining dedicated infrastructure. AWS Lambda, Google Cloud Run, and Modal represent distinct serverless options relevant for enterprise LLM inference.

Performance and resource constraints

These constraints can restrict deployment of large models or batch inference tasks that exceed these limits. In contrast, Google Cloud Run supports containers up to 8 GB of memory (configurable to more in Beta) and runtime up to 60 minutes, enabling larger model footprints and longer processing times. Modal, a newer entrant focused on AI workflows, offers full container support with configurable GPU acceleration, handling multi-GB models but with less mature scalability guarantees.

LLM inference typically requires significant memory and potentially GPU resources. Lambda’s lack of GPU support limits it to small or distilled models or requires model offloading strategies. Cloud Run supports GPU only in beta and requires specific configurations, introducing operational complexity. Modal integrates GPU instances natively in its serverless architecture, targeting ML workloads specifically.

Scaling behavior under variable workloads

AWS Lambda auto-scales by spawning parallel function instances up to account concurrency limits, which can be hundreds to thousands of simultaneous executions based on configuration. This rapid scale-up with sub-second cold starts is advantageous for highly spiky traffic patterns. However, cold starts, though optimized over time, still add latency—measured by AWS to be ~100–200 ms for typical functions, increasing with memory allocation.

Cloud Run supports concurrency at the container instance level, with each instance able to handle multiple requests simultaneously (default concurrency of 80). This results in fewer cold starts under steady load but potentially slower scale-up when load spikes sharply due to container startup times often measured in seconds. Horizontal scaling is limited by maximum container instances per service (default 1000), which is usually sufficient for most workloads.

Modal’s serverless platform allows event-driven scaling of containers with GPU support, designed for batch or streaming inference jobs. Its concurrency model aligns more closely with Kubernetes pod scaling, providing elasticity but possibly slower scale-up compared to Lambda. Modal’s approach suits workloads with moderate scaling bursts and GPU utilization but may incur cold start penalties depending on container image size.

Cost considerations for enterprise deployment

Cost efficiency for LLM inference varies with workload patterns. The absence of GPU charges restricts usage for large inference runs without model size reduction, limiting cost-effectiveness in high-throughput scenarios.

Google Cloud Run pricing includes vCPU ($0.000024 per vCPU-second) and memory ($0.0000025 per GB-second) with additional charges for network egress. GPU pricing is separate and can significantly increase costs depending on the GPU type and duration. Enterprises must balance container compute time versus concurrency management to optimize costs^[1].

Modal offers pay-as-you-go billing with GPU and CPU resource-based pricing, including hourly rates for GPU-accelerated containers. Its targeting of machine learning workloads provides cost transparency for GPU usage but may have higher baseline minimums or minimum job duration charges, impacting cost for infrequent inference.

Integration and operational maturity

AWS Lambda integrates deeply with the broader AWS ecosystem, supporting event sources from API Gateway, S3, and DynamoDB, enabling low-latency triggers and mature observability through CloudWatch. Enterprises with AWS-centric infrastructure benefit from operational maturity and extensive tooling support.

Google Cloud Run, part of Google Cloud’s Anthos and serverless container offerings, fits well with Kubernetes-based CI/CD pipelines and container deployment standards. Its standardized container support simplifies model packaging but requires more container management expertise than Lambda’s function model.

Modal is emerging as a specialized platform for AI workflows, with built-in support for Python-based ML code, GPU scheduling, and data integration. However, given its relative novelty, it lacks the extensive enterprise customer base and integrations of Lambda and Cloud Run, implying additional evaluation is needed before wide adoption in production.

Conclusion: Matching platform to workload profile

For small to medium LLM inference tasks that fit memory and runtime constraints and require rapid scaling on CPU, AWS Lambda provides a cost-effective, mature option with broad integration. Google Cloud Run suits larger containerized models needing more memory or longer processing but requires managing container lifecycles and addresses concurrency differently. Modal offers compelling GPU support and ML-specific capabilities for heavier LLM workloads but remains less proven in enterprise-scale production environments.

Enterprises should benchmark inference latency, startup times, and cost on actual workloads. Hybrid approaches combining Lambda for event-driven lightweight models and Modal or Cloud Run for batch or large-model inference may optimize both cost and performance. Decision-support for MLOps teams includes assessing model size, expected inference concurrency, latency requirements, and existing cloud infrastructure.