High-Throughput Inference

GPU hardware is expensive. A single NVIDIA H100 SXM5 costs $30,000–$40,000 and consumes 700W in operation. The business objective of high-throughput inference is to maximize the useful work — completed AI requests — extracted from each GPU before it returns to idle. The gap between a naive inference deployment and an optimized one is typically 5–20x in requests served per GPU, representing a proportional reduction in infrastructure cost.

The most impactful throughput optimization is **continuous batching** (also called in-flight batching). Traditional static batching waits for a full batch to be assembled before processing, leaving GPU resources idle between requests. Continuous batching interleaves new requests into ongoing generation sequences at the token level — as soon as a sequence completes a token, a new request occupies the freed capacity. This keeps GPU compute saturated regardless of varying request lengths and arrival rates. Combined with **PagedAttention** (vLLM's virtual memory system for KV cache that eliminates memory fragmentation), continuous batching delivers the largest single throughput improvement available in software.

Beyond continuous batching, the throughput optimization stack includes: **speculative decoding** (a small draft model proposes tokens that the main model verifies in parallel, boosting throughput 2–3x for latency-tolerant workloads), **tensor parallelism** (distributing model weights across GPUs to increase the effective memory bandwidth for large models), **quantization** (INT8/INT4 weights occupy less VRAM, allowing larger batch sizes and more concurrent requests), and **request scheduling** (priority queuing and preemption policies that maximize GPU utilization across a mixed workload). At sufficient scale, the engineering investment in a purpose-built throughput optimization stack returns multiples more than the equivalent spend on additional hardware.