Batching and queueing for LLM inference: Throughput vs. latency

Large language models (LLMs) underpin many enterprise AI applications, from customer service automation to real-time document analysis. High-volume inference workloads require architectural decisions that balance throughput—how many requests are processed per unit time—and latency, the response time for individual requests. Batching and queueing are two fundamental techniques to manage this balance.

1. Definitions and scope

Batching groups multiple inference requests into a single processing unit, optimizing GPU utilization and amortizing fixed compute costs. Queueing organizes incoming requests in a buffer, holding them until a batch reaches a desired size or a timeout occurs, enabling efficient batch formation while controlling waiting times. This guide covers batching and queueing strategies for transformer-based LLM inference typically hosted on GPU or TPU hardware.

We focus on latency-sensitive but high-volume scenarios, such as conversational AI or content moderation pipelines, where SLAs often require median latencies between 100 milliseconds and 1 second, and throughput demands can range from hundreds to thousands of requests per second.

2. Throughput benefits of batching

GPU hardware designed for LLM inference achieves peak efficiency when processing large batches. NVIDIA’s Triton Inference Server documentation quantifies throughput improvements of up to 4× when increasing batch size from 1 to 16 on models like GPT-2 and BERT (NVIDIA, 2023). This is because many GPU operations execute better when parallelized over multiple inputs.

Batching reduces overhead caused by kernel launches, data transfer, and model initialization. For example, a Google TPUv4 pod can process a batch of 128 sequences more economically than 128 individual executions (Google Cloud TPUs, 2023). Consequently, proper batching can significantly lower cost per request in pay-as-you-go cloud deployments.

3. Latency trade-offs and queueing strategies

The main latency cost of batching arises from waiting for batch formation. Queueing buffers request arrivals to form batches but introduces additional wait time for initial requests, which can negatively impact user experience in latency-sensitive applications.

A common approach involves setting a maximum batch latency—the maximum time the system will wait to collect requests before processing a partial batch. For example, a 50-millisecond max latency combined with a 16-item batch cap aims to process every request within 50 milliseconds plus the model execution time. This reduces tail latency but may reduce batch sizes and throughput.

Queueing can use priority systems where urgent requests are expedited with smaller or immediate batches, while non-urgent requests accrue for larger batches. This approach appears in enterprise inference platforms like OpenAI’s Azure OpenAI Service, which offer configurable latency targets subject to workload SLAs.

4. Implementing batching and queueing: architectures and tools

At the infrastructure level, batching and queueing are implemented either client-side, server-side, or both. Client-side batching (e.g., aggregating multiple user requests before sending to the model) simplifies server logic but can increase client complexity and variability in latency.

Server-side batching involves middleware or inference servers that buffer requests from multiple clients. NVIDIA Triton Inference Server, for example, supports dynamic batch sizes with a configurable timeout (`preferred_batch_size` and `batch_timeout_milliseconds`), directly enabling queueing and batching on GPU inference endpoints.

Cloud providers typically expose batch inference endpoints and manage queueing internally. For instance, Google Vertex AI offers asynchronous batch prediction jobs with scaling options, while AWS SageMaker provides multi-model endpoints that support dynamic batching via server-hosted inference containers.

The choice of batching strategy depends on workload characteristics: burstiness, request size variability, and latency tolerance. Monitoring metrics like queue wait times, batch sizes, and tail latency is essential to tuning parameters effectively and validating SLA compliance.

5. Cost and performance optimization considerations

Batching improves compute efficiency and can reduce inference cost by 20–40% according to benchmarks from MLPerf Inference results (MLCommons, 2023). However, excessive waiting for batch formation can degrade application responsiveness and user satisfaction.

Enterprises should balance batch size and latency targets to optimize for cost and service level objectives. Setting batch sizes too large risks underutilization when request volume drops, while too-small batches may waste GPU parallelism.

Adaptive batching strategies dynamically adjust batch size and timeout parameters based on observed traffic patterns, improving cost efficiency without violating latency SLAs. Such techniques require robust telemetry and feedback loops integrated into orchestration frameworks.

6. Summary checklist for managing batching and queueing in LLM inference