Low-Latency Inference

Latency in AI inference has two distinct phases that must be optimized separately. **Time-to-first-token (TTFT)** measures the delay from request submission to the first generated token arriving at the client — this is the dominant factor in perceived responsiveness for interactive applications. **Inter-token latency (ITL)** measures the time between subsequent tokens, which determines how smoothly the streamed response populates. A system with 150ms TTFT and 20ms ITL feels fast; one with 800ms TTFT but 5ms ITL still feels slow, regardless of the low per-token speed.

TTFT is dominated by the prefill phase — the model must process all input tokens (system prompt, conversation history, user query) before generating the first output token. Three optimization levers have the greatest impact: (1) **Prompt caching** stores the KV cache from repeated system prompt prefixes so they are not recomputed per request — reducing TTFT by 40–80% for applications with long, static system prompts; (2) **Model size selection** — a 7B parameter model at FP16 runs prefill in 50–100ms on an H100; a 70B model takes 5–10x longer; choosing the smallest model that meets quality requirements is the highest-leverage latency decision; (3) **Dedicated GPU instances** eliminate the queuing latency that occurs under shared infrastructure, ensuring requests are processed immediately rather than waiting behind other tenants.

The hardware selection has an outsized impact. Purpose-built inference hardware (Groq's LPU, Cerebras CS-3) delivers dramatically lower latency than GPUs by eliminating memory bandwidth bottlenecks — Groq has demonstrated 500+ tokens/second on Llama-3 70B compared to ~80 tokens/second on an H100. For most enterprises, standard NVIDIA GPU infrastructure with proper optimization achieves acceptable latency at lower procurement cost, but high-frequency trading analogs (algorithmic decisions requiring sub-100ms total latency) warrant evaluating specialized inference silicon.