Cold Start (Serverless AI)

Cold starts in serverless AI follow a predictable sequence: the platform receives a request with no warm worker available, provisions a new container (2–10 seconds), downloads model weights from object storage to local disk (10–120 seconds depending on model size), loads weights from disk into GPU VRAM (5–30 seconds), and finally processes the request. For small models (7B parameters, ~14GB), this sequence takes 15–30 seconds. For large models (70B+), it can exceed two minutes — a commercial non-starter for interactive applications.

Four mitigation strategies have emerged as production-ready. **Minimum instance counts** maintain a warm floor of pre-loaded replicas, eliminating cold starts for predictable baseline traffic at the cost of idle GPU spend. **Predictive pre-warming** uses historical traffic patterns and calendar events to pre-scale infrastructure before demand arrives. **Model weight caching** co-locates weights on high-speed local NVMe storage attached to the inference instance, reducing load time from network download to disk-to-VRAM transfer. **Lazy loading and quantization** reduce the volume of data that must be loaded — a 4-bit quantized model loads in a fraction of the time of its FP16 counterpart.

The enterprise trade-off is between cost and availability. True scale-to-zero (zero idle spend) always carries cold start risk. The optimal architecture for most enterprises is a hybrid: a minimum warm instance pool sized to handle baseline traffic, with serverless burst capacity that accepts cold start latency for infrequent overflow requests. Monitoring cold start frequency and duration in production — and exposing it as a first-class SLO metric — is the prerequisite for making this trade-off deliberately.