Multi-Tenancy (Model Serving)

Multi-tenancy in model serving exists on a spectrum of isolation. At one end, **shared model, shared instance** — all tenants call the same model endpoint with no resource guarantees, suitable for internal tools with trusted users. In the middle, **shared model, isolated configurations** — each tenant has separate system prompts, guardrails, rate limits, and logging while sharing the same underlying model weights and GPU memory, the standard pattern for B2B SaaS AI features. At the other end, **dedicated instances per tenant** — each customer gets their own model replica, offering the strongest isolation at the highest cost, typically reserved for enterprise contracts with strict data segregation requirements.

The technical challenge of multi-tenancy is fair resource allocation under shared GPU memory and compute. A single long-context request can consume enough VRAM to starve other tenants' requests. Modern inference engines like vLLM address this with PagedAttention — a virtual memory management system for KV cache that enables fine-grained memory sharing across concurrent requests. At the platform level, rate limiting, priority queuing, and per-tenant request quotas prevent noisy neighbor effects where one tenant's traffic spike degrades quality of service for others.

The enterprise platform engineering concern is the control plane: how tenant configurations (model parameters, system prompts, fine-tuned LoRA adapters, guardrails) are stored, versioned, and applied without inference latency overhead. Production patterns use a sidecar configuration service that injects tenant context at the gateway layer, so the model server itself remains stateless and horizontally scalable regardless of the number of tenants.