LoRA (Low-Rank Adaptation)

Full fine-tuning of a large language model requires updating every parameter — billions of floating-point numbers — which demands enormous GPU memory, weeks of compute time, and millions of dollars for the largest models. LoRA circumvents this by observing that the weight updates required for domain adaptation have a low intrinsic rank: they can be expressed as the product of two small matrices rather than a full-rank weight matrix. By training only these low-rank decomposition matrices (the "adapters") and keeping the base model frozen, LoRA achieves comparable fine-tuning quality at roughly 1,000x lower parameter count.

In practice, LoRA adapters are inserted at the attention layers of the transformer — the most impactful location for steering model behavior. During fine-tuning, only adapter parameters are updated; during inference, adapter weights are either merged into the base model (zero inference overhead) or loaded dynamically (enabling rapid switching between multiple domain adapters). A single base model can serve dozens of LoRA adapters — one per customer, department, or use case — sharing the underlying compute infrastructure.

Enterprise use cases for LoRA include: adapting a general-purpose LLM to a specific legal, medical, or financial domain; training a model on internal communication styles and terminology; creating brand-voice-aligned content generators; and improving code completion accuracy for proprietary codebases or DSLs. The key decision is fine-tuning vs. RAG: LoRA is preferable when the adaptation is about style, tone, or ingrained knowledge rather than recall of specific documents — for factual recall, RAG remains the better architecture.