Adapter Layers

The original adapter architecture, introduced before LoRA, works by inserting small bottleneck feed-forward networks (typically two linear layers with a non-linearity) into each transformer block of the pretrained model. During task-specific training, only these adapter parameters are updated; the surrounding model weights remain frozen. At inference time, the adapters participate in the forward pass, modulating the model's representations for the specific task they were trained on.

Adapters and LoRA are both PEFT techniques and are frequently discussed together, but they differ architecturally. Traditional adapters add new sequential computation to the model's forward pass, introducing a modest inference latency overhead proportional to the adapter size. LoRA, by contrast, injects low-rank matrices that can be merged into the existing weight matrices — eliminating inference overhead when adapters are merged, at the cost of losing the ability to swap them dynamically. For enterprise use cases requiring adapter hot-swapping or multi-adapter serving, traditional adapter architectures may be preferable; for use cases where a dedicated adapted model is preferred, LoRA with merging is typically the better choice.

The multi-tenant value proposition of adapter layers is significant for enterprise platform teams. Consider a shared LLM serving infrastructure where the legal department requires formal, citation-heavy responses, the marketing team requires brand-voice-consistent outputs, and the engineering team requires structured JSON for tool integration. With adapters, a single base model deployment can load the appropriate adapter per request based on user context — serving all three use cases from the same hardware footprint, each with their own customized behavioral profile.