To navigate the landscape of parameter-efficient fine-tuning (PEFT), you need to understand the architectural choices that define the field. Three foundational PEFT strategies — adapters, soft prompts, and prefix tuning — differ primarily in where they inject learnable updates into a pretrained model.

Adapters introduce small, trainable modules within the deeper layers of a model, typically after the attention or MLP (feedforward) blocks. These modules learn task-specific transformations while keeping the vast majority of the original model weights frozen. Soft prompts and prefix tuning operate closer to the input, modifying the embeddings or prepending virtual tokens to the model's context, respectively. Soft prompts adjust the input representation directly, while prefix tuning prepends trainable vectors to the key and value matrices in the attention mechanism, influencing how the model attends to input tokens.

Each method thus manipulates a different part of the network: adapters focus on the inner layers (attention and MLP), while prompt-based techniques act at the input or within the attention mechanism itself.

Choosing between adapters, LoRA, and prompt-based methods involves understanding the trade-offs among capacity, memory usage, and inference speed.

• Capacity refers to how much task-specific information the method can encode;
• Memory is the additional storage required for the learnable parameters;
• Inference speed is the runtime efficiency when the model is deployed.

Adapters, by injecting modules into multiple layers, generally offer higher capacity but increase memory usage and may slow inference. LoRA (Low-Rank Adaptation), which decomposes and inserts low-rank matrices into attention or MLP weights, often strikes a balance—offering moderate capacity with relatively low memory and minimal inference overhead. Prompt-based methods, including soft prompts and prefix tuning, typically have the smallest memory footprint and fastest inference, as they only alter the input or the early attention layers, but this comes at the cost of reduced capacity for complex adaptation.

We can view PEFT trade-offs as points inside a triangular simplex:

Adapters lie near the Capacity vertex, LoRA lies on the Capacity–Memory edge, and prompt-based methods lie near the Speed vertex:

Your choice of PEFT architecture has direct implications for expressivity—the ability to capture complex task-specific patterns—and for deployment constraints such as hardware limitations or latency requirements. Adapters, with their deeper integration, excel when you need high expressivity and can afford the extra memory and compute. LoRA is well-suited for scenarios that demand a compromise, offering reasonable expressivity without significant increases in memory or inference time. Prompt-based methods are ideal for resource-constrained environments or applications where rapid inference is critical, though they may struggle with highly complex tasks due to their limited capacity.

Understanding these dynamics allows you to match the right PEFT method to your deployment scenario, ensuring you maximize performance while respecting real-world constraints.

Key Insights:

• Adapters, soft prompts, and prefix tuning differ mainly in where they inject learnable updates: adapters in attention/MLP, prompts in embeddings or attention keys/values;
• The PEFT design space is defined by a trade-off triangle: capacity, memory, and inference speed;
• Adapters maximize capacity but at a cost to memory and speed; prompt-based methods maximize speed and memory efficiency but may limit capacity; LoRA balances all three;
• The right architectural choice depends on your task's complexity and deployment requirements;
• Mastering these trade-offs lets you design efficient, expressive, and deployable fine-tuned models.