To understand how parameter-efficient fine-tuning (PEFT) works at a mathematical level, begin by considering the full weight update for a neural network layer. Suppose you have a weight matrix $W$ of shape $d × k$. During traditional fine-tuning, you compute an update matrix $ΔW ∈ ℝ^{d×k}$, which means you can adjust every entry of $W$ freely. The total number of parameters you can change is $d × k$, and the update space consists of all possible $d × k$ real matrices. This is a very large and high-dimensional space, especially for deep models with large layers.

Now, the low-rank update hypothesis suggests that you do not need to update every single parameter independently to achieve effective adaptation. Instead, you can express the update as the product of two much smaller matrices: $ΔW = BA$, where $B$ ∈ ℝ^{d×r}$and$A$∈ ℝ^{r×k}$. Here, $r$ is a small integer much less than both $d$ and $k$ — in other words, $r ≪ min(d, k)$. This means the update $ΔW$ is restricted to have at most rank $r$, dramatically reducing the number of free parameters from $d × k$ to $r × (d + k)$. By constraining the update to this low-rank form, you are searching for improvements within a much smaller and more structured subset of the full parameter space.

Key insights from this mathematical and geometric perspective include:

• The full update space is extremely large, containing all possible $d × k$ matrices;
• Low-rank updates restrict changes to a much smaller, structured subspace, drastically reducing the number of trainable parameters;
• Geometrically, low-rank updates correspond to projecting gradient information onto a lower-dimensional plane within the full parameter space;
• This restriction enables efficient adaptation with fewer parameters, which is the core advantage of PEFT;
• The success of low-rank PEFT relies on the hypothesis that most useful adaptations can be captured within these low-dimensional subspaces.