Deep learning has undergone dramatic advances due to scaling laws, which reveal that increasing the size of models and the amount of data they are trained on leads to steady improvements in performance. As models grow from millions to billions of parameters and are trained on ever-larger datasets, the computational and memory requirements for both training and fine-tuning these models rise accordingly. This trend has enabled state-of-the-art results across many domains, but it has also introduced new bottlenecks for anyone wishing to adapt large models to specific tasks.

Key Insights:

• Scaling laws drive up the compute and memory required for training and fine-tuning large models;
• Full fine-tuning is expensive due to O(N) parameter updates and optimizer state memory;
• Weight updates in transformers often occupy a low-dimensional (low-rank) subspace;
• PEFT is most effective for narrow tasks or small domain shifts;
• PEFT may fail when tasks require new representations or when there is a large distributional drift.