Continual learning systems are fundamentally challenged by large distribution shifts between tasks. When new tasks are very different from those previously encountered, the model must significantly update its parameters to perform well on the new data. These updates often overwrite the knowledge acquired from earlier tasks, because the parameters that were optimal for the old distribution are no longer suitable. As a result, the model forgets how to solve earlier tasks, and this forgetting is unavoidable if the underlying data distributions have little in common. Even with sophisticated algorithms, a sufficiently large shift in task distribution guarantees that previous knowledge will be compromised.

Another structural reason for failure is the presence of conflicting objectives. In some cases, tasks require mutually incompatible solutions: the optimal parameter configuration for one task directly opposes what is needed for another. No amount of clever optimization or regularization can resolve this, because the conflict is inherent in the objectives themselves. The model is forced to compromise, and the result is that it cannot perform both tasks well. This is a fundamental limitation—if two tasks are truly incompatible, it is impossible for a single set of model parameters to satisfy both.

Capacity exhaustion is a further barrier to continual learning. Neural networks and other models have finite representational resources: they cannot store unlimited amounts of information. As the number of tasks increases, the model must allocate its limited capacity across all of them. Eventually, it runs out of space to encode new knowledge without overwriting what it already knows. This leads to irreversible forgetting, even if the tasks are not highly conflicting. The only way to avoid this is to increase the model’s capacity, but in practice, resources are always limited.

Representation collapse is another failure mode. As the model learns new tasks, the internal features it uses to represent knowledge can be overwritten or repurposed. When this happens, the abstractions that allowed the model to perform previous tasks are lost, and cannot be recovered without retraining on the original data. This collapse of internal representations is especially problematic when tasks are dissimilar, or when the model is forced to discard useful features to make room for new ones.

Key takeaways: there are fundamental limits to continual learning. Some scenarios guarantee forgetting, not because of algorithmic shortcomings, but due to structural constraints: large distribution shifts, conflicting objectives, limited capacity, and representation collapse all create situations where it is impossible to retain all previous knowledge. Recognizing these limits is essential for understanding what continual learning can—and cannot—achieve.