Rehearsal, or the process of replaying stored data from previous tasks, is a highly effective strategy for mitigating catastrophic forgetting in continual learning. By periodically revisiting examples from older tasks, you can help a model maintain performance on prior knowledge while adapting to new information. However, relying on rehearsal is often undesirable in practice. Storing past data can be costly in terms of memory, especially when dealing with large datasets or many tasks. Privacy concerns may also prevent you from saving sensitive or proprietary data, making rehearsal infeasible. Additionally, as the number of tasks grows, the scalability of rehearsal-based approaches becomes a significant challenge, since the storage and computational requirements increase with every new task.

When you remove rehearsal entirely, you are left with rehearsal-free continual learning methods. These approaches do not store any past data and instead rely solely on the implicit memory encoded in the model's parameters. This means that the only way the model can retain information about previous tasks is through the configuration of its weights and biases. Theoretically, this imposes strict limits on what the model can remember. Without the ability to revisit concrete examples, the model must compress all relevant information into a finite set of parameters. As tasks accumulate, the pressure on parameter memory grows, and it becomes increasingly difficult to preserve performance across all tasks.

Rehearsal-free methods are especially vulnerable in certain scenarios. When there are large shifts between tasks, such as changes in data distribution or task objectives, the model's parameters may need to change significantly to perform well on the new task. This can lead to rapid forgetting of previous knowledge. Similarly, when tasks have conflicting objectives, the model may not be able to find a single parameter configuration that satisfies all requirements. If the model's capacity is insufficient relative to the complexity and diversity of the tasks, it will be forced to overwrite older information, resulting in catastrophic forgetting.

The concept of implicit memory is central to understanding the limitations of rehearsal-free continual learning. Implicit memory refers to the way a model's parameters encode information about past tasks. Unlike explicit memory, such as stored data or external notes, implicit memory is fragile. Small changes to parameters can have unpredictable effects on performance for previously learned tasks. As you train on new data, the optimization process may inadvertently erase or distort the representations needed for earlier tasks. This fragility makes it difficult to guarantee reliable long-term retention without some form of rehearsal or additional constraints.

Key takeaways: rehearsal-free continual learning is fundamentally limited by the capacity and expressiveness of parameter memory; some forgetting is unavoidable.