Catastrophic forgetting is a phenomenon where a neural network, when trained sequentially on multiple tasks, loses its ability to perform previously learned tasks after training on new ones. This problem became apparent in the early research on neural networks, where models exhibited a sudden and dramatic drop in performance on earlier tasks after learning new data. The historical emergence of catastrophic forgetting was closely tied to the rise of deep learning and the interest in building systems that could learn continually, similar to how humans accumulate knowledge over time. However, neural networks trained in a standard way tend to overwrite their weights during sequential training, which causes them to lose prior knowledge rapidly and unexpectedly.

The distinction between task interference and true forgetting is essential for understanding catastrophic forgetting. Task interference refers to a temporary drop in performance on a previous task when a new task is introduced; this drop can sometimes be reversible if the network is exposed to both tasks again. In contrast, true forgetting is characterized by an irreversible loss of performance, even if the network is no longer exposed to the new task.

Empirical observations often reveal that interference can occur with short-term fluctuations in accuracy, while forgetting is observed when the model fails to recover its previous performance, even with further training. Theoretical models help clarify this distinction by analyzing how weight updates for new tasks can conflict with those needed for old tasks, sometimes leading to permanent loss of information.

In practice, catastrophic forgetting is observed when a network, after being trained sequentially on tasks A and B, performs poorly on task A despite having previously mastered it. This empirical observation is typically measured by evaluating the model's accuracy on earlier tasks after training on later ones. In theory, catastrophic forgetting is understood as a structural property of neural networks: the same set of parameters is used for all tasks, so updates for new tasks can interfere with or overwrite the information needed for old tasks. This structural nature makes the problem particularly challenging, as it is not simply a matter of insufficient data or tuning, but an inherent limitation of the standard training approach.

Key takeaways:

• Catastrophic forgetting is a structural property of neural networks trained sequentially;
• Task interference and true forgetting are related but distinct concepts; interference may be reversible, while forgetting is not;
• Both empirical evidence and theoretical analysis are needed to fully understand and address catastrophic forgetting in continual learning systems.