Understanding the theoretical limits of neural network compression begins with the concept of entropy. In the context of neural network weights, entropy measures the average amount of information required to represent the weights. This concept comes from information theory, where entropy quantifies the minimal number of bits needed, on average, to encode a random variable. For neural networks, the distribution of weights determines the entropy: if weights are highly predictable or clustered, the entropy is low, meaning the weights can be compressed more efficiently. Conversely, if the weights are highly random or uniformly distributed, the entropy is higher, setting a stricter lower bound on how much the model can be compressed without losing information. Thus, entropy provides a fundamental lower bound for any compression scheme applied to model weights.

The mathematical foundation of rate–distortion theory is captured by the rate–distortion function. It is defined as:

Here, $R(D)$ is the minimum rate (measured in bits per symbol or weight) required to encode the source $X$ such that the expected distortion between the original $x$ and the reconstruction $\hat{x}$ does not exceed $D$. The minimization is over all possible conditional distributions $p(\hat{x}|x)$ that satisfy the distortion constraint. $I(X;\hat{X})$ is the mutual information between $X$ and $\hat{X}$, representing the amount of information preserved after compression. This equation formalizes the best possible trade-off between compression and distortion, and is central to understanding the theoretical limits of neural network compression.