As you move through the layers of a large language model (LLM), each layer applies a learned transformation to its input, reshaping the geometry of the latent space. These transformations are typically linear or affine, followed by non-linearities, and they serve to mix, separate, or otherwise modify the way information is represented. Early layers often capture simple, local features, while deeper layers encode more abstract, global properties. With each transformation, semantic features can become more or less entangled—meaning that the directions in latent space that correspond to distinct concepts may become mixed together or separated further apart. This entanglement is not random: it reflects the model’s learned organization of knowledge and its strategy for solving tasks.

Layer-wise transformations can cause representations of similar inputs to drift apart or cluster closer together, depending on the learned weights. This drift can change the orientation of important subspaces, making certain semantic directions easier or harder to interpret as you go deeper into the network. In some cases, layers may disentangle features, making it easier to isolate specific concepts. In others, they may entangle them, blending multiple features into shared directions. This ongoing reshaping is central to how LLMs process and generate language, but it also complicates interpretability—what is a clean, interpretable direction in one layer may become opaque in another.

To build geometric intuition, imagine that each layer rotates, stretches, and shifts the cloud of data points in latent space. With each transformation, the axes that once aligned with clear semantic features may tilt or curve, and the clusters representing different meanings may merge, split, or drift. This layer-wise drift means that the same semantic concept might be represented very differently across layers, making it challenging to track or manipulate. Changes in subspace orientation can either clarify or obscure the underlying structure, directly impacting your ability to interpret or intervene in the model’s representations.

Key Insights:

• Each layer applies a transformation that reshapes the latent geometry;
• Semantic features can become more or less entangled as representations pass through layers;
• Layer-wise drift changes the orientation and position of subspaces, affecting interpretability;
• The same concept may be encoded differently across layers, complicating direct manipulation;
• Entanglement and drift are essential for the model’s expressivity, but can hinder analysis.