Transformer-Based Generative Models

Introduction to Transformers and Self-Attention

Transformers are a foundational architecture in modern AI, especially in Natural Language Processing (NLP) and generative modeling. First introduced in the paper "Attention is All You Need" (Vaswani et al., 2017), transformers discard recurrence in favor of a mechanism called self-attention, which allows models to consider all parts of the input sequence at once.

Self-Attention Mechanism

The self-attention mechanism enables the model to weigh the importance of different tokens in a sequence relative to each other. This is done using three matrices derived from the input embeddings:

• Query (Q);
• Key (K);
• Value (V).

The attention output is computed as:

Where:

• $Q$, $K$, and $V$ are matrices derived from the input.
• $d_k$ is the dimension of the key vectors.
• $\text{softmax}$ converts the similarity scores to probabilities.

This allows each token to attend to every other token and adjust its representation accordingly.

Transformer Architecture Overview

The transformer model consists of stacked encoder and decoder layers:

• Encoder converts input into a contextualized latent representation;
• Decoder generates output tokens using the encoder’s output and prior tokens.

Each layer includes:

• Multi-Head Self-Attention;
• Feedforward Neural Networks;
• Layer Normalization;
• Residual Connections.

Multi-Head Self-Attention

Instead of computing a single attention function, the transformer uses multiple attention heads. Each head learns to focus on different parts of the sequence.

Where each head is computed as:

Where:

• $W_i^Q, W_i^K, W_i^V$ are projection matrices for queries, keys, and values;
• $W^0$ projects the concatenated heads back to the original dimension.

Feedforward Neural Networks

Each transformer block includes a position-wise feedforward network applied independently to each position:

• It consists of two linear layers with a non-linearity (e.g., ReLU) in between;
• Applies the same transformation across all positions.

Layer Normalization

Layer normalization normalizes the input across the features (channels) instead of batch. It stabilizes training and improves convergence:

Where:

• $\mu$ is the mean of the features;
• $\sigma$ is the standard deviation;
• $\gamma$ and $\beta$ are learnable parameters.

Residual Connections

Residual connections add the input of each sub-layer to its output:

• This helps with gradient flow and enables training of deeper models;
• Used around both the self-attention and feedforward layers.

In decoder-only models (like GPT), only the decoder is used with causal (masked) self-attention.

Generative Pre-trained Transformers (GPT)

GPT models are decoder-only transformers trained to predict the next token in an autoregressive fashion:

Key features:

• Trained on large-scale text datasets;
• Can generate coherent and diverse text;
• Widely used in applications like chatbots and code generation.

BERT and Masked Language Modeling

BERT (Bidirectional Encoder Representations from Transformers) uses only the encoder. It is trained with masked language modeling (MLM):

• Random tokens are replaced with a [MASK];
• The model predicts the original token based on full context.

This makes BERT good at tasks like classification, Q&A, and semantic similarity.

Transformers and LLMs

Transformers are the backbone of Large Language Models (LLMs) like GPT-3, GPT-4, PaLM, LLaMA, and Claude.

LLMs use large datasets and hundreds of billions of parameters, enabling them to:

• Understand and generate human language;
• Perform translation, summarization, Q&A, reasoning;
• Power chatbots, document analyzers, and coding assistants.

Transformers' scalability and ability to model long-range dependencies make them ideal for these models.