Learn Gaussian Distribution within the Exponential Family Framework

To understand how the Gaussian distribution fits into the exponential family, you first need to recall the general exponential family form discussed earlier. A probability distribution belongs to the exponential family if it can be written as:

p(x|\theta) = h(x) \exp\left( \eta(\theta)^\top T(x) - A(\theta) \right)

where:

$h(x)$ is the base measure;
$T(x)$ is the vector of sufficient statistics;
$\eta(\theta)$ is the vector of natural parameters;
$A(\theta)$ is the log-partition function.

Let's break down the standard univariate Gaussian (normal) distribution:

p(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(x-\mu)^2}{2\sigma^2} \right)

To express this in exponential family form, expand the quadratic term in the exponent:

-\frac{(x-\mu)^2}{2\sigma^2} = -\frac{x^2 - 2\mu x + \mu^2}{2\sigma^2} = -\frac{x^2}{2\sigma^2} + \frac{\mu x}{\sigma^2} - \frac{\mu^2}{2\sigma^2}

Now, you can write the density as:

p(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( \frac{\mu x}{\sigma^2} - \frac{x^2}{2\sigma^2} - \frac{\mu^2}{2\sigma^2} \right)

Group terms to match the exponential family structure:

p(x|\mu, \sigma^2) = \underbrace{\frac{1}{\sqrt{2\pi\sigma^2}}}_{h(x)} \exp\left( \underbrace{\left[ \frac{\mu}{\sigma^2}, -\frac{1}{2\sigma^2} \right]}_{\eta(\theta)} \cdot \underbrace{\begin{bmatrix} x \\ x^2 \end{bmatrix}}_{T(x)} - \underbrace{\frac{\mu^2}{2\sigma^2}}_{A(\theta)} \right)

Here, $x$ and $x^2$ are the sufficient statistics, while the natural parameters are functions of $mu$ and $sigma^2$ .

Definition

For the Gaussian, the sufficient statistics are $T(x) = [x, x^2]^T$ , and the natural parameters are $η₁ = μ/σ²$ and $η₂ = -1/(2σ²)$ . These capture all the information about the data relevant for estimating $μ$ and $σ²$ .

Recognizing the Gaussian as part of the exponential family is not just a mathematical exercise — it has direct implications for how you design and train machine learning models. When a distribution is in the exponential family, you benefit from general properties such as:

Having sufficient statistics that enable efficient data summarization;
Allowing for conjugate priors in Bayesian inference, making posterior calculations tractable;
Enabling streamlined maximum likelihood estimation and gradient-based optimization due to the log-partition function structure;
Supporting generalized linear models (GLMs), where the Gaussian leads to linear regression with squared error loss.

In practical terms, this means you can build regression models, perform Bayesian updates, and analyze uncertainty efficiently, all rooted in the exponential family structure of the Gaussian. This framework also guides you in extending these concepts to other distributions you will encounter in machine learning.

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Suggested prompts:

Can you explain what sufficient statistics are in this context?

How does the exponential family structure help with Bayesian inference?

Can you give an example of using the Gaussian in a generalized linear model?

Swipe to show menu

p(x|\theta) = h(x) \exp\left( \eta(\theta)^\top T(x) - A(\theta) \right)