How GMMs Work?

The Gaussian mixture model (GMM) works by iteratively improving the placement of Gaussian distributions to best fit the data:

1. Pick a random number of Gaussians: you start by deciding the number of Gaussian distributions (clusters) to fit the data. This is often predefined or determined using methods like the silhouette score, which measures how well-separated the clusters are;

2. Calculate responsibility: for each data point, calculate the probability of it belonging to each Gaussian distribution. This probability, called the responsibility, depends on how close the point is to the center of each Gaussian and the spread (variance);

3. Shift the Gaussians: based on the calculated responsibilities, the means and variances of the Gaussians are updated to better match the data points. This step ensures that the distributions gradually align with the data structure;

4. Repeat steps 2 and 3: the process of calculating responsibilities and shifting the Gaussians is repeated until the model converges.

When Does GMM Converge?

Convergence occurs when the changes in the Gaussian parameters (mean, variance, and weights) between iterations are very small or fall below a predefined threshold.

Suppose you have two Gaussian distributions attempting to cluster a dataset of heights. Initially, one Gaussian might center at an average height of 5 feet, and another at 6 feet. As iterations proceed, the two Gaussians adjust their positions and spreads. If their means and variances stabilize—e.g., one settles at 5.5 feet and the other at 6.2 feet without further significant adjustments—the model has converged.

First Iteration

After Convergence