All projects Inference & Density Estimation

Expectation–Maximization

Fitting latent-variable models by iterated lower-bound maximization.

Study & implementation 2022 EMLatent variablesMixture models
Expectation–Maximization diagram

Motivation

Expectation–Maximization is the workhorse for maximum-likelihood estimation when data are incomplete or governed by latent variables — Gaussian mixtures being the classic case. It alternates between inferring the latent responsibilities and re-fitting the parameters.

Method

The E-step computes the posterior over latent variables under current parameters; the M-step maximizes the expected complete-data log-likelihood. Each iteration is guaranteed not to decrease the observed-data likelihood.

\[ Q(\theta \mid \theta^{(t)}) = \mathbb{E}_{z\sim p(z\mid x,\theta^{(t)})}\big[\log p(x,z\mid\theta)\big] \]

Mathematical core

EM is variational inference in disguise: the E-step tightens a lower bound by setting the variational distribution to the exact posterior, and the M-step raises the bound. This makes it the natural stepping stone to mean-field VI and ADVI.

Dempster, Laird & Rubin (1977)
← PreviousVariational AutoencoderNext →Mean-Field Variational Inference