Automatic Differentiation VI
Black-box variational inference for any differentiable model.
Motivation
Automatic Differentiation Variational Inference removes the manual derivations of classical VI. Given any differentiable probabilistic model, it transforms constrained variables to an unconstrained space and optimizes the ELBO with stochastic gradients.
Method
ADVI maps the support of each parameter to the reals, places a Gaussian variational family there, and estimates ELBO gradients by Monte-Carlo sampling through the reparameterization trick — the same estimator that powers the VAE.
\[ \nabla_{\phi}\,\mathbb{E}_{q_\phi}[\log p(x,z) - \log q_\phi(z)] \]
Mathematical core
ADVI is where variational inference meets modern autodiff and deep learning: it made VI a general-purpose tool and foreshadows the amortized, gradient-based inference used throughout generative modeling.