Auto-Encoding Variational Bayes

Abstract

A stochastic variational inference algorithm is introduced for efficient inference and learning in directed probabilistic models with continuous latent variables and intractable posteriors, scaling to large datasets.

Abstract

The Stochastic Gradient Variational Bayes (SGVB) algorithm provides an efficient method for approximate posterior inference in directed probabilistic models with intractable posteriors, using a reparameterization of the variational lower bound for optimization.

Strategy

A lower bound estimator for directed graphical models with continuous latent variables is derived, suitable for i.i.d. datasets and maximum likelihood/posterior inference on global parameters and variational inference on latent variables.

Problem Formulation

The problem addresses efficient approximate ML/MAP estimation and posterior inference for parameters and latent variables in directed probabilistic models with continuous latent variables, intractable posteriors, and large datasets.

Recognition Model

A probabilistic encoder, termed a recognition model q(z|x), is introduced to approximate the intractable true posterior po(z|x), with its parameters learned jointly with the generative model parameters.

Variational Lower Bound

The marginal likelihood can be decomposed into a KL-divergence term and a variational lower bound, which is optimized to approximate the marginal likelihood, but direct gradient estimation of the lower bound w.r.t. variational parameters has high variance.

SGVB Estimator

A reparameterization of continuous latent variables z = g(ε, x) allows for a low-variance Monte Carlo estimator of the variational lower bound and its derivatives, enabling efficient optimization with stochastic gradient methods.

Auto-Encoding VB Algorithm

The Auto-Encoding Variational Bayes (AEVB) algorithm uses the SGVB estimator to optimize a recognition model, enabling efficient approximate posterior inference and learning for directed probabilistic models, analogous to autoencoders.

Reparameterization Trick

The reparameterization trick expresses a conditional distribution q(z|x) as a deterministic function of an auxiliary noise variable ε and x, allowing for differentiable Monte Carlo estimation of expectations with respect to q(z|x).

Generative Model Example

A generative model using a neural network for the encoder and a Gaussian or Bernoulli output for the decoder is presented, with parameters optimized jointly using the AEVB algorithm and a reparameterized Gaussian posterior.

Related Work

The paper compares its proposed methods (SGVB and AEVB) to existing algorithms like Wake-Sleep and discusses connections to autoencoders, PCA, and other generative models, highlighting its broader applicability to directed probabilistic models.

Experiments

Generative models for MNIST and Frey Face datasets were trained using AEVB and Wake-Sleep algorithms, demonstrating that AEVB achieves higher variational lower bounds and comparable or better marginal likelihood estimates.

Conclusion

The Stochastic Gradient Variational Bayes (SGVB) estimator and the Auto-Encoding Variational Bayes (AEVB) algorithm provide efficient methods for approximate inference and learning in directed probabilistic models with continuous latent variables.

Future Directions

Future research includes applying SGVB and AEVB to hierarchical generative architectures, time-series models, global parameters, and supervised models, as well as exploring novel noise distributions and model types.