Variational Lossy Autoencoder

Abstract

A Variational Autoencoder model combines VAEs with autoregressive models to learn global representations by encoding data in a lossy fashion.

Introduction

Representation learning aims to disentangle causal factors of data, but generative modeling is ill-posed without further assumptions, leading to the proposal of hybrid models combining VAEs and autoregressive models.

VAES DO NOT AUTOENCODE IN GENERAL

Variational Autoencoders (VAEs) do not always autoencode, and the latent code is often ignored unless the decoder is weakened, a phenomenon understandable through Bits-Back Coding.

Information Preference

The information preference property of VAEs causes latent codes to be ignored when powerful decoders can model data without them, leading to inefficiency in Bits-Back Coding.

Variational Lossy Autoencoder

The Variational Lossy Autoencoder (VLAE) exploits information preference by using autoregressive models with constrained receptive fields to control what information is captured in the latent representation.

Learned Prior with Autoregressive Flow

An Autoregressive Flow (AF) prior in VLAE is equivalent to an Inverse Autoregressive Flow (IAF) posterior, improving Bits-Back Coding efficiency and offering a more expressive generative model at no additional cost.

Experiments

VLAE is evaluated on 2D image datasets, demonstrating its ability to learn lossy codes for global statistics and achieving state-of-the-art results in density estimation.

Density Estimation

VLAE, utilizing AF priors and autoregressive decoders, outperforms VAEs with IAF priors and achieves new state-of-the-art results on various binary image datasets.

Natural Images: CIFAR10

VLAE is applied to CIFAR10, achieving state-of-the-art performance among variational latent-variable models and outperforming most other tractable likelihood models.

Lossy Compression on CIFAR10

VLAE's receptive field size influences the information captured in lossy codes, with larger fields retaining rougher shapes and grayscale receptive fields preserving color information.

Related Work

This work combines variational autoencoders with continuous latent variables and neural autoregressive models, applying a novel architecture for autoregression.

Related Work

This work combines variational autoencoders with continuous latent variables and neural autoregressive models, applying a novel architecture for autoregression.

Related Work

This text reviews prior work on autoregressive models, latent variable models, and VAEs, highlighting connections and differences with the current research.

Conclusion

This paper proposes a VAE model that functions as a lossy compressor, offering controllable representations and improved density estimation at the cost of slower generation.

References

This section lists numerous academic papers and publications relevant to the research, covering topics in machine learning, generative models, and neural networks.

References

This section lists further academic papers and publications relevant to the research, focusing on variational autoencoders, generative models, and neural network architectures.

Appendix A: Detailed Experiment Setup for Binary Images

This section details the experimental setup for binary images, including the ResNet VAE architecture, PixelCNN configuration, latent code dimension, and training parameters.

Appendix B: Additional Experiment Setup for CIFAR10

This section outlines the experimental setup for CIFAR10, detailing latent code representation, prior distribution, encoder/decoder architectures, and the PixelCNN++ decoder used.

Appendix C: Soft Free Bits

This section introduces a modified "free bits" technique for optimizing VAEs, proposing a smoother objective function that encourages sufficient latent code usage while allowing gradual KL divergence increase.

Appendix D: Autoregressive Decoder Without Autoregressive Prior

This section investigates the performance of an autoregressive decoder with and without an autoregressive prior, demonstrating that incorporating an autoregressive latent code improves density estimation and information transmission.

Figure 4: CIFAR10: Generated samples for different models

This figure displays generated samples from CIFAR10 for various models, illustrating the visual quality of generated images at different bit rates.