Variational Lossy Autoencoder

Host: We are starting with a foundational 2017 paper from UC Berkeley and OpenAI that explores how AI systems compress and understand images. The authors introduce the Variational Lossy Autoencoder, aiming to solve a specific problem in representation learning. Guest: What exactly is representation learning trying to achieve in this context? Host: It is about training a model to extract only the most useful parts of the data for downstream tasks, like image classification. For a 2D image, a good representation might capture the overall global shape of an object but completely ignore tiny, detailed textures. Guest: That makes sense, but how do you force a model to ignore those fine texture details? Host: The researchers did it by combining a standard Variational Autoencoder, or VAE, with autoregressive models like RNNs or PixelCNNs. By carefully designing this architecture, they intentionally restrict what the model's central memory, called its latent code, is able to learn. Guest: So the model is forced to drop the less important information, making the whole encoding process lossy? Host: Exactly, the model deliberately autoencodes the data in a lossy fashion to keep only the global structure. Plus, using those autoregressive models for both the mathematical prior and the decoding steps dramatically improves its overall generative performance. Guest: Did this new architecture actually outperform the existing models at the time? Host: It did, achieving new state-of-the-art results on several major density estimation benchmarks, including MNIST and OMNIGLOT.

Host: Let's explore how we can teach machine learning models to genuinely understand the data they are looking at rather than just memorizing it. A key goal here is something called representation learning, where we want the model to identify and untangle the underlying causes of the data. Guest: What does that untangling actually look like in practice? Host: Think about an image. We want the model to capture the big-picture global structure, like the identity or style of an object, while ignoring local variations like a specific texture or random white noise. Guest: That sounds useful, but how do we actually force a model to separate the big picture from the noisy details? Host: One popular method is to use a generative model with latent variables, which essentially learns to recreate the data from scratch. The hope is that by figuring out how to build the image, the model will naturally organize those underlying causes into a neat hierarchy. Guest: You said it is a hope, which makes me think it does not always go according to plan. Host: Exactly, because a model can become incredible at predicting the exact data distribution without ever learning a meaningful underlying representation. Autoregressive models, for example, are incredibly powerful at generating data pixel by pixel, but they do not use those high-level latent variables at all. Guest: Could we just combine an autoregressive model with something like a Variational Autoencoder, which is built to use latent variables? Host: People have tried creating those exact hybrids, but they ran into a frustrating issue. The autoregressive part of the model is so powerful that it ends up explaining all the structure in the data by itself, completely ignoring the latent variables. Guest: Oh, so the big-picture part gets lazy because the pixel-by-pixel part is doing all the heavy lifting. How do we fix that? Host: Past research showed that artificially weakening the autoregressive part forces the latent variables to step up. Now, we are going to look at a principled solution that uses this dynamic to control exactly what information goes into the latent variables, giving us both top-tier image generation and highly meaningful representations.

Host: We are going to look at a surprising reality about Variational Autoencoders, which is that despite their name, they do not actually have to autoencode. Guest: Wait, an autoencoder that doesn't autoencode? Isn't the whole point to compress a data point into a latent code and then reconstruct it perfectly? Host: You would definitely think so, but the authors point out that VAEs only act that way under specific conditions. It all comes down to the math of how they learn, specifically how they handle an otherwise impossible calculation. Guest: Impossible meaning they can't directly calculate the best way to model the data? Host: Exactly. Because they can't compute the exact probability of the data directly, they optimize a stand-in target called a variational lower bound. This target has two competing parts: one part rewards minimizing the reconstruction error, and the other is a penalty that forces the latent code to match a simple, standardized shape. Guest: Okay, so the model is juggling recreating the input and keeping that compressed code simple. How does that stop it from autoencoding? Host: It happens when the decoder part of the model is too powerful, which is common in things like text sequence modeling. A really strong decoder can figure out how to generate the data on its own, so it just ignores the latent code entirely to completely avoid that simplicity penalty. Guest: Oh, that makes sense! If it ignores the code, the penalty drops to zero, but it isn't actually autoencoding anything anymore. Why do VAEs work so well for images, then? Host: In two-dimensional image modeling, the standard practice is to use very simple, factorized decoders. Because the decoder is fundamentally weakened, it isn't powerful enough to cheat, so it is forced to rely on the latent code to perfectly recreate the image.

Host: Let's explore how models decide which information to prioritize when generating complex sequences. When researchers pair a Variational Autoencoder, or VAE, with a really powerful decoder like a Recurrent Neural Network, they run into a frustrating problem where the latent code gets completely ignored. Guest: Wait, isn't the entire point of a VAE to use that compressed latent code to generate the final output? Host: Exactly, but because RNNs are so powerfully expressive, they can just look at previous steps in a sequence and predict the next one on their own. Historically, people blamed this on early training difficulties, assuming the model just matched a default distribution to avoid paying a regularization penalty. Guest: So the assumption was that the model just takes the lazy route early on because the optimization is too hard? Host: That was the common wisdom. But the authors point out a neglected observation, which is that it isn't just an optimization glitch. They argue that even if you could solve the training perfectly, it is actually mathematically optimal for a powerful decoder to ignore the latent code. Guest: Why would ignoring the core feature of the model ever be the mathematically optimal choice? Host: To explain this, they use an information theory concept called Bits-Back Coding, which frames the VAE as a data compression protocol. Imagine transmitting data in a two-part code: first you send the essential structure, which is the latent code, and then you send the remaining modeling errors. Guest: Okay, so a basic approach would just be adding the size of the structure and the size of the errors together. Host: Right, but a naive two-part code is actually inefficient. Bits-Back Coding shows that you can use the encoder's probability distribution to transmit a secondary message alongside the data, which effectively reduces the total size of your transmission. Guest: That is a clever trick, but how does this compression idea explain why the latent code gets completely ignored in practice? Host: Because mathematically, minimizing this expected transmission length is identical to optimizing the VAE. The authors show that when your true mathematical distributions are too complex to calculate perfectly, forcing the data through the latent code adds an information penalty. Guest: Oh, I see, so it costs extra space to use an imperfect code. Host: Exactly. Because of that penalty, when the model wants to compress the data as efficiently as possible, it is much cheaper to just let the powerful RNN decoder do all the prediction work on its own and bypass the latent code entirely.

Host: We're looking at how certain neural networks compress information, specifically when they have to leave some details behind. In these systems, there's always a hidden cost when our network's approximation of the data isn't perfectly precise. Guest: What kind of cost are we talking about? Does it slow the model down? Host: It's actually a mathematical cost measured as Kullback-Leibler, or KL, divergence, which basically means we need extra space to encode the data. Because our model's approximations can't completely perfectly match the true distribution, this extra coding cost is essentially unavoidable. Guest: If the model is always penalized with this extra cost, does it try to find a workaround? Host: It absolutely does, and that leads to a fascinating problem. If you use a really powerful decoder, like an autoregressive RNN, the model realizes it can recreate the data perfectly without even looking at the compressed latent code. Guest: Wait, if it ignores the latent code, doesn't that defeat the whole purpose of an autoencoder? Host: Exactly! By setting the code to a generic default, it completely avoids that KL divergence penalty, choosing instead to handle all the complex information locally within that powerful decoder. Guest: That sounds like a frustrating loophole. How do we force the model to actually use the compressed code? Host: One common method is to intentionally use a weaker, simpler decoder, which forces the model to rely on the latent code for the big picture. Engineers also use tricks like "annealing" to slowly introduce the penalty during training, or they provide "free bits" so the model isn't punished for using the code right away.

Host: We are turning our attention now to how we can build better data representations by creatively combining Variational Autoencoders with autoregressive models. Usually, there is a big catch when you mix these two. Guest: What kind of catch? Do they interfere with each other? Host: They do. Earlier research showed that if you use a powerful autoregressive model in the decoder, it tends to completely ignore the latent code, which is the compressed representation we are trying to learn. The model prefers to just do all the work itself, generating data pixel by pixel. Guest: So if the decoder does all the heavy lifting, our compressed code ends up empty or useless. How do the authors fix that? Host: They actually flip this problem into a feature, creating what they call a Variational Lossy Autoencoder. The trick is to explicitly limit the decoder's power so it can only see a tiny window of pixels around the one it is currently generating. Guest: Oh, so by restricting its view, the decoder can only figure out small, local details like texture? Host: Precisely. We intentionally assign the decoder the job of modeling the fine, local information that we do not want cluttering up our main representation. Guest: Which means the latent code is forced to step up and handle the big picture? Host: You got it. Because the decoder's receptive field is strictly limited to that small local window, any long-range dependencies, like the overall shape or structure of an image, absolutely have to be captured by the latent code. Guest: That is a really clever way to explicitly control exactly what kind of information gets saved in our final compressed representation.

Host: Let's see how we can actually put these theoretical concepts to the test by controlling how the model structures information. If we restrict our decoder to only look at a tiny local window of a 2D image, it will naturally focus on local textures, leaving the global structural information—like the shape of an object—to be entirely captured by the latent code. Guest: So by artificially limiting what the decoder can see, we are forcing the model to divide the labor? Host: Exactly, which gives us explicit control over where information is stored. You could even do the reverse by giving the decoder a blurry, down-sampled view, forcing the latent code to handle all the sharp, high-frequency local details instead. Guest: That makes sense for organizing the data, but the text also brings up inefficiencies in Bits-Back Coding. How do they handle the mismatch between the true and approximate posteriors? Host: To shrink that gap, researchers typically use "Normalizing Flows" to make the approximate posterior more complex. A flow is basically a reversible mathematical mapping that morphs a simple, predictable distribution—like standard Gaussian noise—into a highly complicated one. Guest: And the text mentions Inverse Autoregressive Flow, or IAF, as a powerful way to do that morphing. Is that what they use in this model? Host: They actually propose a clever twist by using a standard Autoregressive Flow, or AF, to parameterize the learnable *prior* instead of the posterior. It turns out that using an AF on the prior is mathematically equivalent to using an IAF on the posterior for improving that Bits-Back coding efficiency. Guest: If they're equivalent for coding efficiency, what is the advantage of applying the flow to the prior? Host: By applying it to the prior instead, you get a much more expressive generative model essentially for free. It’s an elegant solution that improves the model's overall performance while still minimizing those coding inefficiencies.

Host: We are shifting gears to look at how we test a model's ability to map out the underlying probabilities of complex data. To kick things off, the text compares two mathematical approaches for setting up our model: the Autoregressive Flow, or AF prior, and the Inverse Autoregressive Flow, or IAF posterior. Guest: Those sound incredibly technical. What is the actual practical difference between using an AF prior versus an IAF posterior? Host: The text reveals that mathematically, they cost the exact same amount of computing power during the encoding phase. But on the decoding side, the AF prior takes a deeper, more complex path, which gives us a much more expressive model at zero extra training cost. Guest: Getting better performance for free is always a win. How did the researchers test this advantage in the real world? Host: They built what they call a VLAE, or Variational Lossy Autoencoder, and evaluated it on datasets of small, binary images like the famous MNIST handwritten digits. This model combines that highly expressive AF prior with a smart, localized image-generating decoder called a PixelCNN. Guest: You mentioned the name includes the word "lossy," which usually means data is discarded during compression. Did it compress these images more aggressively than a standard model would? Host: It did, and quite significantly. The VLAE compressed the core information of an MNIST digit down to just 19.2 bits on average, whereas a standard model needed over 37 bits to do the same job. Guest: But isn't losing information generally a bad thing? Why is a much lower bit rate an advantage here? Host: It is an advantage here because it forces the main compressed code to only focus on the global picture, like the overall shape of the digit. Because that PixelCNN decoder is so powerful on its own, it can easily fill in all the fine, local pixel details without needing them explicitly spelled out in the code.

Host: Let's look at how our model handles more complex visual data, like natural photographs. When the model decompresses an image, it actually isn't creating a pixel-perfect copy of the original. Guest: If it isn't an exact copy, what exactly is the model reconstructing? Host: It regenerates the image based on the broad, global structure stored in a compressed representation called the lossy code z. The tiny local details, like exact stroke widths, are newly generated, so they might change slightly. Guest: So it captures the overall gist of the image, even if the minor textures shift around? Host: Usually yes, but this code doesn't always capture the right global information, especially in complex datasets like OMNIGLOT where tiny patch variations carry actual meaning. In those cases, the underlying semantics of the image can actually get lost. Guest: That means the compression could ruin the image if those small details matter. How do they address that flaw? Host: It highlights that we have to carefully design the decoder based on the specific statistics we care about for a given task. This directly leads into their next focus, which is improving how well the model estimates the underlying probability distribution of these images, known as density estimation. Guest: How do they attempt to improve that density estimation? Host: They investigate whether using autoregressive models for both the latent distribution and the decoding step boosts performance. To verify if their new Autoregressive Flow prior works well on its own, they first test it using a simple, standard Bernoulli decoder. Guest: That makes sense because keeping the decoder simple isolates the variables, letting them see if the latent distribution is driving the improvement. Host: Exactly, which gives them a clean baseline to verify their approach against the previous best performing models.

Host: Let's look at how well this model actually performs on compressing and generating complex images. The researchers tested their VLAE model, which uses an AF prior and a PixelCNN decoder, and found it set a new state-of-the-art on basic datasets like binarized MNIST. Guest: What exactly made it perform so much better than previous models? Host: They suspect it's because VLAE separates the visual information into two distinct parts. The latent code handles the global structure, or the big picture, while the PixelCNN handles the local statistics, like the fine pixel-level details. Guest: That sounds like a smart way to process 2D images, but does that approach hold up if you swap out the dataset? Host: It holds up remarkably well. They took the exact same settings tuned for MNIST and applied them to other datasets like OMNIGLOT and Caltech-101, and it still shattered performance records. Guest: That's impressive that they didn't even have to tweak the settings. Did they try it on natural images, like standard photographs? Host: They did, specifically targeting the CIFAR10 dataset, which is a notoriously tough benchmark for these models. To handle that extra complexity, they used a network architecture called DenseNet, which helped reduce overfitting. Guest: Did they have to change the training process to keep the model stable on such hard data? Host: Yes, they combined two existing optimization tricks, known as KL annealing and "free bits", to smooth out the learning process. Guest: And what was the final verdict on the CIFAR10 test? Host: It was a massive success. It achieved new state-of-the-art results for variationally trained models, proving just how powerful this split-responsibility design really is.

Host: Let's look at how this new model stacks up against existing research and benchmarks. The authors took the VLAE model, with its settings tuned just on one basic dataset, and tested it on several other binary image datasets. Guest: Did it still perform well without being specifically retuned for each new test? Host: It did incredibly well. It beat previous top results on datasets like dynamically binarized MNIST and Caltech-101 Silhouettes, and tied for the best on OMNIGLOT. Guest: That's impressive, but how do they know the VAE part of the model is actually doing the heavy lifting, rather than just the powerful PixelCNN decoder? Host: To isolate that, they tested the PixelCNN completely on its own as an Unconditional Decoder. This proved the VAE components were essential to the performance boost. Guest: Makes sense. Did they test it on anything more complex than those binary black-and-white images? Host: Yes, they moved on to natural color images using the CIFAR10 dataset, which is a much harder benchmark. To handle this complexity without overfitting, they built the VAE networks using a specific architecture called DenseNet. Guest: I imagine training on complex natural images gets pretty unstable, though. Host: It can, so they introduced a new optimization trick that blends two existing methods—KL annealing and free bits—to stabilize the learning. Guest: And how did those DenseNet models finally perform against the competition on CIFAR10? Host: They set a new state-of-the-art for variationally trained models. They even beat most other tractable likelihood models, coming in just slightly behind the unarchived champion, PixelCNN++.

Host: Let's explore how this model fits into the broader academic landscape, starting right after we wrap up one last fascinating visual experiment. The authors tested their model on small CIFAR10 images to see how the decoder's "receptive field" changes what the system decides to memorize. Guest: What exactly do they mean by a receptive field in this context? Host: It’s the specific block of neighboring pixels the decoder is allowed to look at when predicting the next pixel. They found that if this window is large, the decoder is great at figuring out the structure on its own, so the compressed latent code only needs to store a rough outline of the overall shape. Guest: So if they shrink that viewing window, I'm guessing the latent code has to work harder? Host: Exactly, because with a smaller receptive field, the decoder can't see enough context to build the structure itself. That forces the latent code to preserve highly detailed shape information, though interestingly, they noticed the code often completely forgets about color. Guest: Why would the latent code just drop the color information entirely? Host: Because color tends to be very consistent locally, meaning the decoder can easily guess it just by looking at those nearby pixels. But since color is vital for tasks like object classification, the authors devised a clever trick to force the latent code to remember it. Guest: How do you force the model to prioritize color again? Host: They restricted the decoder so it only sees a grayscale version of that local pixel window. Because the decoder can no longer cheat by looking at neighboring colors, the latent code has to step up and carry that color data itself. Guest: That’s a really elegant way to control exactly what the model learns. Does this specific combination of techniques build on a lot of prior research? Host: It does, and that brings us right into their related work. They position this model as a novel fusion of two major established fields: variational autoencoders that use continuous variables, and these highly structured neural autoregressive models.

Host: Before diving into the mechanics, it helps to see what other researchers were building around the same time. This paper actually sits at the intersection of a few major machine learning breakthroughs from 2016 and 2017. Guest: What kind of breakthroughs are we talking about? Host: Mainly, a family of models that generate data one piece at a time, known as autoregressive models. You might recognize names like PixelCNN for images or WaveNet for audio, which were huge leaps forward in generating highly realistic media. Guest: Got it. But if those models were already so good, why did this paper try to combine them with latent variables? Host: Because that combination gives you better high-level control over what you're generating, but it's notoriously tricky to pull off. Other researchers found that if your generator—or decoder—is too powerful, it completely ignores the broad concepts stored in those latent variables. Guest: How were people fixing that problem before? Host: Mostly with workarounds. For instance, some previous papers forced the model to rely on latent variables by randomly disabling parts of the decoder, a technique called dropout. Guest: That sounds a bit like a hack. Does this team take a different approach? Host: They do. Instead of handicapping the decoder, they use a theory called the "Bits-Back" interpretation to mathematically determine exactly when and how the latent code should be used. Guest: They also mention getting an "exact likelihood" compared to older models. What does that mean? Host: Previous models trying to use continuous latent variables had to settle for rough estimates of the data's probability. This team uses a technique called autoregressive flows to calculate those probabilities exactly, making the math much cleaner. Guest: There's also a mention of lossy compression at the end, comparing it to how a JPEG works. How does that fit into their model? Host: Previous models, like one called ConVDRAW, compressed images but lost information in a somewhat rigid way. This team's model acts as a lossy compressor too, but it uses that autoregressive, step-by-step process to explicitly control exactly what kind of detail gets thrown away to save space.

Host: Let's bring everything together by looking at the big picture of what this research achieved and where it is heading next. The authors essentially cracked the code on exactly when a Variational Autoencoder actually uses its latent space to compress data. Guest: And they used that insight to build their own upgraded model, right? Host: Exactly, they designed a new model called VLAE that acts as a powerful lossy compressor of observed data. It gives us much more control over how data is represented and significantly improves how well the model estimates data distributions. Guest: That sounds like a pure win, but is there a catch to those improvements? Host: The main tradeoff is generation speed. Because VLAE relies on a sequential autoregressive model to build outputs step by step, it is noticeably slower at generating data than a standard, simple VAE. Guest: That makes sense, so where do they plan to take this lossy compression idea in the future? Host: They see a lot of promise in applying these principles to data that has a temporal aspect, like audio and video streams. Guest: Oh, because those formats have so much complex, time-based information to compress? Host: Precisely. They also hope to design representations that strip away noise and only keep the exact information needed for specific downstream tasks, which could be a huge asset for semi-supervised learning.

Host: Looking at the research this work builds upon gives us a great map of its scientific DNA. Guest: It mostly looks like a long list of citations from around 2014 to 2016. Are there specific trends we should be noticing here? Host: Absolutely, because this list is practically a highlight reel of deep generative modeling from that era. You will see pioneers like Geoffrey Hinton and Yoshua Bengio, but the real story is in the specific concepts they cite. Guest: What kind of concepts? I see several papers mentioning autoencoders and latent variable models. Host: Exactly, there is a massive focus on Variational Autoencoders and autoregressive models. Citations for projects like DRAW and PixelVAE tell us this work relies heavily on previous breakthroughs in teaching computers to generate and compress images. Guest: I also noticed a couple of famous architecture papers in there, like the ones introducing deep residual networks and DenseNets. Why include those if the focus is on generative models? Host: Good eye. By 2017, building powerful generative models meant you needed incredibly deep neural networks to handle the math. Citing those specific architectures shows the authors were using the absolute best structural blueprints available to make their own networks deeper and more stable. Guest: So by looking at these references, we can see they are combining the latest structural network designs with cutting-edge generation techniques. Host: Precisely. It frames this work perfectly, showing how it serves as the next logical stepping stone in a very fast-moving period for machine learning.

Host: To truly understand a research paper's DNA, it helps to look at the foundational work it builds upon. Looking at this 2017 ICLR paper's bibliography, it is essentially a who's who of deep generative modeling. Guest: I usually skip right past the citations, but are there specific heavy hitters in this list that tell us what this paper is really about? Host: Absolutely, because you immediately see Diederik Kingma and Max Welling’s seminal 2013 paper introducing Variational Autoencoders, or VAEs. There are also several follow-ups listed that focus on improving how those models learn complex distributions. Guest: So VAEs are a core building block here. What about the other references, like the ones mentioning WaveNet and PixelCNN? Host: Those are crucial too, with multiple citations to researchers like Aaron van den Oord and Tim Salimans. It shows this work is heavily tied to autoregressive models, which are systems that generate highly detailed audio or images one step at a time. Guest: It’s fascinating how a citation list acts like a map of the research ecosystem. I even noticed Yann LeCun's classic 1998 document recognition paper in the mix. Host: Exactly, which proves that even the most cutting-edge generative breakthroughs from 2016 and 2017 still rely on the foundational neural network techniques established decades earlier. Guest: It really puts the pace of AI research into perspective, seeing all these massive ideas collide in one place.

Host: As we get ready to explore the exact setup for the binary image experiments, it helps to first look at the academic foundation of this work. This specific paper was actually published at ICLR in 2017, which is a major international conference for deep learning research. Guest: That is great context to have. Does the text give us a sense of whose prior research they built these experiments on? Host: Yes, it provides a block of foundational citations, heavily featuring work by Aaron van den Oord on models known as PixelRNNs and PixelCNNs. Guest: I am guessing those have something to do with creating the images? Host: Exactly right. Those are neural networks specifically designed to generate images one pixel at a time, which is directly relevant to how this paper tests its own models on simple binary, or black-and-white, image datasets. Guest: Are there any other major themes standing out in this list of references? Host: There is a very strong focus on probabilistic models, specifically "variational" methods and Bayesian inference. For example, the list points to earlier research on Variational Autoencoders for sequence classification, and even older work from 2006 on viewing computer vision as statistical inference. Guest: So by combining these traditional statistical methods with those pixel-generation networks, they are setting the stage for their own new experiments. Host: Spot on. This bibliography basically acts as a map of the authors' technical influences, showing exactly where they drew their mathematical inspiration before setting up their own lab tests.

Host: Let's look at the technical nuts and bolts of how the researchers set up their experiments for binary images. For their core architecture, they combined a Variational Autoencoder, or VAE, with a PixelCNN model using a standard ResNet setup. Guest: If they're building on existing setups, what did they actually change to make these two models work together? Host: The clever modification happens inside the VAE's decoder network. Instead of having it output a simple one-channel image map, they tweaked the decoder to output a four-channel feature map. Guest: Why would they need four channels if they're just working with standard black-and-white, one-channel images? Host: Those four extra channels provide deeper contextual information. The researchers stack them directly onto the original single-channel binary image, creating a combined five-channel input that feeds straight into the PixelCNN. Guest: Ah, so the PixelCNN is essentially processing the original image and the VAE's features at the exact same time. Is that what makes it a "Conditional" PixelCNN? Host: Interestingly, they specifically avoid that name because they aren't using the official Conditional PixelCNN architecture proposed in prior research. Instead, they built a custom version where the weights are tied across all of the model's masked convolution layers. Guest: By tied weights, do you mean the network is forced to use the exact same learned parameters across those different layers? Host: Exactly, and that restriction acts as a strong defense against overfitting on a simple dataset like statically binarized MNIST. They do mention, though, that if you were working with more complex datasets, untying those weights would actually boost your performance.

B ADDITIONAL EXPERIMENT SETUP FOR CIFAR10 Latent codes are represented by 16 feature maps of size 8x8, and this choice of spatial stochas-tic units are inspired by ResNet IAF VAE (Kingma et al., 2016). Prior distribution is factorizedGaussian noise transformed by 6 autoregressive flows, each of which is implemented by a Pixel-CNN (van den Oord et al., 2016a) with 2 hidden layers and 128 feature maps. Between every otherautoregressive flow, the ordering of stochastic units is reversed. ResNet VLAE has the following structure for encoder: 2 ResNet blocks, Conv w/ stride=2, 2 ResNetblocks, Conv w/ stride=2, 3 ResNet blocks, 1x1 convolution and has a symmetric decoder. Channelsize = 48 for 32x32 feature maps and 96 for other feature maps. DenseNet VLAE follows a similarstructure: replacing 2 ResNet blocks with one DenseNet block of 3 steps and each step producesa certain number of feature maps such that at the end of a block, the concatenated feature maps isslightly more than the ResNet VLAE at the same stage. Conditional PixelCNN++ (Salimans et al., 2017) is used as the decoder. Specifically the channel-autoregressive variant is used to ensure there is sufficient capacity even when the receptive field issmall. Specifically, the decoder PixelCNN has 4 blocks of 64 feature maps where each block isconditioned on previous blocks with Gated ResNet connections and hence the PixelCNN decoderswe use are shallow but very wide. For 4x2 receptive field experiment, we use 1 layer of verticalstack convolutions and 2 layers of horizontal stack convolutions; for 5x3 receptive field experiment,we use 2 layers of vertical stack convolutions and 2 layers of horizontal stack convolutions; For 5x3receptive field experiment, we use 2 layers of vertical stack convolutions and 2 layers of horizontalstack convolutions; For 7x4 receptive field experiment, we use 3 layers of vertical stack convolutionsand 3 layers of horizontal stack convolutions; for 7x4 Grayscale experiment, we transform RGB

images into gray-scale images via this specific transformation: (0.299* R) + (0.587G) + (0.114B). Best density estimation result is obtained with 7x4 receptive field experiments. C SOFT FREE BITS "Free bits" was a technique proposed in (Kingma et al., 2016) where K groups of stochastic unitsare encouraged to be used through the following surrogate objective: K Lx = Ex~M [Eq(z/x) [log p(x|z)]] - ∑maximum(x, Ex~M [DKL(q(zj|x)||p(zj))]) j=1 This technique is easy to use since it's usually easy to determine the minimum number of bits/nats, 入, stochastic units need to encode. Choosing A is hence easier than setting a fixed KL annealing schedule (Serban et al., 2016). On the other hand, Kl annealing has the benefit of the surrogate objective will smoothly become the true objective, the variational lower bound where as "free bits" has a sharp transition at the boundary. Therefore, we propose to still use A as hyperparameter to specify at least A nats should be used but try to change the optimization objective as slowly as possible: where 0 < x ≤ 1. LSoft FreeBits (x; 0) =Eq(z|x) [log p(x|z)] – YDKL(q(z|x)||p(z)) And we make the optimization smoother by changing y slowly online to make sure at least 入 nats are used: when Kl is too much higher than 入 (we experimented wide range of thresholds from 3% to 30%, all of which yield improved results, and we tend to use 5% us a threshold), γ is increased, and when Kl lower than λ, γ is decreased to encourage information flow. We found it sufficient to increase/decrease at 10% increment and didn't further tune this parameter. D AUTOREGRESSIVE DECODER WITHOUT AUTOREGRESSIVE PRIOR In this section, we investigate the scenario of just using an autoregressive decoder without using an autoregressive prior. We compare the exact same model in three configurations: 1) using small- receptive-field PixelCNN as an unconditional density estimator; 2) using small-receptive-field as a decoder in a VAE with Gaussian latent variables; 3) replacing Gaussian latent variables with autoregressive flow latent variables in 2). Table 1: Ablation on Dynamically binarized MNIST Model NLL Test KL Unconditional PixelCNN 87.55 0 PixelCNN Decoder + Gaussian Prior 79.48 10.60 PixelCNN Decoder + AF Prior 78.94 11.73 In Table 1, we can observe that each step of modification improves density estimation performance. In addition, using an autoregressive latent code makes the latent code transmit more information as shown in the difference of E[DKL(q(Z|X)||p(z))]. E CIFAR10 GENERATED SAMPLES

Published as a conference paper at ICLR 2017 (a) 4x2 @ 3.12 bits/dim (b) 7x4 @ 2.95 bits/dim Figure 4: CIFAR10: Generated samples for different models Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep networklearning by Exponential Linear Units (ELUs). arXiv preprint arXiv:1511.07289, 2015. Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoderfor distribution estimation. arXiv preprint arXiv:1502.03509, 2015. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-nition. arXiv preprint arXiv:1512.03385 2015. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014. Diederik P Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverseautoregressive flow. arXiv preprint arXiv:1606.04934, 2016. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. InProceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807-814,2010. Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging.SIAM Journal on Control and Optimization, 30(4):838–855, 1992. Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to ac-celerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving thepixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprintarXiv:1701.05517, 2017. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, AaronCourville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for gen-erating dialogues. arXiv preprint arXiv:1605.06069, 2016. Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks.arXiv preprint arXiv:1601.06759, 2016a. Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko-ray Kavukcuoglu. Conditional image generation with pixelcnn decoders. arXiv preprintarXiv:1606.05328, 2016b.