Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
This paper proposes a method for training neural networks that minimizes the description length of the weights, which is shown to improve generalization by making the network simpler. The method adapts the prior distribution of weights based on the data, allowing for more flexible coding and better performance.
Episodes
Chapters
01Abstract
Neural networks generalize well when their weights contain less information than the training data outputs, necessitating methods to penalize information in weights during learning.
1:51Explained021 Introduction
Limiting information in neural network weights is crucial for generalization, especially with limited training data, and various methods like limiting connections, weight-sharing, and quantization have been explored.
1:36Explained032 Applying the Minimum Description Length Principle
The Minimum Description Length Principle suggests that the best model minimizes the combined cost of describing the model (weights) and the misfit between the model and the data.
1:41Explained04Figure 1
Figure 1 illustrates the probability mass associated with a quantized value, showing its approximation by the product of height and width for narrow quantization widths.
1:21Explained05Square Deviation
Minimizing the squared error function aligns with the MDL justification for using Gaussian assumptions in coding.
1:15Explained063 Coding the data misfits
Data misfits are assumed to be finely quantized and encoded separately for each output unit, with Gaussian distributions used for modeling.
1:48Explained074 A simple method of coding the weights
A simple method for coding weights involves assuming a Gaussian distribution, leading to a cost function proportional to the sum of squared weights, which is equivalent to the standard weight-decay method.
1:33Explained08Elaboration of Weight-Decay
An elaboration of weight-decay involves using a mixture of Gaussians to model the weight distribution more accurately, especially when only a few distinct weight values are needed.
1:25Explained095 Noisy weights
Adding Gaussian noise to weights can significantly reduce the information content, and the MDL framework can be adapted to communicate these noisy weights cheaply.
1:40Explained105.1 The expected description length of the weights
The expected description length of noisy weights is equal to the Kullback-Liebler distance between the prior and posterior Gaussian distributions.
1:31Explained115.2 The "bits back" argument
The 'bits back' argument accounts for the random bits used to collapse the posterior distribution, effectively reducing the true communication cost of noisy weights.
1:31Explained12Divergence Formula
The asymmetric divergence between two Gaussians is calculated using their means and variances.
1:23Explained135.3 The expected description length of the data misfits
For networks with one hidden layer and linear output units, the expected squared error of data misfits can be computed exactly using pre-computed tables for means and variances.
1:50Explained146 Letting the data determine the prior
The coding-prior distribution for weights can be adapted during optimization, making it dependent on the data rather than fixed in advance.
1:38Explained156.1 A more flexible prior distribution for the weights
A mixture of Gaussians can provide a more flexible prior distribution for weights, allowing for better encoding of structured weight distributions than a single Gaussian.
1:50Explained16Adaptive Mixture of Gaussians
An adaptive mixture of Gaussians models weight clusters, allowing for efficient encoding of low-variance weights near specific values.
1:38Explained177 A coding scheme that uses a mixture of Gaussians
A coding scheme using a mixture of Gaussians involves randomly picking a Gaussian based on probabilities and communicating the sample value, with 'bits back' accounting for reconstructed posterior distributions.
1:32Explained187.1 A lemma from statistical mechanics
A lemma from statistical mechanics relates the expected code cost to the Helmholtz free energy, where probabilities are exponentially related to energies, and the minimum free energy equals minus the log of the partition function.
1:45Explained198 Implementation
Implementing an adaptive mixture of Gaussians prior requires careful calculation of derivatives and appropriate table sizes, verified through semantic checks.
1:39Explained209 Preliminary Results
Preliminary results show that the adaptive mixture of Gaussians method can fit complex non-linear models to high-dimensional data with scarce training examples, outperforming simple weight-decay.
1:28Explained21Figure 3
Figure 3 displays the final probability distribution used for coding weights, implemented by adapting five Gaussian means, variances, and mixing proportions.
1:42Explained22Figure 2
Figure 2 visualizes the final weights of the network, showing that weights form distinct clusters, and the adaptive mixture of Gaussians successfully models this distribution.
1:43Explained23Comparison of Methods
The new method shows slightly better performance than simple weight-decay and linear regression with optimized regularization on a high-dimensional task with limited data.
2:04Explained2410 Discussion
While intractable, the ideal Bayesian method involves sampling from the posterior distribution, and approximations can be made using Monte Carlo methods or by assuming simpler posterior distributions like Gaussians.
1:09Explained2511 Acknowledgements
This research was funded by NSERC, and the authors acknowledge helpful discussions with colleagues.
1:35Explained