Distilling the Knowledge in a Neural Network

Abstract

Knowledge from an ensemble of models can be compressed into a single, more deployable model through distillation, achieving significant performance improvements.

Introduction to Distillation

Distillation transfers knowledge from a cumbersome, well-generalized model to a smaller, deployable model by training the smaller model to mimic the larger model's softened predictions.

Distillation with Soft Targets

Distillation uses a high-temperature softmax to create soft targets, which provide more information than hard targets and are used to train a smaller model that matches these targets.

Distillation and Logits

In the high-temperature limit, distillation is equivalent to minimizing the squared difference between logits, with intermediate temperatures being optimal when the distilled model is too small to capture all knowledge.

Empirical Results on MNIST

Distillation successfully transfers knowledge to a smaller MNIST model, achieving performance close to a larger regularized model and retaining knowledge of unseen classes.

Distillation of DNN Acoustic Models

Distillation effectively transfers knowledge from an ensemble of DNN acoustic models to a single model, significantly improving performance over a directly trained model of the same size.

Table 1: Performance of Distilled Models

A distilled single model achieves performance comparable to an ensemble of ten models, demonstrating the effectiveness of transferring ensemble knowledge.

Specialist Models for Large Datasets

Specialist models, trained on specific confusable subsets of classes, can reduce the overall computation required to learn an ensemble for very large datasets.

Challenges with Large Datasets

Training on extremely large datasets like JFT is computationally intensive, necessitating faster methods to improve baseline models beyond traditional ensemble training.

Ensemble of Generalist and Specialist Models

A cumbersome model for large class sets can be an ensemble of a generalist model and many specialist models trained on specific confusable class subsets.

Clustering for Specialist Models

Clustering the covariance matrix of generalist model predictions is used to derive groupings of object categories for training specialist models.

Ensemble Classification with Specialists

A two-step process involving a generalist model and relevant specialist models is used for classification, minimizing KL divergence to find the optimal probability distribution.

Table 3: Specialist Model Performance

Combining a baseline system with specialist models yields a significant relative improvement in test accuracy, with more specialists per class leading to greater accuracy gains.

Soft Targets and Data Efficiency

Soft targets enable a new model to generalize well from significantly less data compared to using hard targets, retaining knowledge about the full dataset.

Table 5: Generalization with Soft Targets

Soft targets allow a model to generalize effectively from only 3% of the training data, nearly recovering the performance of a model trained on the full dataset.

Specialists vs. Mixtures of Experts

Specialist models are easier to parallelize than mixtures of experts because their training is independent once class subsets are defined.

Conclusion and Future Work

Distillation effectively transfers knowledge from ensembles or large models to smaller ones, and specialist models improve performance on large datasets, with potential for distilling specialist knowledge back into a large net.