Improved Techniques for Training GANS

Abstract

New techniques for training Generative Adversarial Networks (GANs) achieve state-of-the-art results in semi-supervised classification and generate high-quality images.

1 Introduction

Generative adversarial networks (GANs) learn generative models through a game-theory approach, but traditional gradient descent methods often fail to converge to a Nash equilibrium, leading to instability and poor sample generation.

2 Related work

This work builds upon existing GAN research, incorporating architectural innovations and exploring feature matching, minibatch features, and virtual batch normalization for improved stability and semi-supervised learning performance.

3 Toward Convergent GAN Training

Training GANs involves finding a Nash equilibrium in a non-convex game, which is difficult for standard gradient descent; this section introduces techniques to encourage convergence.

3.1 Feature matching

Feature matching stabilizes GAN training by making the generator match the statistics of real data features, preventing overtraining on the discriminator.

3.2 Minibatch discrimination

Minibatch discrimination prevents generator collapse by allowing the discriminator to consider multiple examples in combination, enhancing sample diversity and visual appeal.

3.3 Historical averaging

Historical averaging modifies the cost function to include past parameter values, inspired by fictitious play, to help find equilibria in non-convex games.

3.4 One-sided label smoothing

One-sided label smoothing modifies classification targets to improve GAN training by preventing the generator from producing samples that are erroneously classified as real.

3.5 Virtual batch normalization

Virtual batch normalization normalizes generator outputs using a fixed reference batch, decoupling the normalization from the current minibatch to avoid issues with batch-dependent statistics.

4 Assessment of image quality

Evaluating GAN performance is challenging due to the lack of an objective function; this section proposes a visual Turing test with human annotators and an automatic Inception score metric.

5 Semi-supervised learning

Semi-supervised learning is performed by adding a "generated" class to the classifier and jointly minimizing supervised and unsupervised losses, where the unsupervised loss is the standard GAN game-value.

5.1 Importance of labels for image quality

Using semi-supervised learning improves generated image quality by biasing the discriminator to focus on features important for object recognition, similar to human perception.

6 Experiments

Experiments were conducted on MNIST, CIFAR-10, SVHN, and ImageNet datasets to evaluate semi-supervised learning and sample generation capabilities.

6.1 MNIST

On MNIST, feature matching achieved state-of-the-art semi-supervised classification, while minibatch discrimination improved visual sample quality but not classification accuracy.

6.2 CIFAR-10

On CIFAR-10, proposed techniques improved semi-supervised learning performance and visual sample quality, with the Inception score correlating well with subjective image quality assessments.

6.3 SVHN

Using the same architecture and experimental setup as CIFAR-10, SVHN results demonstrated competitive performance against previous state-of-the-art methods.

6.4 ImageNet

On ImageNet with high resolution and many classes, modified GANs learned to generate recognizable objects, a significant advancement over previous models.

7 Conclusion

This work addresses GAN training instability and evaluation metric limitations, achieving state-of-the-art semi-supervised learning results and providing practical solutions for improved GAN performance.