ImageNet Classification with Deep Convolutional Neural Networks

Abstract

A large, deep convolutional neural network trained on ImageNet achieved state-of-the-art results in image classification with top-1 and top-5 error rates of 37.5% and 17.0% respectively.

Introduction

To address the limitations of small image datasets, this paper proposes using large, deep convolutional neural networks with significant learning capacity to learn from millions of images and capture the complexity of object recognition.

Motivation and Contributions

This paper details the training of a large convolutional neural network on ImageNet, introducing novel features for improved performance and reduced training time, and making a highly optimized GPU implementation publicly available.

The Dataset

The ImageNet dataset, comprising over 15 million labeled images across 22,000 categories, was used for training and validation, with down-sampled 256x256 images processed with raw RGB values after subtracting the mean.

The Architecture

The network architecture consists of eight learned layers, including five convolutional and three fully-connected layers, designed to maximize multinomial logistic regression objective and utilize novel features for improved performance.

ReLU Nonlinearity

Rectified Linear Units (ReLUs) as a non-saturating nonlinearity significantly accelerate the training of deep convolutional neural networks compared to traditional saturating neurons like tanh.

Training on Multiple GPUs

To overcome GPU memory limitations, the network was distributed across two GPUs with restricted communication between certain layers, achieving a reduction in error rates and a slight decrease in training time.

Local Response Normalization

A local response normalization scheme, inspired by biological neurons, was applied after ReLU non-linearity to enable competition between neuron outputs, leading to improved generalization and reduced error rates.

Overlapping Pooling

Overlapping pooling, achieved by setting the pooling unit stride less than the neighborhood size, was used throughout the network, reducing error rates and making models slightly more resistant to overfitting.

Overall Architecture

The CNN architecture features five convolutional and three fully-connected layers, utilizing ReLU nonlinearity, response normalization, and max-pooling, with specific connectivity patterns between layers and GPUs.

Reducing Overfitting

To combat overfitting in the 60 million parameter network, two primary methods were employed: data augmentation and dropout.

Data Augmentation

Data augmentation techniques, including random translations, horizontal reflections, and alterations of RGB channel intensities via PCA, were used to artificially enlarge the training set and reduce overfitting.

Dropout

Dropout, a technique that randomly sets neuron outputs to zero during training, was applied to the fully-connected layers to prevent complex co-adaptations and improve feature robustness, approximating model combination.

Details of learning

The models were trained using stochastic gradient descent with momentum, weight decay, and a carefully adjusted learning rate, with weights initialized from a Gaussian distribution and biases initialized to accelerate early learning.

Results

The trained CNN achieved record-breaking top-1 and top-5 error rates on ILSVRC-2010 and ILSVRC-2012, significantly outperforming previous state-of-the-art methods.

Qualitative Evaluations

Learned convolutional kernels specialized in frequency and orientation, with distinct color-agnostic and color-specific patterns emerging on different GPUs due to restricted connectivity.

Qualitative Evaluations (cont.)

The network demonstrated effective object recognition, even for off-center objects, and identified semantically similar images based on feature activations in the last hidden layer.

Discussion

The results highlight the importance of depth in CNNs for achieving high performance and suggest that further improvements can be made by increasing network size, training time, and incorporating unsupervised pre-training or video data.

References

This section lists the references cited in the paper, providing a bibliography of related work in machine learning and computer vision.