Training Very Deep Networks

Abstract

Highway networks enable the training of extremely deep neural networks by allowing unimpeded information flow through adaptive gating units.

1 Introduction & Previous Work

Deep neural networks have achieved significant breakthroughs in supervised machine learning, but training them effectively remains a challenge addressed by various optimization, initialization, and architectural strategies.

Introduction to Highway Networks

Highway networks are introduced as a solution to the difficulties in training very deep feed-forward networks by incorporating an LSTM-inspired gating mechanism to facilitate information flow.

2 Highway Networks

Highway networks modify plain feedforward layers by introducing transform and carry gates, allowing for a more flexible transformation of input by adaptively combining input transformation with direct input passage.

2.1 Constructing Highway Networks

Highway networks can be constructed to maintain dimensionality through techniques like sub-sampling, zero-padding, or using plain layers for transformation, with convolutional highway layers utilizing shared weights and local receptive fields for both transform and carry gates.

2.2 Training Deep Highway Networks

Deep highway networks can be trained effectively using SGD by initializing the transform gates with a negative bias, encouraging initial carry behavior and facilitating learning even for networks with hundreds of layers.

3 Experiments

Experiments were conducted using SGD with momentum and decaying learning rates on MNIST and CIFAR datasets to evaluate the performance of highway networks compared to plain networks and state-of-the-art methods.

3.1 Optimization

Highway networks demonstrate superior optimization capabilities compared to plain networks, maintaining performance with increasing depth and converging significantly faster.

3.2 Pilot Experiments on MNIST Digit Classification

10-layer convolutional highway networks achieved competitive performance on MNIST digit classification with fewer parameters than state-of-the-art methods.

3.3 Experiments on CIFAR-10 and CIFAR-100 Object Recognition

Highway networks can be trained effectively in a single stage to achieve high accuracy on CIFAR datasets, outperforming previous methods that required complex two-stage training procedures.

3.3.2 Comparison to State-of-the-art Methods

Highway networks achieve competitive results on CIFAR-10 and CIFAR-100 object recognition tasks using standard data augmentation techniques and a simplified network structure.

4 Analysis

Analysis of trained highway networks reveals that transform gates learn to route information dynamically, with biases influencing selectivity and layer outputs forming stable 'information highways'.

4.1 Routing of Information

Trained highway networks exhibit data-dependent routing, where different blocks are utilized for different inputs, demonstrating that the gating system is crucial for computation, not just easing training.

4.2 Layer Importance

Lesioning experiments show that for complex datasets like CIFAR-100, highway networks utilize most of their layers, while for simpler datasets like MNIST, many layers become idle, indicating efficient depth utilization.

5 Discussion

Highway networks offer a direct training approach with simple gradient descent, overcoming limitations of other depth-handling methods by enabling adaptive information routing through multiplicative gating mechanisms.

Discussion Continuation

Highway networks allow for the examination of necessary computation depth for different problems and their gating mechanism enables useful computations even in deep, narrow architectures.