Identity Mappings in Deep Residual Networks

Abstract

Identity mappings in deep residual networks enable direct signal propagation, facilitating training and improving generalization.

Introduction

Deep residual networks leverage residual functions with identity skip connections to learn additive residuals, enabling the training of very deep architectures.

Analysis of Deep Residual Networks

When both skip connections and after-addition activations are identity mappings, signals can propagate directly through the network, easing optimization.

Discussions

Direct signal propagation is facilitated by identity skip connections and identity after-addition activation, forming clean information paths.

On the Importance of Identity Skip Connections

Modifying identity skip connections with scaling, gating, or 1x1 convolutions impedes signal propagation and leads to optimization difficulties.

Experiments on Skip Connections

Experiments show that constant scaling, exclusive gating, shortcut-only gating, 1x1 convolutional shortcuts, and dropout on shortcuts degrade performance compared to identity skip connections.

On the Usage of Activation Functions

The placement of activation functions, specifically in relation to element-wise addition, significantly impacts training and performance in deep residual networks.

Experiments on Activation

Pre-activation with Batch Normalization and ReLU, compared to post-activation, eases optimization and improves regularization, leading to better results.

Analysis

Pre-activation eases optimization for very deep networks and improves regularization, reducing overfitting and enhancing generalization.

Conclusions

Identity shortcut connections and identity after-addition activation are crucial for smooth information propagation in deep residual networks, enabling training of extremely deep models.