Identity Mappings in Deep Residual Networks

Abstract

Deep residual networks utilize identity mappings as skip connections to enable direct propagation of signals, facilitating easier training and improved generalization.

Introduction

Deep residual networks learn additive residual functions with identity skip connections, enabling direct signal propagation through the entire network in both forward and backward passes.

Identity Mapping Analysis

Identity mappings in skip connections are crucial for easing optimization, while other transforms like scaling, gating, or 1x1 convolutions lead to higher training loss and errors.

Analysis of Deep Residual Networks

Deep residual networks with identity skip connections and identity after-addition activation allow direct forward and backward signal propagation, facilitating optimization and preventing gradient vanishing.

Importance of Identity Skip Connections

Modifying identity skip connections with scaling, gating, 1x1 convolutions, or dropout impedes signal propagation and leads to optimization difficulties in deep networks.

Experimental Results: Shortcut Modifications

Scaling, gating, 1x1 convolutions, and dropout on shortcut connections in deep residual networks result in higher test errors compared to the baseline identity shortcut.

Experimental Results: Gating and Convolutional Shortcuts

Exclusive gating and 1x1 convolutional shortcuts lead to worse performance and higher training errors than identity shortcuts in deep residual networks, indicating optimization issues.

Discussion on Shortcut Manipulations

Multiplicative manipulations on shortcut connections hinder information propagation and cause optimization problems, as evidenced by higher training errors compared to identity shortcuts.

Analysis of Activation Functions

The placement of activation functions (ReLU and BN) in residual units significantly impacts training and performance, with pre-activation designs showing improved results.

Experiments on Activation Function Usage

Moving BN after addition or ReLU before addition in residual units degrades performance, while using BN and ReLU as pre-activations improves results on CIFAR-10/100.

Pre-activation vs. Post-activation

Full pre-activation with BN and ReLU before weight layers in residual units leads to improved classification error rates on CIFAR-10 and CIFAR-100 compared to baseline and other activation configurations.

Figure 6 and Table 3: Training Curves and Performance

Figure 6 displays training curves for ResNet-110 and ResNet-164 on CIFAR-10, comparing BN after addition versus pre-activation, with solid lines indicating test error and dashed lines showing training loss, while Table 3 shows that pre-activation models outperform their baselines.

Impact of Pre-activation on Optimization and Regularization

Pre-activation simplifies optimization by using an identity mapping and enhances regularization through Batch Normalization, particularly evident in very deep networks where it facilitates smoother signal propagation and faster training error reduction.

Table 4: State-of-the-Art Comparisons on CIFAR Datasets

Table 4 compares ResNet performance with state-of-the-art methods on CIFAR-10 and CIFAR-100, showing that the authors' pre-activation ResNet-1001 achieved 4.92% error on CIFAR-10 and 22.71% on CIFAR-100, with a footnote noting an even better result of 4.62% on CIFAR-10.

Pre-activation as a Regularization Technique

The pre-activation unit acts as a regularizer, leading to lower test error despite slightly higher training loss by normalizing inputs to all weight layers, which helps prevent overfitting in deep networks.

Table 5: ImageNet Validation Set Performance

Table 5 presents single-crop error rates on the ILSVRC 2012 validation set, where the authors' pre-activation ResNet-200 achieved a top-1 error of 20.7%, outperforming the original ResNet-152 and ResNet-200, and surpassing Inception v3 when using scale and aspect ratio augmentation.

Conclusions and Implementation Details

Deep residual networks with identity shortcut connections and pre-activation are trainable and achieve improved accuracy, with detailed implementation hyperparameters provided for CIFAR and ImageNet experiments.