How transferable are features in deep neural networks?

Abstract

Features in deep neural networks exhibit varying degrees of generality and specificity across layers, with transferability impacted by neuron specialization and optimization difficulties.

Introduction

This paper investigates the transition from general to specific features in deep neural networks, quantifying layer-wise transferability and exploring its implications for transfer learning.

Generality vs. Specificity Measured as Transfer Performance

Generality of learned features is defined and measured by their performance when transferred to a new task, using pairs of ImageNet subsets to create similar and dissimilar tasks.

Experimental Setup

A standard Caffe implementation is used for experiments to study feature transferability on a well-known convolutional network architecture.

Results and Discussion

Experiments reveal that feature transferability decreases with layer depth due to specialization and optimization challenges, but fine-tuning transferred features can improve generalization.

Similar Datasets: Random A/B splits

On similar datasets, first and second layer features transfer well, but performance degrades in deeper layers due to a combination of co-adaptation loss and feature specificity.

Dissimilar Datasets: Splitting Man-made and Natural Classes Into Separate Datasets

Transfer performance significantly declines with increasing task dissimilarity, especially for higher layers, indicating that feature specificity becomes more dominant.

Random Weights

Random weights in deeper layers of a convolutional network yield near-chance performance, and even distant task transfers outperform random weights.

Conclusions

Feature transferability quantifies generality and specificity, showing that optimization difficulties and task specialization impact transfer, while even distant features and fine-tuning offer performance benefits.