Recurrent Neural Network Regularization

Abstract

A simple regularization technique using dropout is presented for Recurrent Neural Networks with LSTM units, significantly reducing overfitting across various tasks.

1 Introduction

This work demonstrates that dropout, when correctly applied, greatly reduces overfitting in LSTMs, addressing the limitations of existing regularization methods for RNNs.

2 Related Work

This paper shows that by applying dropout only to specific RNN connections, the problem of conventional dropout hurting RNN learning is fixed, allowing RNNs to benefit from regularization.

Introduction to Tasks

This work focuses on applying dropout to LSTMs, a common RNN variant, and evaluates its effectiveness on language modeling, speech recognition, and machine translation.

3 Regularizing RNNs with LSTM Cells

This section describes deep LSTMs and introduces a regularization scheme for them, explaining its effectiveness.

3.2 Regularization with Dropout

The proposed method applies dropout to non-recurrent connections in LSTMs, forcing more robust intermediate computations without erasing long-term memory.

Dropout and RNNs

Unlike standard dropout which perturbs recurrent connections and hinders memory, this method avoids dropout on recurrent connections, allowing LSTMs to benefit from regularization while retaining memorization ability.

4 Experiments

Experiments are presented for language modeling, speech recognition, machine translation, and image caption generation to evaluate the proposed regularization technique.

4.1 Language Modeling

Regularized and non-regularized LSTMs were trained on the Penn Tree Bank dataset, with detailed configurations and training parameters provided for comparison.

Speech Recognition

Dropout improves the frame accuracy of LSTMs for acoustic modeling in speech recognition on a small dataset, leading to better generalization despite a drop in training accuracy.

4.3 Machine Translation

Applying dropout to an LSTM for machine translation improves translation performance, though it does not surpass the best phrase-based SMT system.

4.4 Image Caption Generation

In image caption generation, dropout improves single model performance to match that of an ensemble, demonstrating its effectiveness in this domain.

5 Conclusion

A simple and effective method for applying dropout to LSTMs is presented, yielding significant performance improvements across various tasks and applications.

6 Acknowledgments

The authors acknowledge Tomas Mikolov for his valuable comments on the initial version of the paper.

REFERENCES

This section lists all the references cited in the paper, covering various aspects of neural networks, recurrent neural networks, and dropout.

REFERENCES

This section continues the list of references, including papers on language modeling, speech recognition, machine translation, and dropout techniques.