LSTM: A Search Space Odyssey

# LSTM: A Search Space Odyssey. Klaus Greff, Rupesh K. Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber. This paper presents a large-scale empirical analysis of popular Long Short-Term Memory, or LSTM, variants. Several variants of the LSTM architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years these networks have become state-of-the-art models for a variety of sequence learning problems. This renewed interest motivates an investigation into the roles and utility of various computational components found in typical LSTM variants. We evaluate eight LSTM variants across three representative tasks, namely speech recognition, handwriting recognition, and polyphonic music modeling. For each variant and task we separately optimized hyperparameters using random search and assessed hyperparameter importance using the fANOVA framework. In total we summarize 5,400 experimental runs, corresponding to roughly 15 years of single-CPU computation time. Our main findings are that none of the studied variants significantly improves upon the standard vanilla LSTM, and that the forget gate and the output activation function are among its most critical components. We further observe that the hyperparameters examined are largely independent, and we derive practical guidelines for their efficient adjustment.

# Introduction and Motivation. Recurrent neural networks with Long Short-Term Memory, or LSTMs, are effective and scalable for many sequential-data learning problems. Earlier methods often lacked scalability to long time dependencies or were tailored to specific tasks, while LSTMs are both general and effective at capturing long-term temporal dependencies. LSTMs avoid the difficult optimization issues that affect simple recurrent networks and have driven state-of-the-art results in handwriting recognition and generation, language modeling and translation, acoustic modeling of speech, protein secondary structure prediction, audio analysis, and video data. The central idea of the LSTM architecture is a memory cell that can maintain its state over time together with non-linear gating units that regulate information flow into and out of the cell. Modern LSTM studies incorporate many improvements made since the original formulation, but LSTMs are now applied to diverse problems that differ significantly in scale and nature from those on which improvements were originally tested. A systematic study of the utility of LSTM computational components was missing prior to this work, so the paper addresses the open question of whether the LSTM architecture can be improved by modifying these components. We evaluate the most popular LSTM architecture, which we call the vanilla LSTM, along with eight derived variants, where each variant differs from the vanilla LSTM by exactly one change. This design allows isolating the effect of individual changes on performance while comparing variants on the same footing.

# Vanilla LSTM Overview. The vanilla LSTM setup most commonly used in the literature incorporates several improvements made after the original proposal and uses full gradient training with backpropagation through time. A standard LSTM block features three gates named input, forget, and output, a block input term, a single cell sometimes called the Constant Error Carousel, an output activation function, and optional peephole connections that link the cell to the gates. The outputs of the block are recurrently connected back to the block input and to all the gates. Gate activation functions are typically logistic sigmoid functions that squash values to the unit interval, while input and output activation functions are commonly hyperbolic tangents. At each time step, pre-activation signals for the block input and each gate are computed from the current input, the previous block output, optional peephole contributions from the previous cell state, and biases. The block input is passed through the input activation function and multiplied element-wise by the input gate to determine new candidate cell content, while the forget gate controls how much of the previous cell state is retained. The cell state is updated by adding the gated new candidate content to the gated previous state. The block output is formed by applying the output activation function to the new cell state and multiplying by the output gate. The paper also sketches the backpropagation-through-time computation of deltas inside the block and the computation of gradients for input, recurrent, peephole, and bias parameters used during training.

# History and Variants of LSTM. The original LSTM block included cells and input and output gates but omitted the forget gate and peephole connections, and earlier training used truncated gradient methods rather than full backpropagation through time. Subsequent modifications added the forget gate, which allows the LSTM to reset its own state and thereby to learn continual tasks and decouple memory duration from input history. Peephole connections, which are direct connections from the cell to its gates, were proposed to allow the cell state to influence gate timing and thus improve the modeling of precise time dependencies. Later work provided full gradient BPTT training for the improved architecture and enabled robust implementation and gradient checking. Beyond these canonical changes, other variants have been proposed such as training with extended Kalman filters or evolutionary methods, architectures with linear projection layers to reduce parameter count, scalable gate slope parameters, and recurrent connections between gates inside a block. A prominent simplification is the Gated Recurrent Unit or GRU, which couples input and forget gating into a single update gate and omits peepholes and the output activation in its common variant, producing a different trade-off between parameter efficiency and functional components. Prior comparative studies provided mixed results between GRUs and vanilla LSTMs, motivating a thorough evaluation across datasets and careful hyperparameter tuning, which this paper undertakes.

# Evaluation Setup and Datasets. The study focuses on fair empirical comparison rather than pushing state-of-the-art results, so the experimental setup is kept simple and consistent across variants. Each evaluated variant differs from the vanilla LSTM by a single change, enabling direct attribution of performance differences to that change. Experiments are run on three datasets representing different sequence modeling domains, and for each variant-dataset pair the hyperparameters are tuned individually using random search to find good-performing settings. Random search is easy to implement, trivially parallelizable, and it provides uniform coverage that supports later analysis of hyperparameter importance. Datasets used are TIMIT for frame-wise phoneme classification of speech, the IAM Online Handwriting Database for handwriting-to-character sequence mapping of pen movements, and the JSB Chorales dataset for polyphonic music modeling and next-step prediction. For TIMIT the standard MFCC preprocessing is used producing 39-dimensional inputs per frame and the task is 61-way phone classification; the dataset is split according to established core test sets and SA dialect sentences are removed from training to avoid bias. IAM Online supplies online pen stroke sequences labeled with 81 ASCII character outputs and training uses Connectionist Temporal Classification loss with best-path decoding and character error rate as evaluation. JSB Chorales uses preprocessed piano-rolls sampled every quarter note and the networks are trained for next-step prediction minimizing negative log-likelihood. Network architectures are bidirectional LSTMs for TIMIT and IAM Online, and a single LSTM hidden layer for JSB Chorales, with output layers appropriate to each task.

# LSTM Variants and Hyperparameter Search. The baseline vanilla LSTM employs sigmoid gate activations and hyperbolic tangent input and output activations, optional peepholes, and full BPTT. Eight variants are derived from the vanilla model, each implementing exactly one modification: no input gate, no forget gate, no output gate, no input activation function, no output activation function, coupled input and forget gates (CIFG), no peephole connections, and full gate recurrence which adds recurrent connections among gates. CIFG corresponds to setting the forget gate equal to one minus the input gate, as in some GRU-like formulations, reducing parameter count. Full gate recurrence reintroduces a feature from the original LSTM proposal that adds nine recurrent weight matrices and substantially increases model parameters. Hyperparameters tuned with random search include the number of LSTM blocks per hidden layer sampled log-uniformly, the learning rate sampled log-uniformly across several orders of magnitude, momentum specified via a transformed variable, and the standard deviation of Gaussian input noise sampled uniformly in a range. Each of the nine variants was tuned on each of the three datasets via 200 random trials, for a total of 5,400 trials, and additional boolean settings such as gradient clipping and momentum style were evaluated and selected based on observed effects.

# Results Summary and Variant Comparison. All 5,400 experiments were run on CPU cores and consumed roughly 15 CPU-years of computation time, with each trial taking on average about 24 hours. For TIMIT the best test error observed among trials was close to previously reported LSTM results, while on JSB Chorales and IAM Online the best results varied compared to prior work but are not the main focus because the study emphasizes fair comparison rather than absolute state-of-the-art performance. To compare variants we used Welch’s t-test adjusted for multiple comparisons to identify significant differences in mean test set performance compared to vanilla LSTM, and we analyzed both the full distributions over random trials and the top 10% of trials according to validation set performance to focus on reasonable hyperparameter tuning. The most striking observations are that removing the forget gate or removing the output activation function significantly degrades performance on all datasets, indicating that the ability to forget and the squashing nonlinearity on the output are critical LSTM components. Coupling input and forget gates did not significantly change mean performance and sometimes slightly improved the best results on music modeling, while removing peepholes had little effect and occasionally improved handwriting recognition outcomes. Full gate recurrence generally did not help and often worsened results despite greatly increasing parameter count, so its use is discouraged in practice. Variants that remove the input gate, output gate, or input activation function hurt performance on speech and handwriting domains but showed no consistent negative effect on music modeling, suggesting domain-dependent importance for some components.

# Hyperparameter Importance and Interactions. We used the fANOVA framework with random regression forests to assess the importance of hyperparameters and to estimate marginal performance effects while averaging over other parameters. The learning rate emerged as the dominant hyperparameter, accounting for the majority of test performance variance across datasets. There is often a broad basin of good learning rates spanning up to two orders of magnitude, and training time decreases toward the higher end of that basin, so a practical tuning strategy is to start with a high learning rate and reduce it by factors of ten until performance no longer improves. Hidden layer size is the next most important factor, with larger networks tending to perform better but with diminishing returns and increased training time. Additive Gaussian input noise was generally unhelpful and slightly increased training time, with a small beneficial effect on TIMIT for moderate noise levels. Surprisingly, momentum had negligible effect on both performance and training time in the studied online stochastic gradient setting, contributing less than one percent of variance in most cases. Analysis of hyperparameter interactions showed modest contributions to variance, with learning-rate-by-hidden-size interactions present but relatively small, which implies that hyperparameters can be tuned approximately independently in practice. This independency suggests that the learning rate can be tuned on smaller networks and then applied to larger ones to save experimentation time.

# Conclusion and Practical Recommendations. The large-scale study concludes that the common vanilla LSTM architecture performs robustly across tasks and that none of the eight investigated single-change modifications produce significant, consistent improvements. However, certain simplifications such as coupling input and forget gates or omitting peephole connections simplify the model and reduce parameters without significantly harming performance, making them attractive in resource-constrained settings. The forget gate and the output activation function are identified as the most critical components of the LSTM; removing either harms performance substantially, likely because without the output nonlinearity the cell state can grow unbounded and destabilize learning. The learning rate is by far the most crucial hyperparameter to tune, followed by network size, while momentum appears unimportant under online stochastic gradient descent in the studied setup. Hyperparameter interactions are small enough that practical tuning can treat most hyperparameters as approximately independent, enabling efficient sequential tuning such as setting learning rate on small networks. The study backs several widely used intuitions with systematic empirical evidence and provides actionable guidance for architecture selection and hyperparameter tuning for practitioners working with LSTM networks.