Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Abstract

An end-to-end deep learning approach successfully recognizes English and Mandarin speech, achieving human-competitive accuracy and enabling efficient deployment.

1 Introduction

Deep Speech 2, an end-to-end deep learning system, approaches human accuracy in speech recognition across languages and is deployable in production.

Deep Speech 2 Contributions

Deep Speech 2 advances speech recognition through model architecture, large datasets, and computational scale, achieving significant error rate reductions and competitive performance.

2 Related Work

This work builds upon decades of deep learning and speech recognition research, incorporating RNNs, CTC loss, and scalable training techniques.

3 Model Architecture

The Deep Speech 2 architecture utilizes deep networks with convolutional and bidirectional recurrent layers, optimized with Batch Normalization and a novel curriculum learning strategy.

3.2 Batch Normalization for Deep RNNs

Batch Normalization significantly accelerates training and improves generalization error in deep recurrent networks for speech recognition.

3.3 SortaGrad

SortaGrad, a curriculum learning strategy, improves training stability and performance by processing utterances in increasing order of length.

3.4 Comparison of simple RNNs and GRUs

Gated Recurrent Units (GRUs) outperform simple RNNs in accuracy for a fixed parameter count in deep speech recognition models.

3.5 Frequency Convolutions

Convolutions in both frequency and time domains, particularly 2D-invariant convolutions, substantially improve ASR performance on noisy data.

3.6 Striding

Utilizing non-overlapping bigrams allows for larger strides in English speech recognition without sacrificing Word Error Rate, improving computational efficiency.

3.7 Row Convolution and Unidirectional Models

Row convolution enables unidirectional RNNs to achieve performance comparable to bidirectional models, facilitating easier deployment in online settings.

Figure 3: Row Convolution Architecture

The row convolution layer, placed above recurrent layers, aggregates information efficiently and improves CER by leveraging future context.

RNN Language Model and External LM Integration

RNNs learn implicit language models with strong spelling proficiency, but an external n-gram language model trained on text data further improves Word Error Rate.

Table 6: WER/CER Comparison with External Language Model

External language models significantly improve speech recognition performance, with diminishing relative gains in more complex networks.

End-to-End Mandarin Character Recognition

An end-to-end Mandarin system directly outputs characters, eliminating the need for pronunciation models and simplifying adaptation to new languages.

High-Performance Training System

A highly optimized training system with a C++ deep learning library and CUDA/C++ linear algebra library achieves 45% of theoretical peak throughput on GPUs.

Synchronous Data-Parallelism Training

Synchronous SGD with data-parallelism across GPUs ensures reproducibility and effective scaling for training deep learning models.

Figure 4: Training Epoch Time Scaling

Training time scales near-linearly with the number of GPUs, demonstrating effective weak scaling, supported by an optimized all-reduce operation.

Table 7: Custom All-Reduce Implementation Performance

A custom all-reduce implementation is 2.5 times faster than OpenMPI's for training runs, significantly improving productivity and scalability.

Table 8: CTC Loss Computation Time Comparison

The GPU implementation of the CTC loss function significantly reduces training time, offering substantial productivity gains.

GPU Implementation of CTC Loss Function

A GPU implementation of the CTC loss function, developed to overcome CPU-GPU transfer bottlenecks and computational demands, significantly speeds up training.

Custom Memory Allocator for GPU and CPU

A custom memory allocator using a buddy algorithm for preallocated blocks significantly reduces overhead for large allocations on both CPU and GPU.

Table 9: English Speech Training Datasets

The English DS2 system is trained on a total of 11,940 hours of labeled speech data from various corpora.

GPU Memory Management for Deep Networks

A custom memory allocator and a fallback strategy using page-locked CPU memory enable training of deep networks that exceed GPU memory capacity.

Large-Scale Speech Datasets for Training

The English and Mandarin systems are trained on extensive labeled datasets, comprising 11,940 hours for English and 9,400 hours for Mandarin.

Alignment, Segmentation, and Filtering Pipeline

A pipeline using CTC-aligned bidirectional RNNs segments long audio clips and filters erroneous transcriptions to improve training data quality.

Speech Data Segmentation and Filtering

Utterances are segmented at silence and word boundaries, and erroneous examples are filtered based on word-level edit distance to achieve a WER below 5%.

Noise Augmentation for Robustness

Adding synthesized noise to 40% of randomly selected utterances improves system robustness to noisy speech.

Table 10: Impact of Training Data Size on WER

Increasing labeled training data size by tenfold reduces WER by approximately 40% relative, with consistent gains for both regular and noisy datasets.

Diverse Test Sets for Evaluation

The speech system is evaluated on a variety of test sets, including public benchmarks and internal collections, covering challenging speech environments.

Table 10: English WER vs. Training Data Size

English WER consistently decreases with increasing training data size for both regular and noisy development sets, indicating improved generalization.

Training Parameters and Decoding

Models are trained with SGD and Nesterov momentum, annealed learning rates, and language models, with decoding parameters tuned on a development set.

DS2 Model Architecture and Human Benchmark

The 11-layer DS2 English model outperforms humans on most read speech test sets, demonstrating competitive performance with human workers.

Table 11: Impact of Model Size on WER

English speech system performance consistently improves with model size, reaching optimal generalization error at approximately 100 million parameters.

Table 12: DS1 vs. DS2 WER on Internal Dataset

The 100 million parameter DS2 model achieves a 43.4% relative improvement over the DS1 model on an internal Baidu dataset.

RNN vs. GRU and Model Size Impact

The RNN model is faster and performs better than the GRU network for 100 million parameters, achieving the lowest generalization errors.

Table 13: WER Comparison on Read Speech

The DS2 system outperforms humans on most read speech test sets, indicating limited room for improvement without domain-specific adaptation.

Table 13: DS1 vs. DS2 vs. Human on Read Speech

DS2 consistently outperforms DS1 and is competitive with or better than human performance on read speech across various test sets.

Table 14: WER Comparison on Accented Speech

DS2 significantly improves performance on accented speech compared to DS1, but human performance remains superior for most accents.

Table 15: WER Comparison on Noisy Speech

DS2 substantially improves performance on noisy speech over DS1, but still lags behind human-level performance, especially in real noisy environments.

VoxForge Accented Speech Evaluation

Performance on VoxForge accented speech test sets shows improvement with accented training data, but human performance remains significantly better for most accents.

CHiME Challenge Evaluation on Noisy Speech

DS2 shows significant improvement over DS1 on noisy speech from the CHiME challenge, yet its performance on real noisy environments is still below human level.

Table 16: Mandarin Chinese Architecture Comparison

Deeper Mandarin models with 2D-invariant convolution and BatchNorm outperform shallow RNNs, mirroring trends observed in the English system.

Mandarin Speech System vs. Human Performance

The best Mandarin system transcribes voice-query utterances more accurately than a group of humans and significantly outperforms single human transcribers.

Low-Latency Mandarin Speech Recognition

Modifications to the network and decoding procedure enable a Mandarin system to achieve comparable performance with significantly lower latency for real-time applications.

Efficient Deployment of Deep Neural Networks

Individual request processing for deep learning models is computationally inefficient, suffering from reduced arithmetic intensity and limited parallelism.

Deployment Challenges for RNNs

Evaluating RNNs for real-time applications is challenging due to their sequential nature, requiring the entire utterance for processing and computationally expensive decoding.

Figure 5: Batching Probability and Eager Batching

Batching is most effective under heavy server load, with an eager batching scheme prioritizing low end-user latency over computational efficiency by processing batches as soon as they are ready.

Figure 6: Latency vs. Server Load with Batch Dispatch

Batch Dispatch maintains low median and 98th percentile latencies even with increasing server load by optimizing batch efficiency, enabling high throughput and low latency deployment of large models.

Figure 7: Throughput of Matrix Multiply Kernels

Custom half-precision matrix-matrix multiply kernels outperform standard libraries like CUBLAS and Nervana Systems on GPUs for deployment batch sizes, optimizing for bandwidth and cache usage.

Section 7.3 & 7.4: Beam Search Optimization and System Performance

A heuristic beam search optimization significantly speeds up language model lookups by pruning unnecessary characters, achieving a 150x speedup with negligible impact on Character Error Rate.

Section 8: Deployed System Architecture and Conclusion

The deployed Deep Speech system integrates low-precision computation, efficient batching, and optimized beam search to achieve low latency and high throughput for interactive applications, demonstrating the potential of end-to-end deep learning in speech recognition.

Acknowledgments, References, and Appendix A

The system's training infrastructure is scalable, utilizing optimized GPU implementations for CTC loss and matrix operations, enabling efficient training and deployment with minimal accuracy degradation.