Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Host: We are looking at the big picture of a breakthrough speech recognition system called Deep Speech 2, built by researchers at Baidu. They demonstrate that a single end-to-end deep learning approach can successfully recognize two vastly different languages, English and Mandarin Chinese. Guest: Building one system for two completely different languages sounds incredibly difficult, so how does this end-to-end approach make that possible? Host: Older systems required engineers to hand-craft separate, complex pipelines of rules and features for every single language. An end-to-end neural network replaces all those manual steps by simply taking in raw audio and directly learning to output text, making it highly adaptable to different languages, accents, and noisy environments. Guest: If the neural network is doing all that heavy lifting, wouldn't training it require massive amounts of computing power? Host: It does, which is why a key part of their success relies on high-performance computing techniques to process the data efficiently. They achieved a seven-times speedup over their previous system, shrinking the time it takes to run a single experiment from several weeks down to just a few days. Guest: That makes a lot of sense, because if you can test new ideas in days instead of weeks, you can figure out what works much faster. Host: Exactly, and that rapid iteration allowed them to refine their algorithms until the system's accuracy actually became competitive with human transcribers on standard datasets. Guest: That is amazing for research, but can a heavy deep learning system like this run quickly enough for everyday users interacting with an app? Host: Yes, thanks to a data center technique they call Batch Dispatch. By cleverly grouping incoming user audio requests and processing them together on graphics processing units, they can deliver extremely fast, low-latency responses to users at scale without it becoming wildly expensive.

Host: Let's look at how speech recognition systems are fundamentally changing. For decades, getting computers to understand speech meant piecing together complex, hand-crafted pipelines. Guest: What kind of pieces are we talking about? Host: Things like specialized feature extractors, acoustic models, and pronunciation dictionaries. The problem is that tuning all those separate parts makes it incredibly hard to adapt the system to a new language or environment. Guest: So what is the alternative to building all those separate components? Host: An end-to-end approach built entirely on deep learning, specifically a system called Deep Speech 2. Instead of many separate modules, it uses a single neural network that learns everything directly from the audio data. Guest: Does a single model actually perform as well as those highly specialized pipelines? Host: It actually approaches, and sometimes exceeds, the transcription accuracy of real human workers. By scaling up the training data and computing power, they reduced their English error rates by up to forty-three percent compared to their previous version. Guest: You mentioned earlier that traditional systems struggle with new languages, so can this end-to-end model handle them better? Host: Yes, and that is a massive breakthrough. With very little modification, this exact same system achieved high accuracy in both English and Mandarin, which are structurally completely different languages. Guest: That sounds a lot closer to how a person would learn to process speech. Host: Exactly. The ultimate goal is a single, robust engine that can handle different accents, noisy environments, and new languages from scratch, just like humans naturally do.

Host: To understand what made Deep Speech 2 such a leap forward, we need to look at its three main pillars: model architecture, massive datasets, and computational scale. Because it is an end-to-end system, upgrading these areas directly boosts its ability to predict text from audio. Guest: How massive are we talking when you say massive datasets? Host: They used nearly twelve thousand hours of English speech and over nine thousand hours of Mandarin. They also synthesized additional audio to augment the data even further during training. Guest: Processing that much audio sounds like it would take forever, even for powerful computers. Host: You are right, computing the tens of exaFLOPs required here would take three to six weeks on a single GPU. To speed up their experiments, the team built a highly optimized system that spreads the workload across eight to sixteen GPUs. Guest: Did they have to change how the training algorithm worked to use that many processors at once? Host: Yes, they used an approach called synchronous stochastic gradient descent, meaning all the GPUs update the model together in lockstep. Previous large-scale models let processors update independently, but keeping them synchronized actually converged faster and was easier to debug. Guest: Keeping them perfectly in step must require incredibly fast communication between the chips. Host: Exactly, which is why they wrote custom code to manage memory and speed up data sharing. That extreme hardware efficiency slashed their training time from weeks down to just three to five days. Guest: With that kind of speed and data, how did the final transcriptions actually compare to human listeners? Host: Amazingly well, as the system outperformed real human workers on several common benchmarks. They also managed to significantly close the gap on the absolute hardest audio samples, all while finding network architectures that were efficient enough to deploy to real users.

Host: Before we dive into the core architecture, it helps to look at the previous breakthroughs that paved the way for this system. The authors wrap up their introduction by noting their implementation hits a highly efficient 67-millisecond latency on production servers, and then they map out the existing research landscape. Guest: What does that landscape look like? Deep learning has been part of speech tech for a while, right? Host: It has, with neural networks being used for over twenty years. But the big recent shift is toward end-to-end models, where the system maps variable-length audio directly to text. Guest: Variable-length just means people talk at different speeds, so the audio length doesn't neatly match the word count? Host: Exactly. The researchers note two main ways to handle this: encoder-decoder models, and what's called a CTC-RNN model. This paper goes all-in on the CTC-RNN approach, which uses a specific loss function to handle that tricky timing alignment. Guest: Is there anything unique about how they use that CTC-RNN approach compared to past research? Host: The major difference is that past CTC models usually needed a head start. They had to be pre-trained using older, traditional systems that fed them frame-by-frame alignments. This team figured out how to train their model completely from scratch, without needing that prior alignment. Guest: That sounds like a much cleaner process. How do they pull off training from scratch? Host: It comes down to massive scale and clever data usage. They distribute the intense training workload across multiple GPUs, and they aggressively artificially expand their training data. Guest: Artificially expand it? Like adding background noise to existing recordings? Host: Yes, and even more complex tweaks, like digitally altering the speaker's vocal tract length and speech rate. They also use a trick called bootstrapping, where they take an older speech recognizer to automatically transcribe thousands of hours of unlabelled audio. Guest: Oh, so they just filter out the mistakes from that rough transcription, and suddenly they have a massive new dataset to train the new system? Host: Precisely. By combining that generated data with their artificial augmentations, they give their new model the sheer volume of examples it needs to learn from scratch.

Host: Let's look under the hood at the actual neural network structure needed to process thousands of hours of labeled speech. To handle all that data, the team had to build a much deeper model, exploring architectures with up to 11 layers. Guest: That sounds massive. With 11 layers, isn't the computational cost going to be huge? Host: It is, taking nearly eight times more computation per example than their previous system, Deep Speech 1. To make this manageable, they used techniques like Batch Normalization and a new training method they call "SortaGrad." Guest: Even with those tricks, processing every single fraction of a second of audio must be incredibly slow. Host: To speed things up, the very first layers of the network use "long strides" to essentially skip slightly ahead in the audio, reducing the computational workload by a third. Guest: What exactly is being fed into those first layers? Host: The input is a spectrogram, which measures the audio's power across different frequencies over time. The network ingests those audio slices and eventually outputs text transcriptions. Guest: So it just maps the sound straight to letters? Host: Right, the top layers output character predictions at each time step. For English, that means the alphabet, an apostrophe, a space to separate words, and a special blank symbol, while the Mandarin system outputs simplified Chinese characters. Guest: And what happens in the middle of the network, between the raw audio input and the text output? Host: That is where the recurrent layers come in to track patterns over time. The researchers often used bidirectional layers that look both forward and backward in the audio, but they found that simple one-directional layers still work excellently and are much easier to deploy in the real world.

Host: To see how the system handles deep neural networks, we need to look at its core architecture for processing English and Mandarin speech. The model uses a flexible stack of layers, starting with up to three convolutional layers followed by one to seven recurrent layers. Guest: So it combines different types of layers, but what exactly happens inside those recurrent ones? Host: They process the audio bidirectionally, moving both forward and backward in time, and then sum those activations together. The specific recurrent units can be standard ones, or more complex designs like LSTMs and Gated Recurrent Units, known as GRUs. Guest: Once the audio passes through all those recurrent layers, how does it finally predict the text? Host: It flows into one or more fully connected layers, ending with a Softmax output layer. That Softmax computes a probability distribution to guess the most likely character at each moment. Guest: But people speak at different speeds, so how does it match those frame-by-frame character guesses to the actual transcript? Host: That is handled by the CTC loss function, which automatically aligns the timing of the predictions with the correct text. The model then learns from any errors by updating its parameters using an algorithm called backpropagation through time. Guest: And since they are testing this on both English and Mandarin, do they evaluate those errors the same way? Host: They use Word Error Rate, or WER, for English, but since Mandarin is written with individual characters rather than spaced words, they rely on Character Error Rate, or CER. Ultimately, the architectural improvements they introduce are language-agnostic, meaning this exact same design works for both.

Table 1: Comparison of WER on a training and development set for various depths of RNN, with and without BatchNorm. The number of parameters is kept constant as the depth increases, thus the number of hidden units per layer decreases. All networks have 38 million parameters. The architecture "M RNN, N total" implies 1 layer of 1D convolution at the input, M consecutive bidirectional RNN layers, and the rest as fully-connected layers with N total layers in the network. 3.2 Batch Normalization for Deep RNNS To efficiently scale our model as we scale the training set, we increase the depth of the networks by adding more hidden layers, rather than making each layer larger. Previous work has examined doing so by increasing the number of consecutive bidirectional recurrent layers [24]. We explore Batch Normalization (BatchNorm) as a technique to accelerate training for such networks [63] since they often suffer from optimization issues. Recent research has shown that BatchNorm improves the speed of convergence of recurrent nets, without showing any improvement in generalization performance [37]. In contrast, we demonstrate that when applied to very deep networks of simple RNNs on large data sets, batch normalization substantially improves final generalization error while greatly accelerating training. In a typical feed-forward layer containing an affine transformation followed by a non-linearity f(·), we insert a BatchNorm transformation by applying f(B(Wh)) instead of f(Wh + b), where B(x) = γ x - E[x] (Var[x] + €)1/2 + β. The terms E and Var are the empirical mean and variance over a minibatch. The bias b of the layer is dropped since its effect is cancelled by mean removal. The learnable parameters y and β allow the layer to scale and shift each hidden unit as desired. The constant e is small and positive, and is included only for numerical stability. In our convolutional layers the mean and variance are estimated over all the temporal output units for a given convolutional filter on a minibatch. The BatchNorm transformation reduces internal covariate shift by insulating a given layer from potentially uninteresting changes in the mean and variance of the layer's input. We consider two methods of extending BatchNorm to bidirectional RNNs [37]. A natural extension is to insert a BatchNorm transformation immediately before every non-linearity. Equation 3 then becomes = f(B(Wh¯¹ + 1)). In this case the mean and variance statistics are accumulated over a single time-step of the minibatch. The sequential dependence between time-steps prevents averaging over all time-steps. We find that this technique does not lead to improvements in optimization. We also tried accumulating an average over successive time-steps, so later time-steps are normalized over all present and previous time- steps. This also proved ineffective and greatly complicated backpropagation. We find that sequence-wise normalization [37] overcomes these issues. The recurrent computation is given by = f(B(Wh¯¹) + 1). For each hidden unit, we compute the mean and variance statistics over all items in the minibatch over the length of the sequence. Figure 2 shows that deep networks converge faster with sequence- wise normalization. Table 1 shows that the performance improvement from sequence-wise normal- ization increases with the depth of the network, with a 12% performance difference for the deepest network. When comparing depth, in order to control for model size we hold constant the total

Host: When comparing different recurrent network setups, it really helps to look at how practical training techniques play out in the real world. For instance, Batch Normalization is highly effective during training, but it gets tricky when deploying a speech recognition system. Guest: Why does it become a problem when the system is actually out in the real world? Host: During training, the system processes a large "batch" of audio clips at once, which makes it easy to calculate a stable mean and variance for normalization. But in deployment, the system usually has to evaluate just a single user's voice command at a time. Guest: Right, so you wouldn't have enough data in one short clip to get reliable normalization statistics. Host: Exactly, and trying to normalize over just one single sequence actually degrades performance. To solve this, they store a running average of the mean and variance collected during the training phase, and lock those in to use during deployment. Guest: That’s a clever workaround. I also imagine all those real-world audio clips are completely different lengths, which must be hard to train on. Host: It is, and chopping all the audio sequences down to the same size actually hurts the system's ability to learn long-term dependencies. Instead, they introduce a strategy called SortaGrad, which relies on a curriculum learning approach. Guest: A curriculum? Does that mean the model is taught the easy examples first, and then works its way up? Host: Spot on. Because the specific cost function they use naturally produces higher error penalties for longer audio, longer sequences are inherently harder for the system to process. Guest: So they literally just sort the training data so the shorter, easier audio clips are fed to the model first? Host: That is exactly what they do. By using utterance length as a measure of difficulty, SortaGrad guides the model through a natural learning curve, which ultimately leads to better overall performance.

Host: Let's get into the architectural choices and training tricks that keep these massive speech models stable. One major challenge early in training is numerical instability, which they solved using a curriculum strategy called SortaGrad. Guest: How does a training curriculum like SortaGrad actually keep the model stable? Host: In the very first epoch, it feeds the training data to the model starting with the shortest audio clips and working up to the longest. They do this because long utterances tend to produce massive gradients that can cause the network's internal state to explode early on. Guest: Oh, so starting with shorter audio clips acts like a warm-up phase. Does the type of recurrent layer they use also affect this stability? Host: It does, which is why they compared simple Recurrent Neural Networks against Gated Recurrent Units, or GRUs. They found GRUs are just as accurate as the popular LSTM models, but they train faster and are less likely to diverge. Guest: Did they just plug in the standard GRU, or did they have to modify it for this speech recognition task? Host: They made a clever tweak by rearranging the math so all operations on the previous hidden state can be computed in a single matrix multiplication. They also swapped the standard activation function for a clipped-ReLU to keep things uniform across the entire network. Guest: With those optimizations, do the GRUs end up outperforming the simpler RNNs? Host: It depends on how you measure it. If you strictly limit the number of parameters, GRUs win, but if you look at a fixed computational budget at a massive scale, the simple RNNs actually perform slightly better.

Host: When processing raw speech, the way a neural network takes its very first look at the audio can make a massive difference. Here, the researchers experimented with different arrangements of convolutional layers right at the input. Guest: Why use convolutional layers first, instead of jumping straight into the recurrent layers? Host: Convolutions are great at efficiently scanning the audio sequence across time. By using a technique called striding, where the filter skips forward a few steps at a time, they compress the sequence so the later recurrent layers have much less data to compute. Guest: That makes sense for compressing time. But the text also mentions frequency convolutions, or 2D layers, so what do those do? Host: While 1D convolutions only scan across time, 2D convolutions scan across both time and frequency. Scanning across the frequency domain helps the model better handle variations in pitch from different speakers. Guest: And why do these frequency convolutions need to happen right at the beginning of the network? Host: Because the later fully-connected layers completely flatten and mix up the structural layout of the data. The frequency convolution has to happen early, while the audio's frequency bands are still neatly stacked. Guest: How much of a difference did adding these 2D layers actually make in their tests? Host: They found a huge benefit when testing on heavily noisy audio. Switching from a single 1D layer to three 2D layers improved the word error rate by almost 24 percent on their noisy dataset. Guest: That is a massive jump for noisy environments. Did it help with clean speech too? Host: It provided just a small benefit for clean data, so its real strength is handling challenging, real-world noise. Plus, since convolutions are highly efficient, they achieved this while keeping the network's total size right around 35 million parameters.

Host: We are turning our attention to how these systems handle live, streaming audio without making the user wait. Traditionally, highly accurate speech models are bidirectional, meaning they process an entire audio clip forward and backward before generating text, which causes way too much delay for live dictation. Guest: Right, if I'm speaking to my phone, I want the text to appear immediately, not after I finish my whole sentence. How do they speed that up? Host: They switch to a unidirectional, or forward-only, model that processes the audio as it arrives. To keep the accuracy as high as the older bidirectional models, they add a special layer called row convolution. Guest: How does row convolution help maintain that high accuracy without looking at the whole recording? Host: It relies on the intuition that you don't need the entire rest of the sentence to figure out the current sound, just a tiny peek into the future. It groups the current audio feature with a specific, small number of upcoming time steps. Guest: Oh, so it delays the output just a fraction of a second to gather those few upcoming frames, instead of waiting for me to stop talking entirely? Host: Exactly. The system takes that very small window of future context and applies a mathematical filter across it, giving the model enough information to make an accurate prediction for the current moment. Guest: That sounds like a perfect compromise between getting the necessary context and avoiding massive latency. Host: Spot on. It allows the system to stream transcriptions back to the user in real-time, matching the performance of those heavier bidirectional models while staying fast and efficient.

Host: Let us dive into a unique architectural feature the authors developed called row convolution. The paper includes a diagram showing how this layer operates with a future context size of two, meaning it looks just a couple of steps ahead in the sequence. Guest: How exactly does it process those future steps? There is an equation mentioned, but what is it actually doing in plain terms? Host: Simply put, the equation applies a sliding window of weights to the data, but it does this strictly row by row. It looks at the sequence of hidden states for one specific feature at a time independently, which is exactly why they named it row convolution. Guest: Where does this specific layer sit in the overall model? Host: It is placed right on top, above all the standard recurrent layers. The authors hypothesize that the recurrent layers do the heavy lifting to learn robust features, and then this row convolution layer efficiently gathers that information together for the final classifier. Guest: What is the main benefit of keeping this operation isolated at the very top? Host: It provides two major advantages, the first being that all the lower layers can process streaming audio continuously with very little delay. Only that final layer at the top needs to wait for future context. Guest: That is great for live streaming. Does it help with accuracy as well? Host: It does, which is the second major advantage. This design actually improved the Character Error Rate for Mandarin, beating out even the best bidirectional models. The authors note they will break down the exact deployment results for that Mandarin system later in section seven.

Host: We are now going to look at how the system uses context to figure out exactly what words were spoken. The main neural network is trained on millions of audio clips, which allows it to learn a surprisingly powerful internal language model all on its own. Guest: By internal language model, do you mean it naturally picks up on how words fit together? Host: Exactly, getting really good at spelling and figuring out homophones without any outside help. For example, if it hears about a transaction, it knows to use the word sell, S E L L, rather than cell, C E L L. Guest: If it is already that smart, why would the developers need to add any external language tools? Host: Because the labeled audio data is actually pretty small compared to the massive amounts of pure text available on the internet. To get the word error rate even lower, they supplement the system with an external language model trained on huge text databases. Guest: How huge are we talking here? Host: For English, they use a model that tracks sequences of five words at a time from a massive internet archive called the Common Crawl, resulting in about eight hundred fifty million unique combinations. For Mandarin, they trained a character-level model that yielded about two billion combinations. Guest: So when I speak into the system, how does it actually combine what the audio network hears with that massive text database? Host: It uses a scoring equation to find the best possible transcription. It adds together the probability scores from the audio network and the external text model, using a specific tuning parameter to control how much influence that text model has. Guest: That makes sense, but is there anything to keep the system from just guessing really short sentences to get a safe, high score? Host: Yes, the equation includes one final piece called a word insertion term. This acts like a bonus that encourages the system to include more words in the transcription, ensuring the final sentence is as accurate and complete as possible.

Host: We are turning our attention to how external language models impact transcription error rates across different languages. Specifically, we are looking at a comparison between English and Mandarin speech systems built on simple recurrent neural networks. Guest: What exactly are we measuring to compare the two languages? Host: For English, the metric is Word Error Rate, and for Mandarin, it is Character Error Rate. The baseline tests confirm that plugging in an external language model significantly improves accuracy for both. Guest: Does the size of the underlying neural network change how much that external language model helps? Host: Yes, and the trend is quite interesting, because as the core model gets deeper, the relative boost from the external language model actually shrinks. For example, in English, the improvement from the language model dropped from 48 percent down to 36 percent when the network grew from five layers to nine layers. Guest: Why would a larger, more complex model need less help from the outside? Host: Researchers hypothesize that deeper networks, especially those with more recurrent layers, start developing a stronger implicit language model of their own. They basically learn the linguistic rules and context internally. Guest: Did the external language model help English and Mandarin equally? Host: Actually, the performance gain was much more pronounced in English. This is because a single Chinese character encapsulates a much larger unit of meaning than a single English letter. Guest: So because the English system predicts letter by letter, it makes more basic spelling errors without that external guidance? Host: Exactly right. If the English model were designed to output whole syllables or words directly, it would naturally make fewer spelling mistakes, which would make the impact of an external language model much less dramatic.

Host: Applying these speech recognition techniques to a completely different language structure brings up some fascinating engineering challenges. By using an end-to-end approach, the system can output Mandarin characters directly, entirely skipping the need for a separate pronunciation model. Guest: Why is skipping the pronunciation model such an advantage? Host: Pronunciation models get incredibly complex when you adapt them to new languages. By going straight to character output, the system bypasses the need to explicitly program rules for things like Mandarin's distinct tonal variations. Guest: But there are thousands of Chinese characters, so how massive does the network's output layer have to be? Host: It is designed to produce about 6,000 different characters. That set also strategically includes the Roman alphabet to handle transcripts where a speaker mixes English words into their Mandarin. Guest: What happens if the system encounters a rare character that isn't in that list of 6,000? Host: It simply registers as an out-of-vocabulary error, but it is a very minor concern here. In testing, only 0.74 percent of the characters fell outside of that specific vocabulary. Guest: Since Mandarin doesn't use spaces between words, did they have to change how the system predicts the flow of sentences? Host: Yes, they replaced their word-level language model with a character-level one. They also noticed that the decoding process, which finds the most likely transcription, actually stabilizes much faster. Guest: Does that faster stabilization mean the system uses less computational effort to decode the audio? Host: Exactly, they were able to shrink their beam search size down to just 200 with barely any penalty to the character error rate. It proves these architectural tweaks work just as effectively for Mandarin as they do for English.

Host: Let's explore the immense computational engine required to build these advanced AI models. The networks we are looking at contain tens of millions of parameters, meaning the training process demands tens of exaFLOPs—or quintillions of math operations—just to finish learning. Guest: That sounds like an incredible amount of computing, which must make it really slow to test out new ideas. Host: It definitely can be, which is why the developers built a custom, highly optimized training system from scratch. They needed a way to rapidly evaluate different hypotheses about their data and models without waiting weeks for a result. Guest: What kind of software did they build to actually get that kind of speed? Host: The system relies on two main components: a deep learning library written in C++, and a custom linear algebra library using both C++ and CUDA. CUDA is essentially the programming platform that allows their software to pull maximum performance directly from the graphics cards. Guest: Speaking of graphics cards, what kind of hardware are they running this custom software on? Host: They use dense compute nodes that are each packed with eight Titan X GPUs. By pairing that specific hardware with their optimized software, a single node can sustain a massive throughput of 24 teraFLOPs per second while training a single model. Guest: Is 24 teraFLOPs per second considered efficient for a machine with eight heavy-duty GPUs? Host: Absolutely, because it represents 45 percent of the theoretical peak maximum speed of the hardware, which is notoriously hard to sustain in a real-world workload. And they didn't just stop at one machine; this entire system is specifically designed to scale across multiple nodes.

Host: Let's explore how the system coordinates training across multiple processors at once. The standard approach here is data-parallelism, where a typical setup takes a minibatch of 512 examples and splits it across eight GPUs using synchronous SGD. Guest: So if we have eight GPUs handling that batch, does the training pipeline just assign one process to each of them? Host: Exactly, one process per GPU. They each process their slice of the data, but during the learning phase—specifically backpropagation—they need to share their updates using an all-reduce operation. Guest: How does that all-reduce operation actually work? Host: It takes the gradient matrices calculated by all the individual processes, sums them together, and then sends that combined, identical matrix back to every single GPU. Guest: That means they all have to wait for each other to stay perfectly in sync. Host: They do, and that synchronized, deterministic behavior is actually the main appeal. Because it runs the exact same way every time, the authors know that any non-deterministic behavior is almost certainly a serious bug. Guest: I imagine that makes debugging much easier compared to an asynchronous setup where GPUs aren't waiting on each other. Host: Immensely easier. Asynchronous methods, like those using parameter servers, lack that reproducibility, which makes tracking down errors really challenging. Sticking to synchronous SGD keeps the design straightforward, completely predictable, and scales very well as you add more nodes.

Host: Let's look at how adding more hardware actually affects the time it takes to train our models. Specifically, the authors compared a five-layer and a nine-layer recurrent network to see what happens when they double the number of GPUs. Guest: Did doubling the hardware actually cut the training time in half, or is there a point of diminishing returns? Host: It actually did cut the time in half, which demonstrates what we call near-linear weak scaling. To achieve this, they kept the workload per GPU constant at a minibatch size of 64, meaning the total batch size doubled every time they added more hardware. Guest: That sounds incredibly efficient, but do they just keep adding GPUs indefinitely to speed things up? Host: Not exactly, because while the system handles large batches well, they found the best practical results using 8 or 16 GPUs. The real secret to making this scaling work at all is a custom operation they built called all-reduce. Guest: What does this all-reduce operation do, and why did they need to customize it? Host: It is the step where all the GPUs synchronize and share their learning updates with each other, which usually creates a massive communication bottleneck. To fix this, they built a custom ring algorithm that lets GPUs talk directly to each other without copying data back and forth through the CPU's memory. Guest: So the GPUs are essentially bypassing the main computer's processor to share their data faster? Host: Exactly, using a technology called GPUDirect. When two GPUs share the same internal hardware connection, this setup sends data concurrently in a ring directly between devices, which is fundamental to maintaining the speed needed to scale up training.

Table 7 compares the performance of the custom all-reduce implementation against the one provided by OpenMPI version 1.8.5. The table reports the time spent in the all-reduce operation for a full training run that spans one epoch on the English dataset, using a 5-layer, 3-recurrent-layer architecture with 2560 hidden units for all layers. In this comparison, a minibatch size of 64 per GPU is used, with the algorithmic minibatch size expanding as more GPUs are added. The results show that the custom implementation is considerably faster than OpenMPI's when communication is confined within a single node (8 GPUs or fewer). As the number of GPUs increases and inter-node communication becomes more prevalent, the performance gap narrows, although the custom implementation remains 2-4 times faster. The authors' training runs consistently utilize either 8 or 16 GPUs. In this regime, the custom all-reduce implementation results in 2.5 times faster training for the full training run compared to using OpenMPI directly. Therefore, optimizing the all-reduce operation has yielded significant productivity benefits for experiments and has enabled the scalability of their simple synchronous SGD approach.

Table 8 compares the time spent in seconds for computing the CTC loss function and its gradient over one epoch for two different implementations. The GPU implementation offers substantial time savings, reducing the epoch time by 95 minutes for English and 25 minutes for Mandarin. This translates to an overall training time reduction of 10-20%, which is a significant productivity gain for the experiments.

Calculating the CTC loss function is more complex than performing forward and backward propagation on RNN architectures. Initially, activations were transferred from the GPUs to the CPU, where the loss function was computed using an OpenMP parallelized CTC implementation. However, this approach significantly limited scalability for two primary reasons. Firstly, it became computationally more demanding as the efficiency and scalability of the RNN itself improved. Secondly, transferring large activation matrices between the CPU and GPU consumed valuable interconnect bandwidth for CTC, hindering scalability through data parallelism across more processors. To address this, a GPU implementation of the CTC loss function was developed. This parallel implementation involves a slight refactoring to simplify dependencies within the CTC calculation and utilizes optimized parallel sort implementations from ModernGPU. Further details on this parallelization are provided in the Appendix. Table 8 compares the performance of two CTC implementations, highlighting that the GPU implementation saves considerable time per epoch for both English and Mandarin, leading to a notable reduction in overall training time and enhancing experimental productivity.

The system frequently employs dynamic memory allocations for both GPU and CPU memory, primarily to store activation data for variable-length utterances and intermediate results. Individual allocations can be substantial, exceeding 1 GB for the longest utterances. For these very large allocations, both CUDA's memory allocator and std::malloc introduced significant overhead, resulting in slowdowns of over 2x in some cases. This overhead arises because both cudaMalloc and std::malloc forward large allocations to the operating system or GPU driver to update system page tables. While this is an effective optimization for systems running multiple applications that share memory resources, it represents pure overhead for a system where nodes are dedicated to running a single model. To circumvent this limitation, a custom memory allocator for both CPU and GPU allocations was developed. This implementation adopts the approach of a last-level shared allocator found in jemalloc, where all allocations are carved out of contiguous memory blocks using the buddy algorithm. To prevent fragmentation, all GPU memory is preallocated at the start of training, and individual allocations are subdivided from this block. Similarly, the CPU memory block size forwarded to mmap is set to a substantially larger value of 12 GB compared to std::malloc.

Table 9 provides a summary of the datasets used for training the DS2 system in English. It includes the Wall Street Journal (WSJ), Switchboard, and Fisher corpora, all published by the Linguistic Data Consortium. The LibriSpeech dataset is available online, and the remaining datasets are internal Baidu corpora. The total hours of labeled speech data for English amount to 11,940 hours.

The majority of memory required for training deep recurrent networks is consumed by storing activations across layers for backpropagation, rather than by the network's parameters. For instance, the weights for a 70 million parameter network with 9 layers require approximately 280 MB of memory, whereas storing the activations for a batch of 64, seven-second utterances necessitates 1.5 GB of memory. TitanX GPUs are equipped with 12 GB of GDDR5 RAM, and very deep networks can sometimes exceed this GPU memory capacity when processing long utterances. This can occur unpredictably, particularly when the distribution of utterance lengths includes outliers, making it desirable to avoid catastrophic failures. When a requested memory allocation surpasses the available GPU memory, the system resorts to allocating page-locked GPU-memory-mapped CPU memory using cudaMallocHost. This memory can be accessed directly by the GPU by forwarding individual memory transactions over PCIe at a reduced bandwidth, enabling the model to continue processing even when encountering an outlier. The combination of a fast memory allocation mechanism with a fallback strategy for handling occasional GPU memory overflows in exceptional cases simplifies the system, enhances its robustness, and improves its efficiency.

Large-scale deep learning systems necessitate a substantial amount of labeled training data. The authors have amassed an extensive training dataset for both English and Mandarin speech models, complemented by publicly available datasets. For English, the dataset comprises 11,940 hours of labeled speech data, encompassing 8 million utterances, as summarized in Table 9. The Mandarin system utilizes 9,400 hours of labeled audio, containing 11 million utterances. The Mandarin speech data consists of internal Baidu corpora, representing a mixture of read and spontaneous speech, in both standard Mandarin and accented Mandarin.

Some of the internal English (3,600 hours) and Mandarin (1,400 hours) datasets were derived from raw data captured as long audio clips with noisy transcriptions. The duration of these clips ranged from several minutes to over an hour, making it impractical to unroll them in time within the RNN during training. To address this challenge, an alignment, segmentation, and filtering pipeline was developed to generate a training set with shorter utterances and fewer erroneous transcriptions. The initial step in this pipeline involves using an existing bidirectional RNN model trained with CTC to align the transcription to the audio frames. For each audio-transcript pair $(x, y)$, the alignment that maximizes $P_{CTC}(l|x; \theta)$ is identified, where $l$ represents the alignment. This process is essentially a Viterbi alignment performed using a CTC-trained RNN model. Since Equation 9 integrates over the alignment, the CTC loss function is not explicitly required to produce an accurate alignment. In principle, CTC could choose to emit all characters of the transcription after a certain fixed delay, which can occur with unidirectional RNNs. However, the authors found that CTC produces an accurate alignment when trained with a bidirectional RNN.

Following the alignment, a segmentation step is performed. This involves splicing the audio and its corresponding aligned transcription whenever a long series of consecutive blank labels is encountered, indicating a stretch of silence. By adjusting the number of consecutive blanks, the duration of the generated utterances can be controlled. For the English speech data, a space token is also required within the stretch of blanks to ensure segmentation occurs only at word boundaries. The segmentation is tuned to produce utterances that are, on average, 7 seconds long. The final step in the pipeline is to remove erroneous examples that result from alignment failures. Ground truth transcriptions for several thousand examples were crowdsourced. The word-level edit distance between the ground truth and the aligned transcription is used to classify examples as good or bad. The threshold for the word-level edit distance is set such that the resulting WER on the good portion of the development set is less than 5%. Subsequently, a linear classifier is trained to accurately predict bad examples based on input features generated from the speech recognizer. The features found to be useful include the raw CTC cost, the CTC cost normalized by sequence length, the CTC cost normalized by transcript length, the ratio of sequence length to transcript length, and the number of words and characters in the transcription. For the English dataset, this filtering pipeline reduces the WER from 17% to 5% while retaining over 50% of the examples.

The training data is augmented by adding noise to increase its effective size and improve robustness to noisy speech. While the training data inherently contains some noise, augmentation allows for increased quantity and variety. Excessive noise augmentation can hinder optimization and lead to poorer results, whereas insufficient augmentation results in a less robust system. A balanced approach involves adding noise to 40% of randomly selected utterances. The noise source comprises several thousand hours of randomly selected audio clips, combined to produce hundreds of hours of noise.

The English and Mandarin corpora used in this study are significantly larger than those commonly reported in speech recognition literature. Table 10 illustrates the impact of increasing the amount of labeled training data on WER. This is achieved by randomly sampling the full dataset before training. For each dataset, the model is trained for up to 20 epochs, typically with early stopping based on performance on a held-out development set. The WER exhibits a power-law decrease for both regular and noisy development sets. Specifically, the WER decreases by approximately 40% relative for every tenfold increase in training dataset size. A consistent gap of about 60% relative in WER is observed between the regular and noisy datasets, suggesting that additional data benefits both scenarios equally. This implies that speech system performance continues to improve with more labeled training data. The authors hypothesize that increasing the number of speech contexts captured in the dataset is as important as increasing the raw number of hours. A speech context can encompass any characteristic that makes speech unique, including variations in speakers, background noise, environment, and microphone hardware. Although labels for these specific contexts are not available, the authors suspect that measuring WER as a function of the number of speakers in the dataset would yield larger relative gains than simple random sampling.

To provide a comprehensive assessment of the speech system's real-world applicability, evaluations are conducted on a diverse range of test sets. These include several publicly available benchmarks and internally collected test sets, collectively representing a wide spectrum of challenging speech environments. These environments encompass low signal-to-noise ratios (noisy and far-field), accented speech, read speech, spontaneous speech, and conversational speech.

Table 10 presents a comparison of English WER for regular and noisy development sets as the training dataset size increases. The architecture used is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers, featuring 68 million parameters. The results show a consistent decrease in WER for both regular and noisy development sets as more training data is utilized.

All models are trained for 20 epochs on either the full English dataset (described in Table 9) or the full Mandarin dataset (described in Section 5). Stochastic gradient descent with Nesterov momentum is employed, along with a minibatch size of 512 utterances. If the norm of the gradient exceeds a threshold of 400, it is rescaled to 400. The model that achieves the best performance on a held-out development set during training is selected for evaluation. The learning rate is chosen from the range $[1 \times 10^{-4}, 6 \times 10^{-4}]$ to ensure the fastest convergence and is annealed by a constant factor of 1.2 after each epoch. A momentum of 0.99 is used for all models. The language models employed are those described in Section 3.8. The decoding parameters from Equation 12 are tuned on a held-out development set. A beam size of 500 is used for the English decoder, and a beam size of 200 for the Mandarin decoder.

The best DS2 model for English comprises 11 layers: 3 layers of 2D convolution, 7 bidirectional recurrent layers, and a fully-connected output layer, along with Batch Normalization. The first layer outputs to bigrams with a temporal stride of 3. In contrast, the DS1 model has 5 layers with a single bidirectional recurrent layer and outputs to unigrams with a temporal stride of 2 in the first layer. The reported results on various test sets are for both the DS2 and DS1 models. Neither model is tuned or adapted to specific speech conditions within the test sets. The language model decoding parameters are set once on a held-out development set. To contextualize the system's performance, it is benchmarked against human workers, as speech recognition is fundamentally an audio perception and language understanding task where humans excel. Human-level performance is assessed by paying workers from Amazon Mechanical Turk to hand-transcribe all test sets. Two workers transcribe each audio clip, typically around 5 seconds long, and the better of the two transcriptions is used for the final WER calculation. These workers, primarily located in the United States, spend an average of 27 seconds per transcription. The hand-transcribed results are compared to the existing ground truth to calculate WER. While the ground truth transcriptions may contain minor labeling errors (rarely exceeding 1%), the disagreement between ground truth transcripts and human-level transcripts serves as a reliable heuristic for human-level performance.

The English speech training set is considerably larger than commonly used speech datasets and is further augmented with noise synthesis. To achieve optimal generalization error, an increase in model size is expected to be necessary to fully exploit the patterns within the data. Section 3.2 explored the effect of model depth while keeping the number of parameters constant. In contrast, this section investigates the impact of varying model size on speech system performance. Only the size of each layer is varied, while architectural parameters remain constant. Models are evaluated on the same Regular and Noisy development sets used in Section 3.5. The models in Table 11 differ from those in Table 3 in that the stride is increased to 3 and the output is to bigrams. Due to the increase in model size, reaching up to 100 million parameters, a larger stride is necessary to manage computational and memory constraints. However, in this regime, the performance advantage of GRU networks appears to diminish.

Table 11 compares the effect of model size on the WER of the English speech system on both regular and noisy development sets. The number of hidden units in all layers except the convolutional layers is varied. The GRU model features 3 layers of bidirectional GRUs and 1 layer of 2D-invariant convolution. The RNN model consists of 7 layers of bidirectional simple recurrence and 3 layers of 2D-invariant convolution. Both models output bigrams with a temporal stride of 3. All models contain approximately 35 million parameters and are trained using BatchNorm and SortaGrad. The table shows that performance consistently improves up to 100 million parameters. All subsequent English DS2 results are reported using this 100 million parameter RNN model, as it achieves the lowest generalization errors.

Table 12 compares the DS1 and DS2 WER on an internal test set of 3,300 examples. This test set includes a diverse range of speech, encompassing accents, low signal-to-noise speech, spontaneous speech, and conversational speech. The results indicate that the 100 million parameter RNN model (DS2) achieves a 43.4% relative improvement over the 5-layer model with 1 recurrent layer (DS1) on this internal Baidu dataset.

The simple RNN model performs better than the GRU network and is faster to train, despite having two additional convolution layers, for the 100 million parameter networks. Table 11 demonstrates that system performance consistently improves up to 100 million parameters. All further English DS2 results are reported using this 100 million parameter RNN model, as it achieves the lowest generalization errors. Table 12 shows that the 100 million parameter RNN model (DS2) provides a 43.4% relative improvement over the 5-layer model with 1 recurrent layer (DS1) on an internal Baidu dataset of 3,300 utterances, which includes a wide variety of speech, such as challenging accents, low signal-to-noise ratios from far-field or background noise, and spontaneous and conversational speech.

Read speech, characterized by a high signal-to-noise ratio, is generally considered the easiest task for large vocabulary continuous speech recognition. The system is benchmarked on two test sets from the Wall Street Journal (WSJ) corpus, which consists of read news articles. These datasets are available in the LDC catalog as LDC94S13B and LDC93S6B. Additionally, the recently developed LibriSpeech corpus, compiled from audiobooks from the LibriVox project, is utilized. Table 13 compares the WER for two speech systems (DS1 and DS2) and human performance on read speech. The DS2 system outperforms humans in 3 out of the 4 test sets and is competitive on the fourth. Based on these results, the authors suggest that there is limited room for a generic speech system to further improve on clean read speech without domain-specific adaptation.

Table 13 presents a comparison of WER for two speech systems (DS1 and DS2) and human performance on read speech across different test sets. The DS2 system consistently achieves lower WER than DS1 across all test sets. Notably, DS2 outperforms human performance on the WSJ eval'92 and WSJ eval'93 test sets, and is competitive on LibriSpeech test-clean. On LibriSpeech test-other, DS2's WER is slightly higher than human performance, but still significantly lower than DS1.

Table 14 compares the WER of the DS1 system to the DS2 system on accented speech. The test sets include VoxForge American-Canadian, VoxForge Commonwealth, VoxForge European, and VoxForge Indian. The DS2 system consistently shows lower WER than DS1 across all accented speech test sets. While DS2 significantly improves performance compared to DS1, human performance remains notably better for most accents, except for the Indian accent where DS2 is competitive. This suggests that while the system benefits from more accented training data and an architecture capable of handling it, further improvements are needed to match human-level performance on diverse accents.

Table 15 compares the DS1 and DS2 systems on noisy speech. The test sets include CHiME eval clean, CHiME eval real, and CHIME eval sim. The "CHiME eval clean" serves as a noise-free baseline. The "CHiME eval real" dataset is collected in actual noisy environments, while the "CHiME eval sim" dataset has similar noise synthetically added to clean speech. The results indicate that DS2 substantially improves upon DS1 in noisy conditions. However, DS2's performance is still worse than human level performance on noisy data. The relative gap between DS2 and human performance is larger when the data originates from a real noisy environment compared to synthetically adding noise to clean speech. This highlights the challenge of robustly handling real-world noisy environments.

The source for accented speech is the publicly available VoxForge dataset, which contains clean speech read by speakers with diverse accents. These accents are grouped into four categories: American-Canadian, Indian, Commonwealth (British, Irish, South African, Australian, and New Zealand accents), and European (speakers with accents from European countries where English is not their first language). A test set is constructed from the VoxForge data, comprising 1024 examples from each accent group, totaling 4096 examples. Performance on these test sets serves as an indicator of the breadth and quality of the training data. Table 14 demonstrates that performance improves across all accents when more accented training data is included and an architecture capable of effectively training on such data is employed. However, human-level performance remains notably superior to that of DS2 for all accents except the Indian accent.

The system's performance on noisy speech is evaluated using publicly available test sets from the recently concluded third CHiME challenge. This dataset contains 1320 utterances from the WSJ test set, recorded in various noisy environments such as a bus, a cafe, a street, and a pedestrian area. The CHiME set also includes 1320 utterances with simulated noise from the same environments, alongside a control set containing the same utterances delivered by the same speakers in a noise-free environment. Differences in performance between the control set and the noisy sets provide a measure of the network's ability to handle diverse real and synthetic noise conditions. The CHiME audio consists of 6 channels, and utilizing all of them can lead to substantial performance improvements. For the reported results, a single channel is used, as multi-channel audio is not prevalent on most devices. Table 15 shows that DS2 significantly outperforms DS1, but DS2's performance on noisy data is still below human level. The relative gap between DS2 and human performance is larger for data from real noisy environments compared to synthetically added noise to clean speech.

Table 16 presents a comparison of various architectures trained on Mandarin Chinese speech, evaluated on a development set of 2000 utterances and a test set of 1882 noisy speech examples. This development set was also used for tuning decoding parameters. The results indicate that the deepest model with 2D-invariant convolution and BatchNorm outperforms the shallow RNN by 48% relative. This finding is consistent with the trend observed in the English system, where multiple layers of bidirectional recurrence substantially improve performance.

The best Mandarin Chinese speech system developed by the authors transcribes short voice-query utterances more accurately than a typical Mandarin Chinese speaker. To benchmark against humans, a test involving 100 randomly selected utterances was conducted, with a group of 5 humans labeling all of them. The group of humans achieved an error rate of 4.0%, compared to the speech system's performance of 3.7%. Additionally, a single human transcriber was compared to the speech system on 250 randomly selected utterances. In this comparison, the speech system performed significantly better, achieving 5.7% error rate compared to the human's 9.7%.

Real-world applications typically require speech systems to transcribe in real time or with relatively low latency. The system described in Section 6.1 is not optimally designed for this task due to several factors. Firstly, the RNN's multiple bidirectional layers necessitate the entire utterance to be presented for transcribing the initial part. Secondly, the use of a wide beam during language model decoding can be computationally expensive, particularly in Mandarin, where the number of possible next characters is very large (around 6000). Thirdly, as detailed in Section 3, power normalization is performed across the entire utterance, requiring the complete utterance to be available beforehand. To address the power normalization issue, statistics from the training set are used to perform adaptive normalization of speech inputs during online transcription. The other challenges are tackled by modifying the network and decoding procedure to produce a model with comparable performance but significantly lower latency. The focus is on the Mandarin system due to its inherent complexities, such as the large character set, but the same techniques can be applied to English. In this section, latency refers to the computational delay from the end of an utterance until the transcription is produced. This latency excludes data transmission over the internet and does not measure the time from the beginning of an utterance until the first transcription. The focus on latency from the end of utterance to transcription is crucial for applications that rely on speech recognition.

To deploy relatively large deep neural networks with low latency, significant attention has been paid to deployment efficiency. Most internet applications process requests individually as they arrive in the data center, which simplifies implementation by allowing each request to be managed by a single thread. However, individual request processing is computationally inefficient for two main reasons. Firstly, it requires loading all network weights for each request, reducing the arithmetic intensity of the workload and making computation memory bandwidth bound, as on-chip caches are difficult to utilize effectively when requests are presented individually. Secondly, the parallelism that can be exploited to classify a single request is limited, hindering the exploitation of SIMD or multi-core parallelism. RNNs pose particular deployment challenges because evaluating them involves sampling by...

Figure 5 illustrates the probability of a request being processed in a batch of a given size for a production system running on a single NVIDIA Quadro K1200 GPU, with 10-30 concurrent user requests. The graph shows that batching is most effective when the server is heavily loaded, as the distribution shifts to favor larger batches. Even with a light load of 10 concurrent user requests, more than half of the work is processed in batches of at least two samples. The system employs an eager batching scheme, processing each batch as soon as the previous one is completed, which has proven to be the most effective in reducing end-user latency, despite being less computationally efficient as it doesn't aim to maximize batch size. This approach balances the trade-off between increased batch size for improved efficiency and increased latency due to longer waiting times for user results. The sample relies on sequential matrix vector multiplications, which are bandwidth-bound and difficult to parallelize. To address these issues, a batching scheduler called Batch Dispatch was developed to assemble streams of data from user requests into batches before performing forward propagation.

Figure 6 presents the median and 98th percentile latencies as a function of server load. With 10 concurrent streams, the system achieves a median latency of 44 ms and a 98th percentile latency of 70 ms. As the server load increases, the batching scheduler optimizes for more efficient batches, thereby maintaining low latency. This demonstrates that Batch Dispatch enables the deployment of large models with high throughput and low latency. The system achieves a median latency of 44 ms and a 98th percentile latency of 70 ms when loaded with 10 concurrent streams. As the load increases on the server, the batching scheduler shifts work to more efficient batches, which keeps latency low. This shows that Batch Dispatch makes it possible to deploy these large models at high throughput and low latency.

Figure 7 compares the computational throughput of kernels that compute Ax = b, where A is a matrix with dimensions 2560 × 2560 and x is a matrix with dimensions 2560 × Batch size, for batch sizes ranging from 1 to 10. All matrices are in half-precision format. The graph shows that the custom deployment kernel sustains a higher computational throughput than those from Nervana Systems on the K1200 GPU across the entire range of batch sizes used in deployment. Both the custom kernels and the Nervana kernels are significantly faster than NVIDIA CUBLAS version 7.0. Section 7.2 discusses the deployment-optimized matrix multiply kernels. The paper found that using half-precision (16-bit) floating-point arithmetic does not significantly impact recognition accuracy. This is particularly beneficial for deployment as it reduces memory space and bandwidth requirements, which are crucial for RNN evaluation, as it is dominated by the cost of caching and streaming weight matrices. Standard BLAS libraries were found to be inefficient for the smaller batch sizes used in deployment. To address this, a custom half-precision matrix-matrix multiply kernel was developed. This kernel is optimized for bandwidth by storing the A matrix transposed and using wide vector loads, avoiding transposition after loading. Each warp computes four rows of output for all N output columns. For N ≤ 4, the B matrix fits entirely within the L1 cache, achieving 90 percent of peak bandwidth. While efficiency decreases for larger N as the B matrix no longer fits in the L1 cache, the kernel still provides improved performance over existing libraries up to N = 10.

Section 7.3 delves into the optimization of beam search, a process that involves repeated lookups in the n-gram language model, often resulting in uncached memory reads. A direct implementation of beam search dispatches one lookup per character for each beam at each time-step. For Mandarin, this translates to over 1 million lookups per 40ms stride of speech data, which is too slow for deployment. To mitigate this, a heuristic was developed to prune the beam search by considering only the fewest characters whose cumulative probability is at least p (set to 0.99 in practice), and limiting the search to a maximum of 40 characters. This optimization speeds up the Mandarin language model lookup time by a factor of 150x, with a negligible effect on the Character Error Rate (CER), a relative degradation of 0.1-0.3%. Section 7.4 presents the results, indicating that the system can be deployed at low latency and high throughput without significant accuracy loss. On a held-out set of 2000 utterances, the research system achieved a 5.81% CER, while the deployed system achieved 6.10%, representing only a 5% relative degradation.

The deployed system utilizes a neural network architecture with low deployment latency, reduced precision to 16-bit, a batching scheduler for efficient RNN evaluation, and a simple heuristic to reduce beam search cost. The model comprises five forward-only recurrent layers with 2560 hidden units, a row convolution layer with T=19, and a fully-connected layer with 2560 hidden units. These techniques enable the deployment of Deep Speech at a low cost for interactive applications. Section 8, the conclusion, highlights the potential of end-to-end deep learning to continuously improve speech recognition systems with increased data and computation. The results demonstrate that Deep Speech has significantly narrowed the performance gap with human transcribers by leveraging more data and larger models. The approach's generic nature allows for rapid application to new languages, requiring minimal expert language knowledge. The paper also shows that the system can be efficiently deployed on a GPU server by batching user requests, paving the way for end-to-end Deep Learning technologies. The authors explored various network architectures, including enhancements to numerical optimization, evaluation of RNNs with larger strides, and the use of both bidirectional and unidirectional models. This exploration was facilitated by an optimized High-Performance Computing training system, enabling the training of new, full-scale models in just a few days. The paper concludes that end-to-end Deep Learning methods are valuable for speech recognition across various settings, and the gap between system performance and human capabilities is rapidly closing, making the vision of a single speech system that outperforms humans in most scenarios imminently achievable.

The acknowledgments section expresses gratitude to Baidu's speech technology group for their assistance with data preparation and valuable conversations. The authors also thank Scott Gray, Amir Khosrowshahi, and Nervana Systems for their matrix multiply routines and discussions, as well as Natalia Gimelshein of NVIDIA for her insights on implementing the fast deployment matrix multiply. The references section lists numerous academic papers and resources related to speech recognition, neural networks, and deep learning. Appendix A discusses scalability improvements, detailing the node and cluster architecture, which consists of compute-dense nodes equipped with Intel CPUs and NVIDIA Titan X GPUs, substantial CPU memory for caching input data, and local storage for datasets. The software stack runs on these nodes, with the network primarily used for weight updates. Figure 8 provides a schematic of a training node, illustrating the arrangement of CPUs, PLX switches, and GPUs connected via PCI root complexes, optimized for fast communication between GPUs using GPUDirect. The section also details the GPU implementation of the CTC (Connectionist Temporal Classification) loss function, which involves forward and backward passes. The implementation addresses challenges related to numerical stability, data copying between CPU and GPU, and inefficient SIMD parallelism on the CPU. The GPU implementation computes all elements in each column of the alpha matrix, handling invalid entries with a special summation function that discards inputs of negative infinity. This approach, combined with a key-value reduction using characters as keys, allows for efficient gradient calculation. The GPU implementation utilizes fast shared memory and registers for high performance, storing the alpha matrix in shared memory and processing the beta matrix column by column. Due to floating-point arithmetic inaccuracies, the GPU and CPU implementations are not bit-wise identical but train models equally well when combined with utterance sorting by length.