Neural Turing Machines

Host: Let's dive into an idea that merges modern AI with classic computer design. In a paper titled Neural Turing Machines, researchers from Google DeepMind propose extending standard neural networks by connecting them to an external memory resource. Guest: Does that mean standard neural networks don't usually have a dedicated memory like a regular computer does? Host: Right, they typically rely entirely on their internal connections to remember things, which can be pretty limiting. By adding this external memory bank, the network can selectively interact with it using what the authors call attentional processes. Guest: So the neural network decides where to focus its attention to read or write data? Host: Exactly, and this makes the combined system highly analogous to a Turing Machine or a standard Von Neumann computer architecture, where the processor and memory are kept separate. But the massive breakthrough here is that this system is differentiable end-to-end. Guest: What does being differentiable actually mean in this context? Host: It means the memory operations are mathematically smooth and continuous, rather than rigid, discrete steps. That smoothness allows the entire machine to be trained efficiently using gradient descent, which is the standard way neural networks learn. Guest: That sounds powerful, but what kind of tasks can a neural network learn to do with this setup? Host: Preliminary results show that Neural Turing Machines can actually infer simple, step-by-step algorithms, like how to copy sequences, sort data, or perform associative recall. Amazingly, it figures out the rules for these algorithms simply by observing input and output examples.

Host: Let us look at how we can bridge the gap between traditional computer programming and modern artificial intelligence. Traditional programs rely on basic math, logical branching, and external memory, but standard machine learning has actually ignored the logic and memory parts for a long time. Guest: That seems like a huge limitation, especially for memory; are there any AI models that do not ignore it? Host: Recurrent neural networks, or RNNs, are the main exception because they are designed to process complex data over extended periods of time. They are actually known to be Turing-Complete, which means they theoretically have the capacity to simulate any computational procedure if they are wired up correctly. Guest: You said theoretically, which usually means there is a big catch when you actually try to build it. Host: You hit the nail on the head, because what is possible in principle is not always simple in practice. To fix this, the authors enriched standard recurrent networks by adding a large, addressable memory bank that the network can actively read from and write to. Guest: So they basically gave the neural network its own hard drive or RAM? What do they call this setup? Host: They call it a Neural Turing Machine, inspired by how Alan Turing originally added an infinite memory tape to basic finite-state machines. But unlike a classic Turing machine, this new device is a completely differentiable computer. Guest: What does it mean for a computer to be differentiable, and why is that useful? Host: It means the system's internal operations are mathematically smooth and continuous, rather than rigid. Because of that smoothness, the machine can be trained using standard gradient descent, which gives us a practical way to teach a neural network to learn actual computer programs.

Host: When we look at how human cognition mirrors computer algorithms, the closest match is our own working memory. Even though the brain's exact wiring is a bit mysterious, we generally understand working memory as our ability to hold onto information short-term and manipulate it using specific rules. Guest: So if we translate that to computer terms, the rules are like simple programs, and the information we hold onto is the data those programs use? Host: Exactly. And that brings us to what the authors call a Neural Turing Machine, or NTM, which is designed to act just like that human working memory system. It solves tasks by applying rules to what are called rapidly-created variables. Guest: What exactly is a rapidly-created variable? Host: It is just a piece of data quickly assigned to a memory slot. It works the same way a standard computer puts the numbers three and four into temporary registers to add them together and make seven. Guest: Does the NTM also choose which of these variables to focus on, the way our brains do? Host: Yes, it uses an attentional process to selectively read from and write to its memory. But here is the really exciting difference: while most models just follow a fixed set of procedures, the NTM actually learns how to use its working memory on its own. Guest: That sounds like a massive step forward. How do the researchers prove it actually works? Host: They start by reviewing related research across psychology, neuroscience, and AI before detailing their new memory architecture. After laying out the design, they run it through a battery of specific problem-solving tests to share the results.

Host: To really understand how we manipulate information in the short term, we have to look at how biological brains actually do it. In psychology, working memory is often pictured as having a "central executive" that directs attention and processes data held in a temporary memory buffer. Guest: So it's like a temporary mental workspace. Does that buffer have a strict limit on what it can hold? Host: It does, and psychologists often measure this capacity in "chunks" of information that a person can easily recall at once. These limits show the structural constraints of human memory, though the authors note they are perfectly happy to build computational systems that exceed those human limits. Guest: That makes sense if they are building an artificial model. What about the biological side—where does this memory process physically happen? Host: In neuroscience, working memory is closely tied to a system made up of the prefrontal cortex and the basal ganglia. Researchers study this by giving an animal a brief cue, making it wait through a short "delay period," and then asking it to respond based on that cue. Guest: What exactly is happening in the brain during that silent waiting period? Host: Individual neurons in the prefrontal cortex will actually keep firing continuously while they wait. This persistent firing is literally the brain actively holding onto that specific piece of information. Guest: The text also mentions the "dimensionality" of the population code predicting memory performance. What does that mean here? Host: It means scientists look at the complexity—or dimensionality—of how whole groups of neurons fire together, rather than just single cells. It turns out that the richer and more complex that group activity is during the delay, the better the subject actually remembers the cue.

Host: Let's look at how researchers have historically tried to simulate human working memory. These cognitive models usually fall on a spectrum, ranging from low-level biology, like how individual neurons sustain a signal, to higher-level systems designed to solve specific logic tasks. Guest: I imagine those higher-level systems are closer to what we see in modern AI. Is there a specific model that influenced the authors' work? Host: Yes, they point to a 2006 model by Hazy and colleagues. It is actually very similar to the Long Short-Term Memory, or LSTM, architecture, because it uses mechanisms to gate information into memory slots. Guest: By gating, do you mean the system learns to control what information gets saved and what gets ignored? Host: Exactly, which is great for solving memory tasks based on nested rules. But the authors note a major limitation in Hazy's model, which is that it completely lacks a sophisticated system for memory addressing. Guest: What does memory addressing mean in this context? Host: It is like giving a specific coordinate or index to every piece of information, similar to the way a computer's RAM operates. Without addressing, that older system is restricted to storing and recalling only very simple, atomic pieces of data. Guest: So the authors added addressing to handle more complex data structures. Do traditional neuroscientists agree that the brain works like a computer's RAM? Host: It is highly debated, and addressing is usually left out of computational neuroscience models entirely. However, the authors note that prominent cognitive scientists, like Gallistel, King, and Marcus, have strongly argued that the brain must use some form of addressing to function the way it does.

Host: Let's explore how the study of human thought and language actually grew up right alongside artificial intelligence. Back in the mid-twentieth century, cognitive science, linguistics, and AI all emerged together, heavily inspired by the invention of the computer. Guest: So they initially looked at the human brain as if it were a traditional computer? Host: Exactly, they relied on a symbol-processing metaphor, where intelligence was seen as the step-by-step, rule-based processing of symbols. But that changed when the connectionist revolution introduced neural networks, arguing that thought is actually sub-symbolic and based on patterns. Guest: Did the traditional cognitive scientists and linguists push back against these new neural networks? Host: They did, with two researchers named Fodor and Pylyshyn famously pointing out two major supposed flaws. The first was that neural networks couldn't handle variable-binding, which is the ability to assign a specific piece of data to a specific role. Guest: Could you give an example of variable-binding in everyday language? Host: Take the sentence, Mary spoke to John. To understand it, your brain binds Mary to the role of subject and John to the object, but critics argued an early neural network couldn't keep those slots straight. Guest: That makes sense. And what was the second major flaw they pointed out? Host: They argued that because early neural networks had fixed-length inputs, they couldn't process the variable-length structures we constantly use in human language. Guest: But researchers must have eventually solved those issues, right? Host: Yes, pioneers like Geoffrey Hinton and Paul Smolensky spent years developing mechanisms to handle both variable-binding and variable-length structures in neural networks. That foundational work is exactly what the architecture in this paper builds upon.

Host: Our minds have a remarkable ability to handle complex, layered information through a concept called recursive processing. Think of it as the capacity to embed ideas within other ideas, which is widely considered a defining hallmark of human cognition. Guest: Could you give an example of what embedding ideas actually looks like when we communicate? Host: It is like saying "the cat that chased the mouse that ate the cheese." We naturally process these variable-length, nested structures, and this specific ability actually sparked a massive firefight in the linguistics community over the last decade. Guest: What exactly were the top linguists arguing about? Host: The core issue was whether recursive processing is a uniquely human evolutionary innovation that evolved specifically to enable language. That was the stance supported by prominent figures like Fitch, Hauser, and Chomsky. Guest: That sounds like a bold claim, so what was the alternative theory? Host: The opposing camp, which included Jackendoff and Pinker, argued that recursive processing actually predates language entirely. They believed human language evolved through multiple different adaptations, rather than recursion being the single magic ingredient. Guest: So the debate was basically over whether recursion was born specifically for language, or if it was an older mental tool that language eventually utilized. Host: Exactly right. But despite that fierce disagreement about its evolutionary origins, both sides completely agreed on one final point: recursive processing is absolutely essential to our human cognitive flexibility.

Host: We're turning our attention to how machines handle information over time using Recurrent Neural Networks, or RNNs. These networks operate using a "dynamic state," meaning their next move depends on both the new input they receive and their current internal state. Guest: So it's like they have a running memory of what just happened, which helps them process whatever comes next? Host: Exactly. And unlike older systems like Hidden Markov Models, RNNs use a "distributed state" that gives them a significantly larger and richer memory capacity. This means a signal entering the network right now can alter its behavior much later on. Guest: That sounds powerful, but isn't it hard to keep track of a signal over a long time without the memory fading away or growing out of control? Host: It is, and that exact issue is known as the vanishing or exploding gradient problem, where the network's sensitivity to past inputs either dies out or blows up. To fix this, researchers introduced a crucial innovation called Long Short-Term Memory, or LSTM. Guest: How does LSTM keep the memory stable instead of letting it vanish or explode? Host: It relies on something called "perfect integrators" to store memory. Mathematically, it just takes the old memory state and simply adds the new input to it, which prevents the signals from dynamically shrinking or blowing up. Guest: But if it's always adding new inputs, wouldn't the memory eventually get overwhelmed with useless information? Host: It would, which is why LSTMs attach a programmable gate to that integrator. This gate looks at the context and allows the network to actually choose when it listens to new inputs. Guest: Oh, so it selectively decides what's important enough to add to the memory and what to just ignore? Host: Spot on. By using that context-dependent gate, the network can selectively store the truly important information for an indefinite length of time.

Host: Let us look into how certain neural networks deal with information that doesn't fit neatly into a fixed size. Specifically, Recurrent Neural Networks, or RNNs, are uniquely suited to processing variable-length structures. Guest: What exactly do you mean by a variable-length structure in this context? Host: Think about a spoken sentence or a translated document, where some sequences are short and others are long. Because the data arrives over multiple time steps rather than all at once, RNNs handle this sequential flow naturally without any modification. Guest: That makes sense, and that explains why they are used for cognitive tasks like speech recognition, text generation, and handwriting. Host: Exactly. Because RNNs natively handle varying lengths, the authors point out a major advantage, which is that we no longer need to build explicit parse trees. Guest: What is an explicit parse tree, and why is avoiding it a good thing? Host: Traditionally, researchers used parse trees to manually break down and merge data into rigid grammatical structures. The authors argue that forcing those manual structures is no longer urgent or valuable, thanks to how well RNNs learn sequences. Guest: So the network just figures out the structure on its own over time. The text also mentions models of attention and program search—how do those tie in? Host: Both differentiable models of attention and program search were also built using recurrent neural networks. They act as important precursors to this work, proving the foundational power of RNNs for complex sequential problems.

Host: To understand how a Neural Turing Machine actually works, we need to look closely at its underlying structure. At its core, the system is made of just two basic components: a neural network controller and a memory bank. Guest: I can picture the neural network part handling regular inputs and outputs, but how exactly does it interact with that memory bank? Host: The controller uses specialized network outputs that act as selective read and write "heads," which is a direct nod to classic Turing machines. These heads allow the network to continuously access and modify a separate memory matrix. Guest: But standard computer memory is discrete, meaning you either read a specific location or you don't. How can a neural network learn to do that using standard gradient descent? Host: That is the crucial innovation here; they made every single component differentiable by creating "blurry" read and write operations. Instead of hard-selecting a single digital address, the heads interact with all the elements in the memory to varying degrees. Guest: "Blurry" operations sound like they could get messy. How does the network manage to find the right information without mixing everything up? Host: It relies on an attentional focus mechanism that assigns a normalized weight to each row in the memory matrix. A head can focus sharply on a single location by giving it a high weight, or it can attend weakly across many locations. Guest: Oh, so it's focusing intensely on a tiny portion of the memory and mostly ignoring the rest? Host: Exactly, and because that interaction is highly sparse, the machine is heavily biased toward storing data cleanly without interfering with other stored memories.

Host: Let's look at exactly how a neural network retrieves information from a dedicated bank of memory. We can picture the memory at any given time as a grid, or matrix, with a certain number of locations, where each location holds a vector of data. Guest: So if I have a set number of locations in my memory grid, each location holds a specific, identically sized chunk of data? Host: Exactly. When the system wants to extract data from this grid, it uses a read head that generates a unique mathematical weight for every single memory location. Guest: Do these weights act like a spotlight, highlighting exactly which location to read from? Host: Yes, but instead of pointing to just one single spot, the weights are normalized fractions between zero and one that all add up to exactly one. Guest: Oh, I see, so it's technically reading a little bit from everywhere at once. How does it combine all those different pieces into a single output? Host: It calculates what is called a convex combination, which is essentially a weighted average. It multiplies each memory row by its corresponding fraction and adds them all together to produce a single final read vector. Guest: That sounds more blurred than just picking the single best row, so why blend everything together like that? Host: Because calculating a weighted average is a smooth, continuous mathematical operation, which makes the whole reading process differentiable. That crucial detail means the neural network can use standard calculus to actually learn exactly how to adjust its weights and improve its reading accuracy over time.

Host: Let's look at how this system actually puts new information into its memory bank. Taking a cue from older models like LSTMs, the writing process is split into two distinct steps: erasing old data, and then adding new data. Guest: That makes sense conceptually, but how does an artificial network actually "erase" something mathematically? Host: It uses an "erase vector" made up of values between zero and one, combined with a specific weight for that memory location. The memory is only completely wiped—reset to zero—if both the location's weight and the erase value are exactly one. Guest: So if either the focus weight or the erase value is zero, the memory just stays exactly as it was? Host: Exactly, which gives the system incredibly fine-grained control over what gets kept or cleared, right down to the individual components of a memory. And if there are multiple "write heads" erasing at once, the order doesn't matter because it's just basic multiplication. Guest: That handles the erasing, so how does the second step, the adding part, work? Host: After the erase step, each write head produces an "add vector" that gets multiplied by the location weight and then simply added to the remaining memory. Just like with erasing, the order in which multiple heads add things is totally irrelevant. Guest: Since it's a neural network, does this whole two-step process still allow the model to learn from its mistakes? Host: Yes, and that is the crucial part. Because both the erase and add operations rely on smooth, differentiable math, the entire composite write operation is differentiable, allowing the network to learn exactly how to manage its memory over time.

Host: To understand how a memory network knows exactly where to read or write its data, we need to look at the distinct mechanisms it uses to find those locations. We know the basic equations for reading and writing, but now we need to see how the network actually generates the weightings that target specific memory spots. Guest: So it isn't just randomly picking a spot to store or retrieve data? How does it actually decide where to focus its attention? Host: It combines two complementary methods, starting with what's called "content-based addressing." In this approach, the system's controller produces an approximation of what it's looking for, and zeroes in on the memory location with values that closely match that approximation. Guest: That sounds like searching for a specific lyric to find a song, rather than just looking up track number four. Why would it need another method if it can just search by the actual content? Host: It's a great method for simple retrieval, but think about an arithmetic problem where you have to multiply a variable "x" by a variable "y". The actual numbers stored in "x" and "y" could be absolutely anything, so searching by their specific content won't help you find the right variables. Guest: Ah, I see, because the values constantly change. You just need a recognizable, fixed bucket to put them in, regardless of what's inside. Host: Spot on, and that is why the second method is "location-based addressing." A controller can store those variables in distinct addresses, retrieve them purely by their location, and then run the multiplication algorithm. Guest: Do these two methods compete with each other, or does the system just switch between them depending on the task? Host: It actually employs both mechanisms concurrently to construct the final weighting vector whenever it reads or writes. Technically, content-based addressing is broad enough that you could just encode the location inside the content itself, but treating location-based addressing as its own distinct tool proved essential for solving broader, more generalized problems.

Host: Let us look into how this system actually finds what it needs in memory based on the information itself. When the network wants to read or write, its memory head generates a specific search query, which the text calls a key vector. Guest: Is that key vector basically like typing a string of keywords into a search engine? Host: Exactly, but instead of words, it is an array of numbers that gets compared to every single row currently sitting in the memory bank. To find a match, the system uses cosine similarity, which mathematically measures how closely the key vector aligns with each memory row. Guest: What happens once it figures out which memory rows are the most similar to that search key? Host: It produces a normalized score, or weight, for every single location. The more similar a memory row is to the search key, the higher its weight, meaning the system focuses more of its attention right there. Guest: But what if there are several decent matches, can the system choose to hyper-focus on just the absolute best one? Host: Yes, and it does that using a positive multiplier called key strength to amplify or attenuate the precision of that focus. If the key strength is turned up high, the system will heavily favor the single best match, but if it is low, it will spread its attention across multiple similar memories.

Host: Let's explore how the system navigates its memory banks by shifting its focus across specific physical locations. It can step through memory sequentially or make random-access jumps by mathematically rotating a weighting. Guest: How does rotating a weighting move the focus from one memory slot to another? Host: Think of it like sliding a magnifying glass along a row. If the system's weight is entirely on one spot, a shift of positive one moves the focus to the very next location, and a negative shift slides it backward. Guest: Does it just step forward blindly, or can it combine this movement with the actual data it's looking for? Host: It combines them using an interpolation gate, which is basically a blending dial set between zero and one. This dial mixes the location focus from the previous time-step with a brand new focus generated by the content system. Guest: So if the dial is set to zero, it ignores the new content entirely and sticks with the previous location? Host: Exactly, and if the gate is at one, it completely ignores the old location and jumps to the new content. After this blending is done, the system applies a final "shift weighting" to define exactly how many steps to slide left or right. Guest: How does the network calculate that exact shift amount? Host: The standard way is to output a set of probabilities for allowed moves, like negative one, zero, or positive one. But they also tested a shortcut where the system outputs just a single decimal number to control the shift. Guest: How does a single decimal translate into a shift over distinct, whole-number memory locations? Host: It essentially splits the shift between the two nearest integers. For example, if it outputs the number 6.7, it applies a 30 percent weight to shifting six spaces, and a 70 percent weight to shifting seven spaces.

Host: We're going to explore how a neural memory system controls its focus and the different ways it can navigate through data. Think of this process like moving a spotlight over a circular row of memory slots, which is done using a mathematical operation called circular convolution. Guest: Circular convolution sounds pretty technical, so how does that actually move our spotlight? Host: It takes your current focus and shifts it left or right by applying a set of shift weights. But if those weights aren't perfectly sharp—say, giving an 80% weight to staying put and 10% to moving left or right—the spotlight starts to blur across multiple memory slots. Guest: I see, so the memory system starts losing its precise grip on exactly which slot it's trying to read? Host: Exactly, which causes what they call leakage or dispersion over time. To combat this, the system applies a sharpening scalar called gamma, which is greater than or equal to one, to pinch that blurred weighting back into a tight, focused beam. Guest: That’s a clever fix. So once it has this sharp spotlight, how does the system choose where to look next? Host: It has three complementary modes it can use, starting with pure content addressing. In this first mode, the spotlight just jumps straight to a memory slot because the data inside perfectly matches what the system is searching for. Guest: What if it finds the right general area, but actually needs a specific piece of data right next to it? Host: That’s the second mode, where it finds a location by its content, and then shifts the focus over. In computing terms, it's perfect for finding a contiguous block of data and then accessing a specific element inside that block. Guest: Got it, find the neighborhood, then shift to the exact right house. And what's the third mode? Host: The third mode just keeps shifting the spotlight from its previous position without looking at the content at all. This lets the system easily iterate through a sequence of addresses, stepping forward by the same distance at every single time step.

Host: Let's focus on the brain coordinating all these operations, the controller network. When designing a Neural Turing Machine, the most significant architectural choice you have to make is what type of neural network to use as that controller. Guest: What are the main options for that? Host: You generally choose between a recurrent network, like an LSTM, or a standard feedforward network. If you think of the external memory matrix as a computer's RAM, a recurrent controller acts a lot like the central processing unit. Guest: Is that because a recurrent network has its own internal memory built in? Host: Exactly, its hidden states act just like the temporary registers inside a CPU. That internal memory allows the recurrent controller to effortlessly mix information across multiple time steps. Guest: Then why would someone choose a feedforward network if it lacks that internal memory? Host: Well, a feedforward controller can actually mimic internal memory by reading and writing to the exact same location in the external memory at every step. The big advantage there is transparency, since tracking those literal read and write locations is much easier for us to interpret than an RNN's hidden state. Guest: Is there a catch to relying completely on the external memory like that? Host: Yes, it creates a computing bottleneck based on how many read and write heads the network has. With just one read head, a feedforward network can only process one memory vector at a time, while a recurrent network can just store past reads internally to avoid that limit entirely.

Host: It's time to see how this system performs when we actually put it to the test. The researchers started with simple algorithmic tasks, like having the network copy and sort sequences of data. Guest: Those sound like really basic operations. What were they hoping to prove by keeping the tasks so simple? Host: They wanted to see if the Neural Turing Machine could do more than just memorize patterns by learning what they call a compact internal program. If it actually learns the underlying rules of a task, it should be able to generalize way beyond its training data. Guest: So it wouldn't just be regurgitating what it saw before. What does that generalization look like in practice? Host: Well, they were curious if a network trained to copy a short sequence of just 20 items could suddenly copy a sequence of 100 items without any extra training. To see if the NTM could pull this off, they compared it directly against a standard LSTM network. Guest: How did they structure these tests to compare the different architectures fairly? Host: They made all the tasks episodic, meaning they wiped the slate clean at the start of every new sequence. For the standard networks, they reset the hidden states, and for the NTM, they also completely cleared out its memory and read vectors. Guest: That makes sense, so every new sequence is a completely fresh start. How did they measure if the networks were succeeding? Host: The tasks were set up as supervised learning problems where the network had to predict binary targets, basically strings of ones and zeros. They then calculated the error in "bits-per-sequence" to measure exactly how far off those predictions were.

4.1 Copy. The copy task tests whether NTM can store and recall a long sequence of arbitrary information. The network is presented with an input sequence of random binary vectors followed by a delimiter flag. Storage and access of information over long time periods has always been problematic for RNNs and other dynamic architectures. We were particularly interested to see if an NTM is able to bridge longer time delays than LSTM. The networks were trained to copy sequences of eight bit random vectors, where the sequence lengths were randomised between 1 and 20. The target sequence was simply a copy of the input sequence (without the delimiter flag). Note that no inputs were presented to the network while it receives the targets, to ensure that it recalls the entire sequence with no intermediate assistance. As can be seen from Figure 3, NTM (with either a feedforward or LSTM controller) learned much faster than LSTM alone, and converged to a lower cost. The disparity between the NTM and LSTM learning curves is dramatic enough to suggest a qualitative,

rather than quantitative, difference in the way the two models solve the problem. We also studied the ability of the networks to generalise to longer sequences than seen during training (that they can generalise to novel vectors is clear from the training error). Figures 4 and 5 demonstrate that the behaviour of LSTM and NTM in this regime is radically different. NTM continues to copy as the length increases, while LSTM rapidly degrades beyond length 20. The preceding analysis suggests that NTM, unlike LSTM, has learned some form of copy algorithm. To determine what this algorithm is, we examined the interaction between the controller and the memory (Figure 6). We believe that the sequence of operations performed by the network can be summarised by the following pseudocode: initialise: move head to start location while input delimiter not seen do receive input vector write input to head location increment head location by 1 end while return head to start location while true do read output vector from head location emit output increment head location by 1 end while This is essentially how a human programmer would perform the same task in a low-level programming language. In terms of data structures, we could say that NTM has learned how to create and iterate through arrays. Note that the algorithm combines both content-based addressing (to jump to start of the sequence) and location-based addressing (to move along the sequence). Also note that the iteration would not generalise to long sequences without the ability to use relative shifts from the previous read and write weightings (Equation 7), and that without the focus-sharpening mechanism (Equation 9) the weightings would probably lose precision over time.

The repeat copy task extends copy by requiring the network to output the copied sequence a specified number of times and then emit an end-of-sequence marker. The main motivation was to see if the NTM could learn a simple nested function. Ideally, we would like it to be able to execute a “for loop” containing any subroutine it has already learned. The network receives random-length sequences of random binary vectors, followed by a scalar value indicating the desired number of copies, which appears on a separate input channel. To emit the end marker at the correct time the network must be both able to interpret the extra input and keep count of the number of copies it has performed so far. As with the copy task, no inputs are provided to the network after the initial sequence and repeat number. The networks were trained to reproduce sequences of size eight random binary vectors, where both the sequence length and the number of repetitions were chosen randomly from one to ten. The input representing the repeat number was normalised to have mean zero and variance one. Figure 7 shows that NTM learns the task much faster than LSTM, but both were able to solve it perfectly. The difference between the two architectures only becomes clear when they are asked to generalise beyond the training data. In this case we were interested in generalisation along two dimensions: sequence length and number of repetitions. Figure 8 illustrates the effect of doubling first one, then the other, for both LSTM and NTM. Whereas LSTM fails both tests, NTM succeeds with longer sequences and is able to perform more than ten repetitions; however it is unable to keep count of of how many repeats it has completed, and does not predict the end marker correctly. This is probably a consequence of representing the number of repetitions numerically, which does not easily generalise beyond a fixed range. Figure 9 suggests that NTM learns a simple extension of the copy algorithm in the previous section, where the sequential read is repeated as many times as necessary.

The previous tasks show that the NTM can apply algorithms to relatively simple, linear data structures. The next order of complexity in organising data arises from “indirection”—that is, when one data item points to another. We test the NTM's capability for learning an instance of this more interesting class by constructing a list of items so that querying with one of the items demands that the network return the subsequent item. More specifically, we define an item as a sequence of binary vectors that is bounded on the left and right by delimiter symbols. After several items have been propagated to the network, we query by showing a random item, and we ask the network to produce the next item. In our experiments, each item consisted of three six-bit binary vectors (giving a total of 18 bits per item). During training, we used a minimum of 2 items and a maximum of 6 items in a single episode. Figure 10 shows that NTM learns this task significantly faster than LSTM, terminating at near zero cost within approximately 30,000 episodes, whereas LSTM does not reach zero cost after a million episodes. Additionally, NTM with a feedforward controller learns faster than NTM with an LSTM controller. These two results suggest that NTM's external memory is a more effective way of maintaining the data structure than LSTM's internal state. NTM also generalises much better to longer sequences than LSTM, as can be seen in Figure 11. NTM with a feedforward controller is nearly perfect for sequences of up to 12 items (twice the maximum length used in training), and still has an average cost below 1 bit per sequence for sequences of 15 items. In Figure 12, we show the operation of the NTM memory, controlled by an LSTM with one head, on a single test episode. In “Inputs,” we see that the input denotes item delimiters as single bits in row 7. After the sequence of items has been propagated, a delimiter in row 8 prepares the network to receive a query item. In this case, the query item corresponds to the second item in the sequence (contained in the green box). In “Outputs,” we see that the network crisply outputs item 3 in the sequence (from the red box). In "Read Weightings,” on the last three time steps, we see that the controller reads from contiguous locations that each store the time slices of item 3. This is curious because it appears that the network has jumped directly to the correct location storing item 3. However we can explain this behaviour by looking at “Write Weightings.” Here we see that the memory is written to even when the input presents a delimiter symbol between items. One can confirm in “Adds” that data are indeed written to memory when the delimiters are presented (e.g., the data within the black box); furthermore, each time a delimiter is presented, the vector added to memory is different. Further analysis of the memory reveals that the network accesses the location it reads after the query by using a content-based lookup that produces a weighting that is shifted by one. Additionally, the key used for content-lookup corresponds to the vector that was added in the black box. This implies the following memory-access algorithm: when each item delimiter is presented, the controller writes a compressed representation of the previous three time slices of the item. After the query arrives, the controller recomputes the same compressed representation of the query item, uses a content-based lookup to find the location where it wrote the first representation, and then shifts by one to produce the subsequent item in the sequence (thereby combining content-based lookup with location-based offsetting).

The goal of the dynamic N-Grams task was to test whether NTM could rapidly adapt to new predictive distributions. In particular we were interested to see if it were able to use its memory as a re-writable table that it could use to keep count of transition statistics, thereby emulating a conventional N-Gram model. We considered the set of all possible 6-Gram distributions over binary sequences. Each 6-Gram distribution can be expressed as a table of 2^5 = 32 numbers, specifying the probability that the next bit will be one, given all possible length five binary histories. For each training example, we first generated random 6-Gram probabilities by independently drawing all 32 probabilities from the Beta(1,1) distribution. We then generated a particular training sequence by drawing 200 successive bits using the current lookup table. The network observes the sequence one bit at a time and is then asked to predict the next bit. The optimal estimator for the problem can be determined by Bayesian analysis (Murphy, 2012): P(B = 1|N1, N0, c) = (N1 + 1)/(N1 + N0 + 1), where c is the five bit previous context, B is the value of the next bit and N0 and N1 are respectively the number of zeros and ones observed after c so far in the sequence. We can therefore compare NTM to the optimal predictor as well as LSTM. To assess performance we used a validation set of 1000 length 200 sequences sampled from the same distribution as the training data. As shown in Figure 13, NTM achieves a small, but significant performance advantage over LSTM, but never quite reaches the optimum cost. The evolution of the two architecture's predictions as they observe new inputs is shown in Figure 14, along with the optimal predictions. Close analysis of NTM's memory usage (Figure 15) suggests that the controller uses the memory to count how many ones and zeros it has observed in different contexts, allowing it to implement an algorithm similar to the optimal estimator.

This task tests whether the NTM can sort data—an important elementary algorithm. A sequence of random binary vectors is input to the network along with a scalar priority rating for each vector. The priority is drawn uniformly from the range [-1, 1]. The target sequence contains the binary vectors sorted according to their priorities, as depicted in Figure 16. Each input sequence contained 20 binary vectors with corresponding priorities, and each target sequence was the 16 highest-priority vectors in the input. Inspection of NTM's We limited the sort to size 16 because we were interested to see if NTM would solve the task using a binary heap sort of depth 4.

Figure 16 illustrates an example of the input and target sequences for the Priority Sort Task. The input sequence is composed of random binary vectors and random scalar priorities. The target sequence, on the other hand, is a subset of these input vectors, specifically those that have been sorted according to their assigned priorities. This task is designed to test the network's ability to process and order information based on a given priority. Figure 17 delves into the memory usage of the Neural Turing Machine (NTM) during this Priority Sort Task. The left panel shows the hypothesized write locations, which are determined by fitting a linear function of the priorities to the observed write locations. The middle panel displays the actual observed write locations within the memory. The right panel illustrates the read locations, indicating where the network retrieves information from memory. The analysis suggests that the network uses the priorities to decide where to write information in memory. This hypothesis is supported by the observation that the locations predicted by the linear function closely match the actual observed write locations. Furthermore, the network reads from memory locations in an ascending order, effectively traversing the sorted sequence. The learning curves presented in Figure 18 further highlight the performance of the NTM. They demonstrate that an NTM equipped with both feedforward and LSTM controllers significantly outperforms a standard LSTM on this task. Notably, achieving optimal performance with a feedforward controller required the use of eight parallel read and write heads. This might be attributed to the inherent difficulty of sorting vectors using only unary vector operations, as discussed in Section 3.4.

The experimental details for the NTM and LSTM controllers are provided in Tables 1, 2, and 3. For all experiments, the RMSProp algorithm was employed for training, with a momentum of 0.9, as described by Graves (2013). Tables 1 and 2 detail the network configurations and learning rates used for the NTM with feedforward and LSTM controllers, respectively. All LSTM networks utilized three stacked hidden layers. It is important to note that the number of parameters in an LSTM network grows quadratically with the number of hidden units due to its recurrent connections. This contrasts with the NTM, where the parameter count does not increase with the number of memory locations. During the backward pass of training, all gradient components were clipped elementwise to the range of (-10, 10).

In conclusion, the Neural Turing Machine (NTM) has been introduced as a novel neural network architecture that draws inspiration from both biological working memory models and the design principles of digital computers. Similar to conventional neural networks, the NTM is fully differentiable, allowing it to be trained effectively using gradient descent. The experimental results presented in this work demonstrate that the NTM is capable of learning simple algorithms from example data and, crucially, of generalizing these learned algorithms to perform well outside of its training regime. This capability highlights the potential of the NTM for a wide range of sequence processing and algorithmic tasks.