Scaling Laws for Neural Language Models

Host: We are diving into the underlying mathematical rules that predict exactly how language models improve as they grow. This foundational research, led by Jared Kaplan, Sam McCandlish, and a team at OpenAI, set out to understand those very rules. Guest: How do they actually measure that improvement? What does "performance" mean in this context? Host: They measure it using something called cross-entropy loss, which essentially calculates how surprised a model is by the correct next word in a text. The lower that loss is, the better the model is at predicting language. Guest: And what factors actually drive that loss down? Host: They discovered that the loss drops following a very predictable mathematical curve, known as a power-law. This trend is driven by three main factors: the size of the model, the size of the dataset, and the total computing power used for training. Guest: Does the specific design of the neural network matter, like how deep or wide it is? Host: Surprisingly, no. They found that within a wide range, those architectural details have a minimal effect compared to just scaling up the model size, data, and compute. Guest: If compute is one of those main factors, how should a team decide where to spend a limited computing budget? Host: The equations they found point to a highly specific strategy. Because larger models are significantly more efficient at learning from data, the best approach is to train a very large model on a relatively modest amount of data. Guest: That sounds counterintuitive. Do they just stop training before the model has even fully learned that smaller dataset? Host: Exactly. The most compute-efficient method is to stop training significantly before the model reaches its maximum potential, or convergence. It is actually faster and cheaper to slightly under-train a massive model than to fully train a smaller one.

Host: To understand how artificial intelligence improves, it helps to look at the foundations of language modeling. Language is the perfect testing ground for AI because reasoning tasks can be easily evaluated through text, and there is an endless supply of internet data for unsupervised learning. Guest: That makes sense, since we already use language to express complex reasoning. But how do researchers measure an AI's progress as it reads all that text? Host: They look at a metric called language modeling loss, which essentially measures how often a model makes mistakes when predicting text. Thanks to the Transformer architecture, recent deep learning models have gotten so good at this that they are approaching human-level performance on many specific tasks. Guest: If the architecture is already approaching human levels, what exactly are we investigating here? Host: Instead of tweaking the architecture itself, this investigation focuses entirely on scale. The goal is to see exactly how performance changes depending on the size of the model, the amount of computing power used, and the size of the dataset. Guest: Ah, so testing the idea that bigger is better. Do we just see random improvements when we scale up, or is there a pattern? Host: That is the major discovery here, because the improvements are not random at all. By studying these models across more than seven orders of magnitude in scale, researchers found that performance follows precise power-law scalings. Guest: What does a power-law scaling mean in this context? Host: It means there is a highly predictable mathematical relationship at play. For example, if you consistently increase your computing power or dataset size, the model's error rate will drop by a steady, predictable percentage every single time.

Host: Let's dive right into the foundational rules of how language models improve. The biggest takeaway here is that a model's performance relies heavily on its scale, specifically three things, which are the number of parameters, the size of the dataset, and the amount of computing power used. Guest: Does the actual design of the model matter much, like how deep or wide the neural network is? Host: Not nearly as much as you might think. As long as you stay within reasonable limits, the shape of the model barely matters compared to those three main scaling factors. Guest: So if we want a better model, can we just crank up the model size and keep the dataset exactly the same? Host: If you do that, you will quickly hit diminishing returns. You have to scale the model size and the dataset in tandem, but interestingly, if you increase your model size by eight times, you only need to increase your data about five times to avoid a performance penalty. Guest: That is surprising, I would have guessed they needed to grow at the exact same rate. What happens if we have a strict limit on our computing power? Host: That leads to a fairly counterintuitive finding, which is that training a model to total convergence is actually inefficient. If you have limited compute, you should not just train a smaller model until it stops improving. Guest: Wait, so what is the optimal strategy instead? Host: You get the best results by building a massive model and stopping the training well before it fully converges. It turns out large models are incredibly sample efficient, meaning they reach the same performance using far fewer data points and steps than smaller models. Guest: That makes sense. So bigger models essentially learn faster, and everything follows highly predictable mathematical trends as you scale up.

Host: We can actually predict how well a language model will perform as we scale up our resources by looking at a few fundamental mathematical rules. The researchers found that a model's error rate, known as its test loss, follows a strict power law when it is bottlenecked by just one specific limit. Guest: What kind of limits are we talking about? Like running out of computing power? Host: Exactly, or running out of training data, or hitting the limit of the model's own parameter size. If you isolate just one of those factors—parameters, data, or compute—and provide plenty of the other two, the model's performance improves in a highly predictable way. Guest: When you say it improves predictably, how does that power law actually work in practice? Host: It means that every time you multiply your limiting resource, the model's mistakes drop by a consistent, calculable percentage. The paper even provides three specific equations to map this out for parameters, data, and compute budgets. Guest: So what happens if you get a massive budget increase, like a billion times more compute? Do you just scale up the model size, the data, and the training time equally? Host: That is the natural assumption, but the answer is no. To use that compute optimally, the vast majority of it should go toward increasing the model's size. Guest: Really? I would have assumed you would need a massive increase in training data to feed that much bigger model. Host: Surprisingly, the data requirements grow relatively slowly. You only need a small increase in data to ensure the model doesn't just memorize the same text over and over. Guest: If the data is not growing nearly as fast as the model size, how is all that extra compute actually being applied? Host: Aside from the model size, a lot of it goes into increasing the batch size, which lets the computers process the data in parallel. The actual serial training time, meaning the number of sequential steps the model takes, barely needs to increase at all.

Host: We are moving into the practical side of things now, looking at how to perfectly balance our resources when building these models. Researchers found that a model's performance loss follows highly predictable mathematical equations based mostly on dataset size and model size. Guest: Just those two factors? How reliable are those equations if you completely change the model's underlying design? Host: They are incredibly robust. These relations hold true across eight orders of magnitude in compute, and they barely care about the model's shape, like its depth or width, as long as the training dataset stays the same. Guest: That is wild to think about. So if we trust the math, how should we balance things when we scale up? Host: They actually created a single combined equation for model size and dataset size to figure that out. It reveals that as you increase your model size, you should increase your dataset size sublinearly. Guest: Sublinearly? So the dataset doesn't need to grow as fast as the model's parameter count? Host: Exactly. In fact, if you suddenly get a massive new compute budget, the math dictates you should spend it primarily on making the model itself larger, rather than training for way longer. Guest: I would have assumed you'd just feed a massive new dataset into the model and let it run. Host: It's a common assumption, but the optimal formula shows model size taking the lion's share of the new budget, scaling at a power of 0.73. Meanwhile, your batch size and training steps only see very minor increases. Guest: Why does the bigger model not need an equally massive boost in training steps or data? Host: Because as models grow larger, they become increasingly sample efficient. They extract better patterns from the data they do see, so you don't need dramatic increases in training time to see big improvements.

Host: Before we can see how language models scale, we need to agree on exactly what we are measuring and how we are defining the math. To start, the authors lay out their core notation, and one crucial detail is how they count the model size, which they label as N. Guest: Usually, a model's size is just its total number of parameters. Are they calculating it differently here? Host: Yes, they intentionally exclude all the parameters used for the vocabulary and positional embeddings. They do this because leaving those specific parameters out produces significantly cleaner scaling laws. Guest: That makes sense, so N represents just the core network itself. What other main variables are they tracking? Host: They track L, which is the cross-entropy loss measuring how well the model predicts the next token over a context of 1024 tokens. They also carefully track C, which stands for the total training compute. Guest: How do they calculate that compute? Are they just counting how many days the servers were running? Host: They actually use a math formula to estimate it instead of wall-clock time. They calculate compute as roughly six times the core model size, multiplied by the batch size and the number of training steps. Guest: So every time you update the parameters or feed the model more data in a batch, that compute number goes up. Host: Exactly, and they measure that total compute in massive units called petaflop-days. By standardizing these math definitions and training everything on an extended dataset called WebText2, they create a strictly level playing field for their experiments.

Host: Let's transition into the actual experimental results and the core scaling rules these researchers uncovered. To make sense of those findings, we first need to know how they estimated the computational cost of their experiments. Guest: How exactly did they measure the computing power required to train all these models? Host: They used a straightforward formula: about six floating-point operations per training token for every parameter in the model. This includes the forward pass, where the model makes a prediction, and the backward pass, where it actually learns and updates its weights, which takes roughly twice as much compute. Guest: That makes sense. What did the day-to-day training process look like? Host: They mostly used a standard optimization tool called Adam for a set number of steps, but for their absolute largest models—those with over a billion parameters—they had to switch to a more memory-efficient optimizer called Adafactor. They also used a learning rate schedule that briefly warmed up and then gradually decreased to zero. Guest: And what kind of text were they feeding the models during this training? Host: They used an expanded dataset called WebText2, which was built by scraping web pages linked from Reddit posts that had at least three "karma" points. That three-karma threshold acted as a clever quality filter, ensuring real people found the link interesting or useful. Guest: That's a smart way to filter the internet. How much data did that actually yield? Host: It resulted in over 20 million documents and nearly 23 billion tokens of text, though they held some back to test the models later on data they hadn't seen. Guest: So with the compute and the data locked in, what specific dials were they turning to see how scaling works? Host: They systematically varied five main factors, most notably the model size—ranging from a tiny 768 parameters all the way up to 1.5 billion—and the dataset size, testing anywhere from 22 million to 23 billion tokens. Guest: Did they change anything about how the models were structured? Host: Yes, they tweaked the model shapes by altering the depth and width, and they also experimented with different context lengths and batch sizes. Adjusting these specific variables is what allowed them to isolate exactly how scaling drives performance.

Host: Let's look at whether the specific physical architecture of a language model actually dictates how well it learns. It turns out that a model's performance relies almost entirely on its total number of non-embedding parameters, rather than its specific shape. Guest: When you say the model's shape, are you talking about how the neural network is structured? Host: Exactly, shape refers to structural choices like the number of layers, the width of the network, or the number of attention heads. As long as you keep the total parameter count the same, changing those details barely affects the final performance. Guest: That is surprising, so a short, wide model with just six layers could perform just as well as a really deep one with forty-eight layers? Host: Yes, researchers found that even if you vary that aspect ratio by a factor of forty, the difference in performance is only about three percent. But there is a catch here, because this rule only holds true if you strictly look at the non-embedding parameters. Guest: Why is that distinction so important? What happens if you just count all the parameters, including the embeddings? Host: Embeddings are basically the model's dictionary for translating raw text into numbers, and they can take up a massive portion of the parameter count. If you include them in your calculations, it creates an illusion that the number of layers strongly dictates performance. Guest: Oh, I see, so the size of that dictionary ends up skewing the data and hiding the true behavior of the network's core logic. Host: Precisely. Once you filter out those embedding parameters, the performance of models across wildly different depths and widths all magically converge onto one single, highly predictable trend line. Guest: Does this convergence hold true for literally any shape you can build? Host: The only time it breaks down is if you build something extreme, like a very shallow model with fewer than two layers, or one with incredibly disproportionate depth-to-width ratios.

Host: Let's explore how a model's performance shifts when we look closely at the size of its dataset and the specific parameters we choose to measure. To see the clearest trends in how these models improve, researchers found they had to exclude the "embedding" parameters and just look at the core network size, which they call N. Guest: Why exclude the embedding parameters? Do they just throw off the math? Host: Exactly, they obscure the underlying trend because you can actually shrink the embedding matrix without really hurting the model's performance. Once you isolate N, you see a perfect power-law improvement, not just on the training data, but on completely unseen datasets too. Guest: How does this scaling hold up if you compare a Transformer to an older architecture, like an LSTM? Host: When matched on that core parameter count, LSTMs actually perform just as well as Transformers on the first few words of a text. But as the context gets longer, Transformers pull way ahead because larger models are much better at quickly recognizing long-distance patterns. Guest: That makes sense for the data it was trained on, but does this improvement hold up if we test the model on totally different types of text? Host: It does, which is a massive finding; the models' performance on new, unseen datasets improves in direct parallel with their size. Generalization depends almost entirely on how well the model learned its primary training data, rather than how long it trained or how many layers deep it is. Guest: So if we want to keep improving that performance, how does the size of the dataset itself factor in? Host: The dataset size follows its own simple power-law trend, where feeding the model more tokens predictably lowers the error rate. And since we can easily estimate the total compute used based on the model size, batch size, and number of training updates, we can mathematically scan for the most efficient model setup.

Host: We're now going to look at what happens when we push the limits of both data and model size simultaneously to understand overfitting. But before jumping into the math, the authors noted something fascinating about how well these models generalize to new data outside their training set. Guest: Do they struggle when tested on completely different data distributions? Host: There is a small offset in performance, but the ability to handle new data actually improves smoothly as the model size grows. Even better, generalization depends entirely on how well the model learned its training data, rather than what specific phase of training it happens to be in. Guest: That's pretty reassuring. So how do they actually map out this relationship between model size, data, and overfitting? Host: They propose a new equation for the loss that combines Model Size, which they call "N", and Dataset Size, or "D". By varying N and D together, they can predict exactly how much data we need to train larger models while keeping overfitting in check. Guest: How do they figure out the exact formula for that combined equation? Host: They built it around three logical principles. First, they knew that if you change the vocabulary size or how text is chopped up into tokens, the overall loss just scales by a constant factor. The equation naturally allows for this rescaling using built-in constants. Guest: Okay, so those constants absorb any quirks in the vocabulary. What's the second principle? Host: It's about the extremes. If you freeze your dataset size and make your model infinitely large, your performance is entirely bottlenecked by the data. Conversely, with infinite data, your bottleneck is purely the model size. Guest: That makes perfect sense; the equation has to cleanly default to whichever resource is limiting you. And the third rule? Host: The third is a bit more theoretical. It dictates that the math should be smooth—or "analytic"—as the dataset approaches infinity. That just means the formula can be expanded out into a clean series of fractions, though the authors admit this rule has less solid theoretical proof than the first two.

Host: Let's explore how a model's error rate shifts when we change both the size of the neural network and the amount of training data. When we have a massive dataset, making the model larger improves its performance predictably along a smooth power law curve. Guest: But what happens if the dataset is small and we just keep making the model bigger anyway? Host: Performance eventually hits a wall and stops improving, which is a classic sign of overfitting. The researchers found that the severity of this overfitting is primarily driven by the ratio between the model's capacity and the dataset's size. Guest: Did they figure out the underlying rules for why it behaves that way? Host: They did, proposing three core principles, starting with a simple data bottleneck. This first principle states that your model's performance is always limited by whichever is smaller between the model size or the dataset size. Guest: That makes sense, since a giant model can't learn everything from just a few examples. What are the other two principles? Host: The second principle notes that models simply cannot reach their theoretical best possible performance if they only have a finite amount of data. The third principle is a bit more speculative, tying overfitting to the statistical variance in the dataset. Guest: Does that dataset variance explain why changing the data size has a different impact than changing the model size? Host: Exactly, because variance naturally shrinks as a dataset gets larger, which explains why the two variables behave asymmetrically. They actually built a master equation using these principles that fits their experiments perfectly, because it models both variables together rather than in isolation.

Host: We are turning our attention to how the size of our data batches and the laws of scaling actually interact during training. The researchers found that the critical batch size follows a power law tied directly to the loss, and it is largely independent of the model's actual size. Guest: Wait, so the ideal batch size depends on the loss, not how big the model itself is? Host: Exactly, and the relationship is remarkably consistent. Every time the loss decreases by thirteen percent, the critical batch size approximately doubles because of the gradient noise scale. Guest: What about the overall size of the training dataset as the model gets bigger? Host: To avoid overfitting, they found that the dataset size only needs to grow sub-linearly relative to the model size. Specifically, the required data scales to the model size to the power of 0.74. Guest: So if I massively increase my model's parameters, I don't need to increase my training data by the exact same proportion? Host: Right, but the authors warn that simply avoiding overfitting doesn't automatically give you the most compute-efficient training overall. They note that regularization wasn't even optimized during these specific experiments. Guest: Then how do we figure out the truly optimal way to allocate our computing power? Host: That brings in a new metric they define called a universal training step, which helps account for any non-optimal batch sizes. This allows them to fit the loss dependence accurately and predict the absolute best balance between model size and total training time.

Host: Let's look at how adjusting the number of data examples processed at once impacts our overall training efficiency. Specifically, researchers rely on a threshold known as the critical batch size, often called B-crit. Guest: What exactly makes a batch size critical? Host: It is essentially the sweet spot for balancing time and computing power. If your batch size stays below B-crit, your compute efficiency remains pretty strong, but if you exceed it, you start seeing diminishing returns. Guest: So feeding the model too much data in one go just wastes computing resources. How do we know where that threshold actually is? Host: The researchers propose using something called the gradient noise scale to predict it. When you train right at this critical batch size, you get an optimal tradeoff, requiring exactly twice the absolute minimum number of training steps and twice the absolute minimum data examples. Guest: That seems like a very precise balance to strike. Does this critical threshold change if you are training a much larger model? Host: Surprisingly it does not, because B-crit is completely independent of the model size. Instead, it depends entirely on your target loss, meaning how low you want your error rate to be. Guest: So as the model learns and gets closer to its lowest possible error, the ideal batch size shifts? Host: Exactly, it follows a specific mathematical curve, or power law, based on that target loss. The researchers use this relationship to estimate the absolute minimum compute you will need to hit your target error rate, regardless of the model scale.

Host: Let's explore how a model's error rate shrinks as we increase both its size and its training time. The researchers found a surprisingly simple equation that predicts this loss based on just the number of parameters and the minimum training steps, assuming infinite data. Guest: Is that prediction actually accurate? I'd imagine training massive neural networks would be too complex for one simple formula. Host: The researchers admit the fit isn't flawless, but it's incredibly compelling precisely because it's so simple. The fact that the loss follows a predictable power-law drop-off tells us something profound about the mathematical terrain the optimizer is navigating. Guest: By terrain, are you talking about the shape of the loss landscape? Host: Exactly, specifically something called the Hessian eigenvalue density, which essentially measures the curvature of that landscape. The math suggests that this curvature remains almost completely independent of the model's size. Guest: That makes sense for infinite data, but what if our dataset is limited? Don't we have to stop training early to prevent overfitting? Host: We do, and that's why the paper derives a specific lower bound for that early stopping point. They noticed that learning curves for finite and infinite data look identical right up until that crucial stopping step. Guest: So how do they calculate exactly when to pull the plug on training? Host: They use an inequality that relates the early stopping point to the gap between the current loss and the theoretical converged loss you would get with infinite data. The amount of expected overfitting ends up being directly proportional to that mathematical gap.

Host: Let's look at how we actually spend our computing resources when training these models. If you have a strict limit on your compute budget, it turns out there is a specific, optimal model size you should aim for to get the best performance. Guest: So just making the model as big as possible isn't the right move if my budget is capped? Host: Exactly, because if the model is too big, you won't have enough compute left over to train it properly. Interestingly though, smaller models actually require more training steps to reach a specific performance level, while larger models require fewer steps. Guest: That feels a bit counterintuitive, so larger models essentially learn faster per step? Host: They do, as long as you look past the initial messy phase of training and enter what's called the power-law region. Once you are in that stable region, the relationship between how much compute you spend—measured here in Petaflop-days—and how much your error rate drops becomes incredibly predictable. Guest: Is that predictability what allows researchers to figure out exactly how to spend their budget? Host: Spot on. The paper uses that predictable trend to find the perfect mathematical allocation of compute between increasing the model's size and processing more training data. Guest: And does the way we feed that data into the model matter for maximizing that budget? Host: It absolutely does, which is why the paper focuses on training at the critical batch size. By feeding the model that optimal chunk of data at one time, their theoretical formulas and actual experiments perfectly agree on how to best split the overall compute budget.

Host: Let's explore how to get the absolute best performance out of whatever computing budget we have available. When we allocate our compute optimally, we find a really clean mathematical relationship, or power-law, between the amount of compute we use and the lowest possible error we can achieve. Guest: If I get a massive increase in my compute budget, what exactly am I spending it on to hit that optimal error rate? Host: The paper shows you should primarily spend it on building a bigger model. The ideal model size scales up significantly alongside compute, following a trend where the size grows by a power of 0.73 relative to your computing budget. Guest: So if most of the new budget goes toward scaling up the model's size, what happens to the batch size and the total number of training steps? Host: Your critical batch size also needs to grow, but at a slower rate, scaling by a power of 0.24. The really surprising finding, though, is what happens to the optimal number of training steps. Guest: Let me guess, do the training steps stay mostly the same even if the model gets huge? Host: Exactly, the optimal number of steps grows incredibly slowly, at a power of just 0.03 relative to compute. That growth rate is so tiny the researchers note it is essentially consistent with zero increase. Guest: That means highly efficient training relies on using massive models and larger batches, rather than running endless optimization steps. Host: You hit the nail on the head. Because optimal training uses surprisingly few steps, the paper suggests that finding new ways to speed up those early training dynamics would be a highly valuable area for future research.

Host: Let's explore how we can forecast a language model's ultimate performance just by knowing our total computing budget. The authors take an equation they built earlier—which calculated loss based on model parameters and training steps—and adapt it to focus entirely on compute. Guest: What's the trick to converting training steps into total compute? Host: They use a straightforward mathematical substitution, defining the minimum training steps as the total compute divided by six times the model size and batch size. By plugging that in, the math predicts the lowest possible loss for any fixed amount of compute. Guest: So if I have a strict compute budget for my project, this formula tells me the absolute lowest error rate I can expect? Host: Exactly, it gives a specific power-law prediction for the loss. When the authors compared this theoretical prediction to the actual results on their experimental charts, the agreement was excellent. Guest: Does the framework also tell you exactly how many parameters your model should have for that specific budget? Host: It does. The math predicts that the optimal model size scales proportionally with the compute budget, dictated by a specific ratio of their scaling exponents. Guest: And did that prediction for optimal model size hold up in their real-world tests too? Host: It matched their observed data within just a few percent. Ultimately, the authors conclude that these scaling laws aren't just observations, but rather a highly reliable, predictive framework for building language models.

Host: We're going to explore how different training factors behave as we scale up, and where those mathematical patterns might actually break down. For instance, when researchers increase their computing budget, the optimal size of the AI model grows incredibly fast. Guest: Since the model is getting that much bigger, do they also have to increase the number of training steps to match? Host: Surprisingly, no. The number of optimization steps grows very slowly, which means any extra data they process is mostly just packed into much larger batch sizes. Guest: That sounds like a mismatch. Doesn't it cause a problem if the model size explodes but the training steps barely increase? Host: It does, and the researchers highlight a fascinating contradiction here. Because the data needed for efficient training grows so slowly, the performance predictions based on computing power eventually clash with what that data can actually support. Guest: What exactly happens when those two prediction curves collide? Host: The paper suggests the scaling laws will simply break down at that intersection point. Performance trends have to level off eventually anyway, because human language has a certain amount of inherent unpredictability, which they call "non-zero entropy." Guest: So that intersection point where the math breaks down might actually be a hard ceiling? Host: Exactly, that is their main conjecture. They believe this specific intersection point estimates the exact moment where Transformer language models will reach their absolute maximum performance.

The paper further elaborates on the contradiction identified in Section 6.3. It states that to maintain control over overfitting, dataset size should scale as D ∝ N^0.74 ∝ C^0.54, using compute-efficient N(Cmin) from Figure 14. This scaling is compared to the data requirements of compute-efficient training, where data usage grows with compute as D(Cmin) ∝ (Cmin)^0.26, assuming training at the critical batch size and no data reuse. This implies that dataset size grows much slower than required by the overfitting control strategy. The paper suggests this implies that compute-efficient training might encounter overfitting problems even without data reuse. The loss is expected to scale as L(D) ∝ D^-0.095 when bottlenecked by dataset size, leading to L(D(Cmin)) ∝ C^-0.03. This contradicts the prediction L(Cmin) ∝ C^-0.050 from Figure 13. The intersection point of L(D(Cmin)) and L(Cmin) is estimated to occur at C* ~ 10^4 PF-Days, N* ~ 10^12 parameters, D* ~ 10^12 tokens, and L* ~ 1.7 nats/token. The numerical values are noted as highly uncertain. The most straightforward interpretation is that the scaling laws break down at or before this point, which is still many orders of magnitude away in compute and model size.

The paper speculates that the intersection point identified in Section 6.3 might have a deeper meaning, suggesting that if model size cannot be increased beyond N* without different data requirements, then perhaps all reliable information in natural language data has been extracted. In this scenario, L* could represent the entropy per token of natural language. The loss trend might level off at or before L*. The paper considers a functional form for L(Cmin) as it levels off, by imagining adding noise to the training dataset. This would artificially boost the loss by a constant additive factor, making the distance from the noise floor L - Lnoise a more meaningful performance metric. The critical point would not change in absolute value, potentially remaining meaningful even after leveling off. Section 7 reviews related work on power-law scalings in density estimation, random forests, and neural network models. It discusses early work on scaling between performance and dataset size, and more recent investigations into scaling between model size and data size. Parallels are drawn with findings on optimal compute allocation and power-law learning curves. The paper also mentions EfficientNets and their scaling behavior, as well as studies on scaling with both dataset and model size. Comparisons are made regarding architectural hyperparameters, the role of deep versus wide networks, and the concept of a "jamming transition" in overparameterized models.

Section 8 provides a discussion of the findings. The paper reiterates the consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as captured by Equations (1.5) and (1.6). It notes the weak dependence on architectural and optimization hyperparameters. The diminishing returns with increasing scale are attributed to the power-law nature of these scalings. The paper draws an analogy to the ideal gas law, suggesting that these scaling relations describe macroscopic properties in a universal way, independent of microscopic details. The authors conjecture that these scaling relations may apply to other generative modeling tasks with maximum likelihood loss and in other settings. They express interest in testing these relations on domains like images, audio, and video, and in exploring whether a theoretical framework can be derived from these observations. The importance of investigating whether continued improvement in loss translates to improvements in relevant language tasks is highlighted, noting that smooth quantitative change can mask major qualitative improvements. The results strongly suggest that larger models will continue to perform better and be more sample-efficient, potentially making big models more important than big data. The paper also discusses the potential benefits of further investigation into model parallelism, pipelining, and techniques like sparsity or branching for faster training of large networks.

The paper concludes by summarizing the observed scaling laws and their implications. It states that the scaling relations go beyond mere observation to provide a predictive framework for compute scaling, overfitting, early stopping, and data requirements in large language models. The authors suggest that these relations are analogous to the ideal gas law, relating macroscopic properties universally. They propose that these scaling laws might apply to other generative modeling tasks and settings, and express a desire for a theoretical framework to derive them. The paper emphasizes that while smooth improvements in loss are observed, these may hide qualitative changes in capability. The findings strongly suggest that larger models will continue to outperform, and that sample efficiency will increase. The authors also touch upon the potential of model parallelism and other techniques to accelerate the training of large networks. The acknowledgements section thanks various individuals for their discussions and feedback. The appendices provide a summary of power laws and an empirical model of the compute-efficient frontier.

The compute budget. We start with the Equation (1.6), repeated here for convenience: L(N, S) = (N/S) + (S/N) * (S_c/S). Here, S represents the number of parameter updates when training at the critical batch size [MKAT18], which was defined in Equation (5.2): B(L) = B* * (1/a_B * L^(-1/a_B)). We would like to determine optimal training parameters for a fixed compute budget, so we replace S = C / (6NB(L)), where C is the number of FLOPs used in the training run: L(N, C) = (N/C) + (C/N) * (a_N / (6B_S * S_c)). Now, we set dL/dN|c = 0 to find the condition for optimality: 0 = dL/dN|c = -C/N^2 + (C/N^2) * (a_N / (6B_S * S_c)) - (C/N^2) * (a_N / (6B_S * S_c)) * (1 - S_c/N). This simplifies to (N/C) = (a_N / (6B_S * S_c)) * (1 - S_c/N). Equation (B.3) and (B.4) together determine the compute-efficient frontier.

B.2 Efficient Training. Now we assemble the implications of (B.3) and (B.4). First, note that inserting (B.4) into (B.3) yields L(Neff(C), C) = (1 + a_N/a_S) * L(Neff, infinity), which implies that for compute-efficient training, we should train to a fixed percentage a_N/a_S ≈ 10% above the converged loss. Next, let's determine how the optimal loss depends on the compute budget. Eliminating N yields a power-law dependence of performance on compute: L(C) = (C/C_c)^(-a_C), where we defined a_C = 1/(1/a_S + 1/a_B + 1/a_N) ≈ 0.052. C_c = 6N_c B_S * (1 + a_N/a_S)^(1/a_S) * (a_N/a_S)^(1/a_N). Similarly, we can eliminate L to find N(C): N(C) = (C/C_c)^(1/a_C) * (1 + a_N/a_S)^(-1/a_S), and S(C) = (C/C_c)^(-a_C/a_S) * (1 + a_N/a_S)^(-1/a_S) * (C_c / (6N_c B_S)). There is a slight ambiguity here: we can imagine training either at a constant batch size B (L_target), or we could instead train at a variable batch size B(L), where B is the instantaneous critical batch size (as opposed to B_S, which is the averaged version). These two prescriptions result in the same number of steps, so we can ignore this subtlety (see [MKAT18]).

B.3 Comparison to Inefficient. Typically, researchers train models until they appear to be close to convergence. In this section, we compare the efficient training procedure described above to this more typical setup. We define a the convergence factor f as the percent deviation from the converged loss: L(N, C) = (1 + f) * L(N, infinity). For compute-efficient training we have f = a_N/a_S ≈ 10% from the previous section, but researchers typically use a much smaller value. Here, we choose f' = 2% as an estimate. For a fixed value of the loss, we predict: N_f = N * (1 + f) / (1 + f')^(1/a_N) ≈ 2.7 * N. S_f = S * (1 + f') / (1 + f)^(1/a_S) ≈ 0.13 * S. C_f = C * (1 + f') / (1 + f)^(1/a_C) ≈ 0.35 * C. So that compute-efficient training uses 7.7x fewer parameter updates, 2.7x more parameters, and 65% less compute to reach the same loss.

B.4 Suboptimal Model Sizes. We can solve A.1 to find an expression for the amount of compute needed to reach a given value of the loss L with a model of size N: C(N, L) = 6B_S * (N/L)^(1/a_B) * (N/L)^(1/a_N) * (N/L)^(1/a_S). Using A.6 and A.9, we can eliminate L in favor of N_eff(L), the model size which reaches L most efficiently. From there, we find an expression for the excess compute needed as a consequence of using a suboptimal model size: C(N, N_eff) / C(N_eff, N_eff) = (N/N_eff)^(1 - 1/a_S) * (a_N / a_S)^(1/a_S) * (N/N_eff)^(1/a_N). The result is shown in Figure X. Models between 0.6x and 2.2x the optimal size can be used with only a 20% increase in compute budget. Using a smaller model is useful when accounting for the cost inference. A larger model can be trained the the same level of performance in fewer steps, allowing for more parallelism and faster training if sufficient hardware is available (see Figure Y): S(N, N_eff) / S(N_eff, N_eff) = (N/N_eff)^(-1/a_S) * (1 + a_N/a_S)^(-1/a_S) * (N/N_eff)^(1/a_N). A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute. Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve after initial transient effects.

C Caveats. In this section we list some potential caveats to our analysis. At present we do not have a solid theoretical understanding for any of our proposed scaling laws. The scaling relations with model size and compute are especially mysterious. It may be possible to understand scaling at very large D holding model size fixed [AS17], and also the shape of learning curves late in training, by modeling the loss with a noisy quadratic. But the scaling with D at very large model size still remains mysterious. Without a theory or a systematic understanding of the corrections to our scaling laws, it's difficult to determine in what circumstances they can be trusted.

Figure 16 Left: We characterize the step on which early stopping occurs, as a function of the extent of overfitting. The red line indicates a lower bound for early stopping that is derived in Section 5.3. Right: We display train and test loss for a series of 300M parameter models trained on different sized dataset sub-samples. The test loss typically follows that of a run done with unrestricted data until diverging. Note that the degree of overfitting (as compared to the infinite data limit) is significantly overestimated by L_test - L_train (denoted by a black bar for each run). We are not especially confident in the prediction of B_crit(L) for values of the loss far outside the range we have explored. Changes in B_crit could have a significant impact on trade-offs between data parallelism and the number of serial training steps required, which would have a major impact on training time. We did not thoroughly investigate the small data regime, and our fits for L(N, D) were poor for the smallest values of D (where an epoch corresponded to only 40 steps). Furthermore, we did not experiment with regularization and data augmentation. Improvements in these could alter our results, quantitatively or qualitatively. We used the estimated training compute C ≈ 6NBS, which did not include contributions propor-tional to n_ctx (see Section 2.1). So our scalings with compute may be confounded in practice in the regime of very large n_ctx, specifically where n_ctx ≥ 12d_model. We tuned learning rates, and we experimented with learning rate schedules. But we may have neglected to tune some hyperparameter (e.g. initialization scale or momentum) that have an important effect on scaling. The optimal choice of learning rate is sensitive to the target loss. When training close to convergence, it may be necessary to use a smaller learning rate to avoid divergences. But when conducting a short training run (e.g. due to compute limitations), it may be possible to use a larger learning rate. We did not experiment with higher learning rates for training runs that did not proceed to convergence.

D Supplemental Figures. D.1 Early Stopping and Test vs Train. In section 5.3 we described the result shown in Figure 16, which provides a prediction for a lower bound on the early stopping step. We also show the train and test loss for a given model size when training on different sized datasets. D.2 Universal Transformers. We compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17. These models re-use parameters, and so perform slightly better as a function of N, but slightly worse as a function of compute C. We include several different different possibilities for parameter re-use. D.3 Batch Size. We measure the critical batch size using the data displayed in figure 18. This made it possible to estimate B_crit(L) in figure 10.

Figure 17 We compare recurrent Transformers [DGV+18], which re-use parameters, to standard Trans-formers. Recurrent Transformers perform slightly better when comparing models with equal parameter count, but slightly worse when accounting for reuse and comparing per FLOP. Figure 18 These figures demonstrate fits to Equation (5.1) for a large number of values of the loss L, and for two different Transformer model sizes. These fits were used to measure B_crit(L) for Figure 10. D.4 Sample Efficiency vs Model Size. It is easy to see from figure 2 that larger models train faster, and are therefore more sample efficient. We provide another way of looking at this phenomenon in figure 19, which shows when different models reach various fixed values of the loss. Figure 19 The number of minimum serial steps needed to reach any fixed value of the test loss decreases precipitously with model size. Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to a very large one.

Figure 20 This figure provides information about the performance per token as a function of model size and training time. Left: Loss per token as a function of its position T in the 1024-token context. Loss scales predictably as a power-law in T. Right: Test loss per token as a function of training step. Figure 21 In addition to the averaged loss, individual tokens within the 1024-token context also improve smoothly as model size increases. Training runs with shorter context n_ctx = 8 (dashed lines) perform better on early tokens, since they can allocate all of their capacity to them. D.5 Context Dependence. The trends for loss as a function of model size are displayed for different tokens in the context in Figure 21. We see that models trained on n_ctx = 1024 show steady improvement with model size on all but the first token. Fixing model size, it appears that the loss scales as a power-law as a function of position T in the context, see Figure 20. This may be a consequence of underlying power-law correlations in language [EP94, ACDE12, LT16], or a more general feature of the model architecture and optimization. It provides some suggestion for the potential benefits (or lack thereof) from training on larger contexts. Not only do larger models converge to better performance at T = 1024, but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting patterns with less contextual information. In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step. The model begins by learning short-range information, and only learns longer-range correlations later in training. We have also included models trained with a tiny context n_ctx = 8 in order to compare with our longer context models. Even modestly sized models trained on n_ctx = 8 can dominate our largest n_ctx = 1024 models on very early tokens. This also suggests that further improvements should be possible with much larger models trained on large contexts. D.6 Learning Rate Schedules and Error Analysis. We experimented with a variety of learning rates and schedules. A host of schedules and resulting test performances for a small language model are plotted in Figure 22. We conclude that the choice of learning rate schedule is mostly irrelevant, as long as the total summed learning rate is sufficiently large, and the schedule includes a warmup period and a final decay to near-vanishing learning rate. Variations among schedules appear to be statistical noise, and provide a rough gauge for the scale of variation between different training runs. Experiments on larger models suggest that the variation in the final test loss between different random seeds is roughly constant in magnitude for different model sizes. We found that larger models require a smaller learning rate to prevent divergence, while smaller models can tolerate a larger learning rate. To implement this, the following rule of thumb was used for most runs: LR(N) ≈ 0.003239 + -0.0001395 log(N). We expect that this formula could be improved. There may be a dependence on network width, likely set by the initialization scale. The formula also breaks down for N > 10^10 parameters. Nevertheless, we found that it works sufficiently well for the models we considered.

Figure 22 We test a variety of learning rate schedules including cosine decay, linear decay, as well as other faster/slower decays schedules on a 3 million parameter model, shown on the left. For these experiments we do not decay to zero, since we find that this tends to give a fixed improvement close to the end of training. We find that, as long as the learning rate is not too small and does not decay too quickly, performance does not depend strongly on learning rate. Run-to-run variation is at the level of 0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level. Figure 23 The trend for performance as a function of parameter count, L(N), is fit better by a power law than by other functions such as a logarithm at a qualitative level. We experimented with a number of functional forms for the fits to L(N), L(C), and L(D); the power-law fits were qualitatively much more accurate than other functions such as logarithms (see Figure 23). For L(C), we do not include small models with only 1 layer in the fit, as the transition from 1 to 2 layers causes a noticable lump in the data. For L(N) we also do not include very small models with only 1 layer in the fit, and we exclude the largest models that have not trained fully to convergence. Fit parameters change marginally if we do include them, and the trend extrapolates well in both directions regardless. D.8 Generalization and Architecture. In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed. It seems to depend only on the performance on the training distribution.

Figure 24 We show evaluations on a series of datasets for models with approximately 1.5 Billion param-eters. We observe no effect of depth on generalization; generalization performance depends primarily on training distribution performance. The 12-layer model overfit the Internet Books dataset and we show the early-stopped performance; we have not seen this surprising result in other experiments. List of Figures. 1 Summary of simple power laws. 3. 2 Illustration of sample efficiency and compute efficiency. 4. 3 How to scale up model size, batch size, and serial steps. 4. 4 Performance when varying model and data size, or model and training steps, simultaneously. 5. 5 Weak dependence of performance on hyperparameter tuning. 8. 6 Comparison of performance trend when including or excluding embeddings. 8. 7 LSTM and Transformer performance comparison. 9. 8 Generalization to other test datasets. 10. 9 Universality of overfitting. 11. 10 Critical batch size. 12. 11 Performance versus compute budget or number of parameter updates. 14. 12 Training on suboptimal models. 15. 13 Comparison between empirical and adjusted compute trends. 15. 14 Optimal model size and serial number of steps versus compute budget. 16. 15 Contradiction between compute and data trends. 17. 16 Early stopping lower bound and training curves for overfit models. 23. 17 Universal transformers. 24. 18 Batch size scans. 24. 19 Another look at sample efficiency. 24. 20 Power-law dependence of performance on position in context. 25. 21 Performance at different context positions versus model size. 25. 22 Learning rate schedule scan. 26. 23 Comparison of Power-Law and Logarithmic Fits. 26. 24 Generalization versus depth. 27.

List of Tables. 1 Parameter and compute counts for Transformer. 7. 2 Fits to L(N, D). 11. 3 Fits to L(N, S). 14. 4 Key trend equations. 20. 5 Key parameters to trend fits. 20. 6 Trends for compute-efficient training. 20. References. [ACDE12] Eduardo G Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti. On the origin of long-range correlations in texts. Proceedings of the National Academy of Sciences, 109(29):11582–11587, 2012. 25. [AS17] Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv, 2017, 1710.03667. 11, 18, 22. [BB01] Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disam-biguation. In Proceedings of the 39th annual meeting on association for computational linguis-tics, pages 26-33. Association for Computational Linguistics, 2001. 18. [BHMM18] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv, 2018, 1812.11118. 18. [Bia12] GÊrard Biau. Analysis of a random forests model. Journal of Machine Learning Research, 13(Apr):1063-1095, 2012. 18. [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019, 1904.10509. URL http://arxiv.org/abs/1904.10509. 19. [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805. 2. [DGV+18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni-versal transformers. CoRR, abs/1807.03819, 2018, 1807.03819. URL http://arxiv.org/abs/1807.03819. 6, 9, 23, 24. [EP94] Werner Ebeling and Thorsten Pöschel. Entropy and long-range correlations in literary english. EPL (Europhysics Letters), 26(4):241, 1994. 25. [Fou] The Common Crawl Foundation. Common crawl. URL http://commoncrawl.org. 7. [GARD18] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. 2018, arXiv:1812.04754. 18. [GJS+19] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. arXiv, 2019, 1901.01608. 18. [GKX19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net op-timization via hessian eigenvalue density. CoRR, abs/1901.10159, 2019, 1901.10159. URL http://arxiv.org/abs/1901.10159. 18. [Goo01] Joshua Goodman. A bit of progress in language modeling. CoRR, cs.CL/0108005, 2001. URL http://arxiv.org/abs/cs.CL/0108005. 18. [GRK17] Scott Gray, Alec Radford, and Diederik P Kingma. Gpu kernels for block-sparse weights. ope-nai.com, 2017. 19. [HAD19] Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Compu-tational challenges in deep learning. In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming, PPoPP '19, pages 1-14, New York, NY, USA, 2019. ACM. doi:10.1145/3293883.3295710. 18.

[HCC+18] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, abs/1811.06965, 2018, 1811.06965. URL http://arxiv.org/abs/1811.06965. 19. [HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kia-ninejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is pre-dictable, empirically, 2017, 1712.00409. 18. [JGH18] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571-8580, 2018. 18. [KB14] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014, 1412.6980. 7. [Kom19] Aran Komatsuzaki. One epoch is all you need, 2019, arXiv:1906.06669. 18. [KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS'12, pages 1097–1105, USA, 2012. Curran Associates Inc. URL http://dl.acm.org/citation.cfm?id=2999134.2999257. 19. [LCG+19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2019, 1909.11942. 9. [LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretrain-ing approach. CoRR, abs/1907.11692, 2019, 1907.11692. URL http://arxiv.org/abs/1907.11692. 2. [LSP+18] Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198. URL http://arxiv.org/abs/1801.10198. 2, 6. [LT16] Henry W Lin and Max Tegmark. Criticality in formal languages and statistical physics. arXiv preprint arXiv:1606.06737, 2016. 25. [LXS+19] Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720. 18. [MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, 2018, arXiv:1812.06162. 3, 5, 6, 12, 13, 21. [Pap18] Vardan Papyan. The full spectrum of deep net hessians at scale: Dynamics with sample size. CoRR, abs/1811.07062, 2018, 1811.07062. URL http://arxiv.org/abs/1811.07062. 18. [RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. 2, 6. [RRBS19a] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, 1909.12673. 18. [RRBS19b] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, arXiv:1909.12673. 18. [RSR+19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019, arXiv:1910.10683. 2. [RWC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. openai.com, 2019. 2, 5, 6, 7, 8. [SCP+18] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanan-takool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers, 2018, 1811.02084. 19. [SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, 2015, 1508.07909. 6.

[SLA+18] Christopher J. Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training, 2018, arXiv:1811.03600. 12. [SS18] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235, 2018, 1804.04235. URL http://arxiv.org/abs/1804.04235. 7. [THK18] Stefan Thurner, Rudolf Hanel, and Peter Klimek. Introduction to the theory of complex systems. Oxford University Press, 2018. 18. [TL19] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.11946, 2019, 1905.11946. URL http://arxiv.org/abs/1905. 11946. 18. [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. 2, 6. [VWB16] Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks, 2016, arXiv:1605.06431. 8, 18. [Was06] Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006. 18. [WPN+19] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2019, 1905.00537. 2. [WRH17] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Growing a brain: Fine-tuning by in-creasing model capacity. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi:10.1109/cvpr.2017.323. 19. [WYL19] Wei Wen, Feng Yan, and Hai Li. Autogrow: Automatic layer growing in deep convolutional networks, 2019, 1906.02909. 19. [YDY+19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2019, arXiv:1906.08237. 2. [ZK16] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Procedings of the British Machine Vision Conference 2016, 2016. doi:10.5244/c.30.87. 18. [ZKZ+15] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor-ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi:10.1109/iccv.2015.11. 7. [ZLN+19] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, and Roger B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. CoRR, abs/1907.04164, 2019, 1907.04164. URL http://arxiv.org/abs/1907.04164. 12, 18.