Scaling Laws for Neural Language Models

Abstract

Language model performance follows power-law scaling with model size, dataset size, and compute, with larger models being more sample-efficient, suggesting optimal training involves very large models on modest data.

Table of Contents

This paper investigates the scaling laws of Transformer language models, focusing on the relationships between performance and model size, dataset size, compute, and training time.

Introduction

Language modeling performance scales smoothly and predictably with model size, dataset size, and compute, with optimal training requiring concurrent scaling of all three factors.

Summary of Scaling Laws

Test loss in Transformer language models follows predictable power-law relationships with non-embedding parameters, dataset size, and optimally allocated compute.

Scaling Laws and Optimal Allocation

Optimal compute-efficient training involves using very large models, a slightly increased dataset size, and large batch sizes, with model size being the primary driver of performance gains.

Notation and Training

Key notation for scaling laws is defined, and the background details the use of the WebText2 dataset and Transformer architecture for training and evaluation.

Empirical Results and Basic Power Laws

Transformer language model performance scales smoothly with non-embedding parameter count, dataset size, and compute, following power-law relationships.

Performance Independence and Generalization

Language model performance is largely independent of Transformer shape hyperparameters but strongly correlated with in-distribution validation loss, indicating robust generalization.

Overfitting and Dataset Size

Language model performance improves predictably with increasing dataset size, and overfitting is managed by scaling dataset size sublinearly with model size according to a specific equation.

Overfitting and Compute

Optimal test loss is governed by a combined scaling law dependent on both model and dataset size, providing guidance for managing overfitting when increasing model size.

Scaling Laws of Loss

Performance saturates with increasing model size for a fixed dataset size, indicating overfitting, and the extent of overfitting is primarily determined by the ratio of model size to dataset size.

Critical Batch Size and Scaling Laws

The critical batch size follows a power law in loss and is independent of model size, with dataset size growing sub-linearly with model size to prevent overfitting.

Training at Critical Batch Size

Training at the critical batch size optimizes the time/compute tradeoff by doubling the required steps and data examples processed, with the critical batch size being predictable from the gradient noise scale.

Loss Dependence on Model Size and Compute

The loss dependence on model size and training time in the infinite data limit is described by a power-law equation, and a lower bound for the early stopping step in data-limited training is derived.

Performance and Compute Budget

An optimal model size exists for a fixed compute budget, and training below the critical batch size follows a power-law relationship between loss and compute budget.

Optimal Performance Allocations

Optimal model size scales with compute budget as a power-law, and minimal training steps grow very slowly, indicating that increasing model size is the primary driver for scaling with optimal computation allocation.

Predictions from L(N, Smin)

The L(N, Smin) equation predicts the loss as a function of training compute and optimal model size, aligning with empirical observations of scaling laws.

Scaling Relationships and Contradictions

Optimal model size grows rapidly with compute, while optimization steps grow slowly, leading to a contradiction where data requirements for compute-efficient training do not match overfitting control needs.

Intersection Point and Interpretation

The intersection point of compute-efficient and data-limited loss curves may indicate maximal performance, potentially related to the entropy of natural language, after which performance may level off.

Related Work and Comparisons

This section reviews prior research on power-law scalings in various machine learning models and density estimation tasks, drawing parallels and distinctions with the current findings.

Discussion of Findings

Consistent scaling laws of language model loss with model size, dataset size, and compute are observed, suggesting diminishing returns with scale and potential applicability to other generative modeling tasks.

Conclusion and Future Work

The paper concludes that scaling laws provide a predictive framework for language models, suggesting larger models will continue to improve and be more sample-efficient, with potential applicability to other domains.

The Compute Budget

Optimal training parameters for a fixed compute budget are determined by setting the derivative of the loss with respect to the number of parameters to zero.

Efficient Training

Compute-efficient training should proceed to a fixed percentage above the converged loss, with performance showing a power-law dependence on compute budget.

Comparison to Inefficient Training

Compute-efficient training uses significantly less parameter updates, more parameters, and less compute to reach the same loss compared to typical researcher practices.

Suboptimal Model Sizes

Using model sizes within a certain range of the optimal size results in only a small increase in compute budget, with larger models training faster at the cost of increased compute.

Caveats

The proposed scaling laws lack theoretical understanding and systematic correction analysis, making it difficult to determine their applicability.

Figure 16 Analysis

Early stopping is characterized as a function of overfitting, and training curves for models on different dataset sub-samples are displayed, with potential confounds in compute scaling and hyperparameter tuning noted.

Supplemental Figures

This section describes figures related to early stopping, universal transformers, batch size measurements, and sample efficiency versus model size.

Figure 17-19 Analysis

Recurrent Transformers show trade-offs between parameter count and FLOPs compared to standard Transformers, and sample efficiency greatly improves with model size.

Figure 20-22 Analysis

Performance per token scales with model size and training step, with models trained on larger contexts showing steady improvement, and learning rate schedules having minimal impact.

Figure 23-24 Analysis

Power-law fits better describe performance trends than logarithmic fits, and generalization performance is primarily dependent on training distribution performance, not network depth.

List of Figures

This is a list of all figures included in the document, with their corresponding page numbers.

List of Tables and References

This section provides a list of tables and references cited in the document.

References Continued

This section continues the list of references cited in the document.

References Continued

This section continues the list of references cited in the document.