Attention Is All You Need

Title

The Transformer is a new network architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely, achieving superior quality and faster training for machine translation.

Introduction

The Transformer model eschews recurrence and convolutions, relying entirely on attention mechanisms to draw global dependencies, enabling significantly more parallelization and achieving state-of-the-art translation quality with faster training.

Model Architecture

The Transformer model employs stacked self-attention and position-wise feed-forward layers in both its encoder and decoder, utilizing residual connections and layer normalization for improved performance.

Attention Mechanisms

Scaled Dot-Product Attention computes output as a weighted sum of values based on query-key compatibility, while Multi-Head Attention projects queries, keys, and values multiple times and runs attention in parallel to jointly attend to information from different representation subspaces.

Model Applications

Multi-head attention is applied in three ways: encoder-decoder attention allows the decoder to attend to the input sequence, encoder self-attention allows positions to attend to previous encoder layers, and decoder self-attention allows positions to attend to preceding decoder positions while preventing leftward information flow.

Positional Encoding

Since the Transformer lacks recurrence and convolution, positional encodings are added to input embeddings to inject information about token order, using sine and cosine functions of different frequencies which allow the model to easily learn to attend by relative positions.

Training

The Transformer models were trained on standard machine translation datasets using the Adam optimizer with a specific learning rate schedule, and employed residual dropout and label smoothing for regularization.

Results

The Transformer significantly outperforms previous state-of-the-art models on English-to-German and English-to-French translation tasks, achieving better BLEU scores at a fraction of the training cost.

Model Variations

Experiments with Transformer architecture variations indicate that the number of attention heads and key dimensions are important, larger models generally perform better, and dropout is crucial for preventing overfitting, while sinusoidal and learned positional embeddings yield similar results.

Conclusion

The Transformer, an attention-based sequence transduction model, achieves state-of-the-art results in machine translation and generalizes well to other tasks like constituency parsing, with plans to extend its application to multi-modal data and explore local attention mechanisms.

References

This section lists numerous academic papers and resources related to neural networks, machine translation, and sequence modeling.

References

This section continues the list of academic references, covering topics such as natural language processing, deep learning, and attention mechanisms.

Figure 3: Attention Mechanism for Long-Distance Dependencies

This figure illustrates how multiple attention heads in layer 5 of the encoder self-attention mechanism focus on distant dependencies to complete phrases, specifically around the word 'making'.

Figure 4: Attention Heads for Anaphora Resolution

This figure displays two attention heads in layer 5 of the encoder self-attention, which are involved in anaphora resolution, showing sharp attentions for the word 'its'.

Figure 5: Attention Heads Exhibiting Sentence Structure Behavior

This figure presents two examples from layer 5 of the encoder self-attention, demonstrating that attention heads learn to perform different tasks related to sentence structure.