Neural Machine Translation by Jointly Learning to Align and Translate

Abstract

A new neural machine translation approach extends encoder-decoder models by allowing the network to search for relevant parts of the source sentence when predicting a target word, achieving state-of-the-art performance.

Introduction

Neural machine translation models, often using an encoder-decoder architecture, are jointly tuned to maximize translation performance, but suffer from a bottleneck when encoding long sentences into a fixed-length vector.

Background: Neural Machine Translation

Neural machine translation models aim to jointly train a single neural network for translation, typically using an encoder-decoder structure, which has shown promising results but struggles with long sentences due to fixed-length vector encoding.

Learning to Align and Translate

A novel neural machine translation architecture uses a bidirectional RNN encoder and a decoder that adaptively searches through source sentence annotations to generate context vectors for each target word, improving translation quality.

Experiment Settings

The proposed RNNsearch model and a conventional RNNencdec model were evaluated on English-to-French translation using WMT'14 corpora, trained with sentence lengths up to 30 and 50 words.

Results

The RNNsearch model significantly outperforms the RNNencdec model across sentence lengths and achieves performance comparable to conventional phrase-based systems, demonstrating robustness to sentence length.

Quantitative Results

The RNNsearch model consistently outperforms the RNNencdec model in BLEU scores, especially on longer sentences, and matches the performance of traditional phrase-based systems on sentences without unknown words.

Qualitative Analysis: Alignment

The proposed model's soft-alignment mechanism allows for intuitive inspection of word alignments between source and target sentences, capturing both monotonic and non-monotonic relationships, and naturally handling phrases of different lengths.

Qualitative Analysis: Long Sentences

The RNNsearch model demonstrates superior performance in translating long sentences compared to the RNNencdec model, accurately preserving the original meaning by selectively encoding relevant parts of the input.

Related Work

Previous research has explored learning alignments in handwriting synthesis and using neural networks as features in statistical machine translation, but this work proposes a novel end-to-end neural machine translation system.

Conclusion

The proposed RNNsearch architecture addresses the fixed-length vector bottleneck in neural machine translation, achieving state-of-the-art performance comparable to phrase-based systems and showing improved robustness for long sentences.

References

This section lists all cited works, including papers, theses, books, and conference proceedings, spanning areas like machine translation, recurrent neural networks, and deep learning.

Model Architecture Choices

The model uses a gated hidden unit for its RNN activation function, which is similar to LSTM and allows for better learning of long-term dependencies through specific computation paths.

Encoder and Decoder Architecture

The encoder uses a bidirectional RNN with shared word embeddings, while the decoder employs gated hidden units and a context vector computed by an alignment model to generate translated sentences.

Training Procedure and Model Details

The model is trained using SGD with Adadelta for adaptive learning rates, employing specific minibatch strategies and parameter initialization techniques for efficient training.

Long Sentence Translations

Comparative translations of long sentences demonstrate that the RNNsearch-50 model generally produces better translations than the RNNenc-50 and Google Translate.