GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

Abstract

GPipe is a library for scaling deep neural networks via micro-batch pipeline parallelism, enabling larger models by distributing them across accelerators.

Introduction to GPipe

GPipe enables scaling of neural networks beyond single-accelerator memory limits by partitioning models across multiple accelerators using pipeline parallelism and batch splitting.

GPipe Algorithm

GPipe partitions a neural network into cells on separate accelerators, using a batch-splitting pipeline algorithm with synchronous mini-batch gradient descent to train large models efficiently.

Performance Evaluation

GPipe significantly increases model capacity for image classification and machine translation by partitioning models across many accelerators, with negligible bubble overhead when micro-batches exceed partitions.

Throughput and Communication Analysis

GPipe achieves near-linear training throughput scaling with the number of devices for Transformer models and demonstrates low communication overhead, even on systems without high-speed interconnects.

Multilingual Machine Translation

GPipe enables training of massive Transformer models for massively multilingual machine translation, achieving better performance than bilingual models and demonstrating the benefits of increased model capacity.

Comparison with Other Model Parallelism Techniques

GPipe offers a balanced solution for model parallelism, achieving high hardware utilization and linear scaling with minimal communication overhead through its synchronous batch-splitting pipeline algorithm.

Conclusion

GPipe is a scalable, flexible, and reliable library for training giant neural networks using model parallelism, providing significant empirical results and addressing key scaling challenges.

References

This section lists numerous research papers and preprints related to deep learning, distributed training, and neural machine translation.