Distributed Training with Data Parallelism Method in TensorFlow 2

Abstract

Deep learning is becoming more and more popular, especially well suited for large data sets. Besides, dee learning network training also requires vast computing power. Taking advantage of the power of GPU or TPU can partly solve the massive computing of deep learning. However, training an extensive neural network like Resner 152 on an ImageNet database of about 14 million image is not easy. That's why in this article, we are talking about not only leveraging the power of one GPU but also leveraging the power of multiple GPUs to reduce the training time of complex models by data parallelism method with two approaches Multi-worker Training and Parameter Server Training on two datasets flower and 30VNFoods.