In recent years, AI models based on deep learning have expanded their scales. To train models, we use multiple GPUs on the same computer, or even more GPUs over networks. Such attempts that speed up training time are known as distributed training. In the area of training time reduction, Data Parallel and Distributed Data Parallel methods have been proposed. In this method, mini-batches of data are distributed to multiple GPUs and trained in parallel. The commonality of these methods is that they aim to reduce training time by training different data in parallel on the GPUs. Distributed Data Parallel requires synchronizing the model’s gradients between GPUs during training iterations. Gradient synchronization is considered to have room for reducing training time, and it has been studied as gradient compression. We propose to combine efficient gradient compression using low-rank approximation and gradient quantization, and we evaluate the situation of narrow or wide bandwidth of GPUs connected. At wide bandwidth, we clarify that the low-rank approximation method does not show not to contribute to reducing training time, and we demonstrate the effectiveness of the gradient quantization. We find that low-rank approximation and gradient quantization have harmful effects on test accuracy of several percent, and there is a trade-off relationship with training speed.