This paper introduces a novel adaptation of parallelization techniques designed for training large-scale deep learning models on multiple GPUs in a High-Performance Computing (HPC) environment. Traditional methods such as Data Parallelism face challenges due to high memory demands and synchronization issues. The proposed approach, called Cyclic Data Parallelism, revolutionizes micro-batch execution by implementing sequential processing with uniform delays. This innovative paradigm provides a consistent memory space for model activations and effectively distributes gradient communications throughout training despite slight delays in gradient updates. By integrating Model Parallelism, the method optimizes GPU utilization by efficiently sharing GPUs across micro-batches, thereby reducing the overall GPU requirement. To validate the effectiveness and adaptability of this approach, extensive experiments are conducted on benchmark datasets such as SVHN and Pascal VOC, demonstrating its robustness across diverse data domains and task complexities on multiple GPUs.