This paper presents optimized strategies for the MPI_Alltoallv collective communication operation, crucial for high-performance applications such as FFT solvers. Due to the dominance of GPU-enabled systems, we evaluated both CUDA-aware and Copy-to-CPU methods, using the MPI Advance framework to enhance scalability and efficiency. CUDA-aware methods allow MPI to optimize the path of communication, such as through direct GPU-to-GPU transfers with GPUDirect communication, whereas Copy-to-CPU algorithms incorporate persistent memory buffers to reduce allocation overhead, excelling with larger data sizes. The study’s structured analysis compares these algorithms on the Lassen and Tioga supercomputers under varying message sizes and process counts. Results reveal distinct performance benefits: on Lassen, manually copying to the CPU and initializing all messages with non-blocking sends and receives achieves up to a 3 \(\times \) speedup over CUDA-aware Spectrum MPI with large message sizes, outperforming CUDA-aware approaches in high process-count scenarios. However, on Tioga, performing a batched nonblocking exchange directly between GPUs demonstrates superior scalability up to 5 \(\times \) faster at large message sizes and higher process counts, maintaining lower execution times than Cray-MPICH Alltoallv. The paper also integrates these optimizations into real world applications such as HeFFTe, a state-of-the-art FFT solver, and tests with Beatnik, a mini-app that simulates 3D Raleigh-Taylor instabilities, achieving a 2–3 \(\times \) speedup in communication time due to optimized Alltoallv performance on Lassen. Overall, the study suggests a poly algorithm approach dynamically selecting between CUDA-aware and Copy-to-CPU methods based on data size and system configuration to maximize efficiency across diverse HPC environments.