Large language models have become essential for text-processing applications, yet their inference remains time-consuming, as tokens must be generated sequentially. While model sparsification techniques, such as Mixture of Experts (MoE) models, help mitigate computational costs, they introduce new challenges. In MoE models, only a subset of experts is activated at each layer. Note that not all expert subsets (typically pairs) are selected with equal probability. When experts are mapped across multiple GPUs, load imbalances can arise if frequently co-activated experts are assigned to the same GPU, leading to load unbalance and inefficient execution. This paper proposes a strategy that leverages the non-uniform selection of expert subsets to optimize their mapping across GPUs. By ensuring that frequently co-activated experts are placed on separate GPUs, we improve parallel processing and reduce inference time. Despite the NP-completeness of this mapping problem, we introduce simple yet effective greedy strategies that significantly minimize sequential expert processing. Our proof-of-concept experiments on the Mixtral model demonstrate the effectiveness of our approach for reducing inference time.