Despite extensive research on side-channel attacks (SCAs) against lattice-based Key Encapsulation Mechanisms (KEMs), there has been limited attention to the portability of existing deep-learning-based SCA distinguisher, especially concerning the National Institute of Standards and Technology (NIST) KEM standard Kyber. Our work addresses the portability challenges that stem from the device and measurement variations in SCAs against Kyber. We focus on the plaintext checking oracle-based SCA against Kyber, a prominent method in the field. We propose the Ablated Multiple Leakage Point Model (Ablated-MLPM) approach to optimize deep learning models, enhancing intra-board (same device with different EM probe placement) and inter-board (different devices) portability while mitigating overfitting concerns. Our contributions include the first systematic analysis of portability issues in SCAs against Kyber, highlighting their negative impact on attack efficiency. Real-world implementations are conducted on an STM32F407G board with an ARM Cortex-M4 microcontroller, using code from the well-known open-source pqm4 library. The results demonstrate that our Ablated-MLPM approach achieves more than \(99\%\) accuracy in all datasets, significantly enhancing both intra-board and inter-board portability. Furthermore, we introduce a lightweight model, Ablated-MLPM-LW, reducing the training parameters by \(79.63\%\) at the cost of requiring more queries.