Forecasting the Interplanetary Magnetic Field (IMF) $B_{z}$ component is a critical and persistent challenge in space weather, often termed the “ $B_{z}$ problem”. As Coronal Mass Ejections (CMEs) are the primary drivers of strong and sustained southward $B_{z}$ events, this study investigates a multi-modal deep-learning framework for CME event-driven IMF $B_{z}$ forecasts at 12-hour intervals up to a 96-hour lead time. We propose a novel attention-based architecture, $B_{z}$ 4SWx, to fuse CME kinematic parameters with their associated solar magnetograms. This model employs a dual-branch network enhanced by a Convolutional Block Attention Module (CBAM). To evaluate its effectiveness, we compared its performance against several baseline models, including a uni-modal MLP (numerics), a uni-modal CNN (images), and a naive concatenation-based fusion model. The $B_{z}$ 4SWx model achieved the best overall performance, yielding an MAE of 3.270 nT, an RMSE of 4.124 nT, and a bias of −2.61 nT, with timing precision competitive with other multi-modal approaches. Interpretability analysis confirmed that while magnetograms provide the dominant predictive signal, CBAM was critical for dynamically focusing the model on relevant solar active regions. We conclude that attention mechanisms provide a powerful and interpretable framework for CME event-based IMF $B_{z}$ forecasting, representing a significant step toward resolving the persistent $B_{z}$ problem.
