Agricultural pest outbreaks pose serious risks to crop productivity, demanding timely and accurate forecasting methods. This work introduces a novel multimodal hybrid framework that fuses spatial features extracted from Sentinel-2 imagery with meteorological time series data to predict olive fruit fly populations one week ahead. A convolutional neural network is employed to capture spatial dependencies from fused satellite images, while different machine learning algorithms including Lag-Llama, Random Forest and XGBoost, are integrated for predictive modeling. Experimental results indicate that the hybrid approach, particularly when using Lag-Llama, outperforms models based solely on individual data modalities. These findings highlight the potential of deep learning and data fusion techniques in enhancing pest management and improving forecasting accuracy in agricultural settings.