Accurate apple yield prediction is critical for sustainable orchard management and food security, especially under the growing threat of climate variability. This study presents a comprehensive framework that integrates multisource time-series data with explainable artificial intelligence techniques to enhance apple yield forecasting. We collect and align daily weather data, extreme weather records, and annual apple yield data, focusing on major apple-producing regions in the United States. Sequential machine learning models are employed to capture temporal dependencies in climate and yield patterns. To address the interpretability challenges of complex models, we apply SHapley Additive exPlanations (SHAP) to provide post-hoc insights into feature contributions, further aligned with key apple phenological stages. Importantly, the generated explanations exhibit strong consistency with established domain knowledge, confirming the biological relevance of key climate-yield interactions. Our results demonstrate that LSTM with an attention mechanism achieves the highest predictive accuracy (86.64%) across all regions, while SHAP-based interpretations reveal the dynamic influence of climate factors at different growth stages. This study highlights the importance of integrating XAI into agricultural modeling, enabling stakeholders to make informed decisions based on both accurate predictions and transparent, domain-aligned explanations.