Correct prediction of crop yields and the identification of the major factors that affect them are necessary to maximize agricultural productivity, maximize the use of resources, and increase food security in the face of increasing environmental and climatic uncertainties. Historical-based traditional models for predicting yields mostly do not address the intricacy of today's precision agriculture. This research puts forward an innovative AI-based framework incorporating sophisticated machine learning algorithms like CatBoost, XGBoost, Support Vector Regression (SVR), and deep learning, used with various datasets like high-resolution satellite and UAV images, weather conditions, and soil properties. To solve the problem of interpretability of the model, the authors have included explainable AI (XAI) methods, namely SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-Agnostic Explanations), to ensure transparent, reliable, and actionable insights. The outcomes reveal a 17.6 increase in prediction accuracy compared to conventional machine learning methodologies as well as a 22.4 enhancement in model generalization. Through determining the most critical factors impacting yield and yielding understandable outputs, this work enables farmers, agronomists, and policymakers to make informed and sure decisions. The suggested method presents a strong and scalable solution to sustainable and resilient agricultural planning and plays an important role in world food security endeavors.