Agricultural forecasting faces complex interactions between crop traits, environmental conditions, and shifting consumption patterns, requiring adaptive predictive models. In this paper, we propose AFFAC (Advanced Forecasting Framework for Agricultural Crops), a data-driven framework designed to enhance production forecasts, with a focus on olive cultivation a cornerstone of Mediterranean agriculture. AFFAC integrates tailored parameters for olive groves and leverages deep learning architectures (LSTM, CNN-LSTM, GRU, and Transformer) to improve decision-making. Experimental results demonstrate an 18% reduction in prediction error versus ARIMA (RMSE=7.4 vs. 12.5), with sensitivity analysis identifying 17 key parameters (95% cumulative importance). The framework excels in modeling non-linear patterns (MAE < 0.5 for critical months) and offers a modular design for customizable model integration, providing actionable insights for sustainable farming practices such as dynamic yield prediction under drought scenarios, aiding policymakers in risk management.