Interpreting predictions from machine learning models in healthcare presents challenges due to limited alignment between interpretability approaches and end-user perspectives. This research addresses these issues by incorporating healthcare professionals’ perspectives into the design of explanations for an AI-based risk prediction model. Through a series of user studies involving healthcare professionals, feedback was gathered on visualization preferences, model transparency, and the usability of AI explanations. Participants emphasized the need for simple, clear visualizations and interactive features to improve understanding. The study employed both qualitative and quantitative methods, utilizing the UTAUT model to assess the AI’s acceptance, trust, and perceived risks. Results suggest that while user-centric explanations foster trust, they must be complemented by transparency, local validation, and data management to ensure effective AI integration in clinical workflows. This research underscores the importance of designing user-friendly AI tools to enhance decision-making and patient care outcomes in healthcare settings.