Provenance data captures and provides a detailed lineage of datasets, models, and analytical pipelines, ensuring end-to-end traceability. In the explainable AI (XAI) process, recent techniques, including LIME and Grad-CAM, provide local explanations for deep learning models. However, achieving trustworthy AI requires synergizing these local explanations with the global observability and traceability of entire XAI pipelines. Many factors introduce substantial complexity at both local and global levels, generating a vast combinatorial space for experimental tuning to improve model quality. Ensuring the reproducibility of these experiments is critical for maintaining AI performance amidst subtle variations and large-scale changes. In this paper, we propose a provenance network model for cloud-based XAI pipelines. The provenance graph data is stored in the graph database. We demonstrate our model’s implementation in cloud environments, validating its effectiveness in enhancing observability, traceability, and reproducibility. Our approach is tested across multiple cloud platforms and open-source models.