A W3C PROV–Aligned Metamodel for Tracing End-to-End Provenance in ML Pipelines
摘要
Once specified and enacted, end-to-end machine learning (ML) pipelines, together with contextual information about how artifacts are consumed and produced during execution, constitute valuable assets that can be shared or published for reuse. Indeed, ML pipelines involve multiple repetitive data transformation steps before, during, and after model training, and evaluating alternative configurations at each stage requires continuous monitoring of the workflow, particularly to support informed model selection for deployment. Although existing monitoring and experiment-management solutions record metrics and configurations, these logs are typically expressed using ad hoc data models that capture only limited relationships between artifacts and pipeline steps, and lack support for tracing complete derivation paths from training data to deployed models. To overcome such limitation, we propose, in this paper, a unified, W3C PROV-compatible metamodel for capturing end-to-end provenance of ML pipelines. We demonstrate the applicability of the proposed metamodel through a pipeline for detecting challenging behaviours in children with autism spectrum disorder from wearable physiological signals, and we illustrate its usefulness through decision-support queries that relate model behaviour and evaluation metrics to upstream data transformations, configurations, and execution environments, enabling systematic model selection, debugging, and governance across the ML lifecycle.