Reducing Aleatoric and Epistemic Uncertainty Through Multi-modal Data Acquisition
摘要
To generate accurate and reliable predictions, modern AI systems need to combine data from multiple modalities, such as text, images, audio, spreadsheets, and time series. However, collecting training and test data for many modalities is challenging and time-consuming, creating a need for cost-efficient multi-modal data acquisition. In this paper we advocate that this can be realized by disentangling epistemic and aleatoric uncertainty. It is commonly assumed in the machine learning community that epistemic uncertainty can be reduced by collecting more data, while aleatoric uncertainty is irreducible. We claim that this assumption can be challenged in modern multi-modal AI systems, and we introduces an innovative data acquisition framework where uncertainty disentanglement leads to actionable decisions, allowing cost-efficient sampling in two directions: sample size and data modality. The main hypothesis is that aleatoric uncertainty decreases as the number of modalities increases, while epistemic uncertainty decreases by collecting more observations. We provide a theoretical analysis and proof-of-concept implementations on various multi-modal datasets to prove the usefulness of our framework, which combines ideas from active learning, active feature acquisition and uncertainty quantification.