Understanding cross-model perceptual invariances through ensemble metamers
摘要
Understanding the perceptual invariances of artificial neural networks (ANNs) is essential for improving both model explainability and the alignment of machine vision with human perception. A compelling method for investigating these invariances is the use of metamers–stimuli that, although physically distinct, evoke identical neural responses in a network. This counterintuitive phenomenon provides a unique lens through which we can probe the inner workings of neural networks, revealing how they perceive and process visual information in ways that might not align with human vision. Despite their potential, existing methods for generating and analyzing metamers often focus on individual networks in isolation, limiting our understanding of how architectural biases influence the perceptual invariances across different models. In this work, we propose a novel method for generating metamers by leveraging ensembles of artificial neural networks. Our approach captures shared representational subspaces across various architectures, including convolutional neural networks (CNNs) and vision transformers (ViTs). To characterize the properties of the generated metamers, we employ a suite of image-based metrics that assess factors such as semantic fidelity and naturalness. Our results show that CNNs tend to generate more recognizable and human-like metamers, while ViTs, although producing realistic images, generate metamers that are less transferable across different settings. These findings underscore the impact of architectural design on the perceptual invariances of deep neural networks.