Characterizing Trainability, Expressivity and Generalization of Neural Architecture with Metrics from Neural Tangent Kernel
摘要
Zero-shot neural architecture search aims to predict multiple characteristics of neural architectures using proxy indicators without actual training, however, most methods focus on evaluating only a single characteristic of neural networks. We propose two novel metrics based on NTK, which evaluates trainability using the ratio of the partial sum of the larger NTK eigenvalues to the total and measures expressivity by analyzing output variations for similar inputs through NTK kernel regression. Based on these two metrics for trainability and expressivity, along with a generalization metric, we define the NTK-score for comprehensive evaluation of architectural characteristics in neural architecture search. Moreover, to exploit three metrics of NTK-score, we employ a Weighted Borda Count approach on NTK-score to rank architectures in neural architecture search. Compared with state-of-the-art proxies, experimental results demonstrate that the NTK-score correlates well with both the test accuracy and training time of architectures, and outperforms comparison proxies across various search spaces and methods, including NAS-bench-201 and DARTS, as well as pruning, reinforce, and evolutionary algorithm.