Zero-shot neural architecture search aims to predict multiple characteristics of neural architectures using proxy indicators without actual training, however, most methods focus on evaluating only a single characteristic of neural networks. We propose two novel metrics based on NTK, which evaluates trainability using the ratio of the partial sum of the larger NTK eigenvalues to the total and measures expressivity by analyzing output variations for similar inputs through NTK kernel regression. Based on these two metrics for trainability and expressivity, along with a generalization metric, we define the NTK-score for comprehensive evaluation of architectural characteristics in neural architecture search. Moreover, to exploit three metrics of NTK-score, we employ a Weighted Borda Count approach on NTK-score to rank architectures in neural architecture search. Compared with state-of-the-art proxies, experimental results demonstrate that the NTK-score correlates well with both the test accuracy and training time of architectures, and outperforms comparison proxies across various search spaces and methods, including NAS-bench-201 and DARTS, as well as pruning, reinforce, and evolutionary algorithm.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Characterizing Trainability, Expressivity and Generalization of Neural Architecture with Metrics from Neural Tangent Kernel

  • Xinyu Chen,
  • Jinlong Li,
  • Faiza Amin,
  • Huanhuan Chen

摘要

Zero-shot neural architecture search aims to predict multiple characteristics of neural architectures using proxy indicators without actual training, however, most methods focus on evaluating only a single characteristic of neural networks. We propose two novel metrics based on NTK, which evaluates trainability using the ratio of the partial sum of the larger NTK eigenvalues to the total and measures expressivity by analyzing output variations for similar inputs through NTK kernel regression. Based on these two metrics for trainability and expressivity, along with a generalization metric, we define the NTK-score for comprehensive evaluation of architectural characteristics in neural architecture search. Moreover, to exploit three metrics of NTK-score, we employ a Weighted Borda Count approach on NTK-score to rank architectures in neural architecture search. Compared with state-of-the-art proxies, experimental results demonstrate that the NTK-score correlates well with both the test accuracy and training time of architectures, and outperforms comparison proxies across various search spaces and methods, including NAS-bench-201 and DARTS, as well as pruning, reinforce, and evolutionary algorithm.