An Evolutionary Search-Based Operator Fusion Method with Binary Representation for Deep Learning Inference Acceleration
摘要
The deployment of deep neural network (DNN) models on various devices presents a significant challenge due to the diverse characteristics of deployment environments. Operator fusion, which enhances DNN model inference efficiency by combining multiple operators into a single one, is critical in this context. However, most deep learning compilers use a uniform fusion scheme, failing to consider environment-specific optimizations. We propose a novel approach to determine optimal operator fusion schemes tailored to different deployment environments. By representing fusion schemes with fixed-length binary sequences and using model inference time as feedback, we apply an evolutionary search strategy to identify effective fusion schemes. Our experiments with multiple DNN models across diverse environments show substantial performance gains over static fusion schemes used by TVM and DNNFusion. Specifically, we observed an average performance improvement of 17% on the NVIDIA Tesla V100 GPU, with a maximum improvement of 79%. On the NVIDIA Orin NX GPU, our method achieved an average improvement of 13%. On CPUs, the average performance increase was 25%, peaking at 44%. These results underscore the effectiveness of our method in enhancing DNN model inference across various deployment environments.