Batched data layout optimization for Im2col-based convolutions on CPUs
摘要
Im2col-based convolution is a widely used technique in deep learning frameworks to efficiently implement convolution operators. However, in batched input scenarios, it suffers from inefficient weight reuse due to large reuse distances, often becoming a performance bottleneck. Particularly for CPUs with cache, large reuse distances of convolution weight data lead to increased memory access latency and degraded performance. To address this issue, we propose a novel batched-data-layout-optimization (BDLO) method. BDLO optimizes the input matrix layouts during the Im2col phase to reduce reuse distances of weight data, thereby enhancing memory access efficiency in batched Im2col-based convolutions. This method improves performance of batched convolutions on CPUs and is available for both NCHW and NHWC data layouts. Experimental results demonstrate that BDLO outperforms the standard batched Im2col-based convolutions implemented in the widely used open-source deep learning framework LibTorch, while the weight matrix size is larger than the input matrix size. Specifically, BDLO achieves up to 1.40