As the size of AI models grows dramatically to achieve higher accuracy, more and more models are trained on supercomputers with massive computing and memory. However, it is still challenging to accelerate deep learning applications on supercomputer systems with many-core architectures. In this paper, we propose swDarknet: a Darknet-based heterogeneous parallel deep learning framework on the SW26010 Pro processor. First, We designed three heterogeneous parallel DNN layers: adaptive block-splitting strategy for matrix-multiplication layer, optimized memory access for convolutional layer, and block-based parallel pooling layer. Second, we proposed a model averaging (MA) method to efficiently scale swDarknet across many processors. Finally, we evaluated the performance of swDarknet on a core group. As compared with darknet, the convolutional layers achieved an average speedup of 32x. When training AlexNet and VGG-16, swDarknet can achieve up to 36.79 \(\times \) and 181.35 \(\times \) speedup, respectively.