Quantizing deep neural networks (DNNs) to ultra-low precision (<4 bits) with mixed-precision strategies—employing variable bit-widths across layers—effectively alleviates storage bottlenecks and accuracy degradation for on-device edge deployment. However, the low-power constraints of edge devices persistently limit the full exploitation of inference performance. To address this challenge, this paper proposes a Variable-Precision Fusion Unit (VPFU) leveraging bit-serial computation to enable dynamic ultra-low-precision multiply-accumulate operations through logical fusion of Basic Blocks, achieving high parallelism while maintaining low power consumption. Capitalizing on the intrinsic channel-level parallelism of convolutional layers, we further design a VPFU-based 2D array architecture that significantly enhances convolutional computation efficiency. We implement the VPFU in RTL and synthesize it at a 45 nm technology node. Using cycle-accurate simulations on selected convolutional layers from four representative models, we evaluate the performance of the VPFU array. Experimental results show that compared to Bit Fusion’s Fusion Unit, the proposed VPFU achieves a 33.6% area reduction and a 35.16% power reduction, while delivering an average 1.33 \(\times \) speedup. When the precision is reduced to 1-bit, the average speedup further increases to 2.14 \(\times \) .