Deep Neural Networks (DNNs) are widely used across different AI fields, but their computational demands often limit edge device deployment. While domain specific accelerators (DSAs) have higher throughput, their invoking latency makes them less practical for latency-sensitive applications. By combining Single Instruction Multiple Data (SIMD) Extensions, zero invoking latency, CPUs deliver the speed and efficiency needed for small-batch DNN inference, making them a preferred choice for edge AI and real-time applications. However, performance side effects and increased design and verification complexity limit the throughput of SIMD architecture. In this paper, we propose LSCA, a low-latency scalable convolution accelerator based on Large-Small core architecture. The large core utilizes a high-performance processor for general-purpose tasks, while the small core incorporates a dedicated Convolution Acceleration Unit (CAU) for DNN computation. A novel accelerator interface framework, embedded within the large core’s pipeline, enables efficient collaboration between cores. Additionally, LSCA leverages Fast Direct Convolution (FDC) algorithm to minimize memory access and alleviate bandwidth pressure. Experimental results demonstrate that LSCA reduces invoking latency by 99.2% compared to NVIDIA GPU and achieves a 6.3× peak performance improvement over Intel AVX-512. This highlights LSCA's effectiveness for small-batch DNN inference tasks with low-latency requirement.