Deploying deep neural networks for small object detection in resource-constrained scenarios necessitates lightweight architectures capable of capturing both local and global dependencies. While State Space Models (SSMs) like Mamba achieve a favorable trade-off between accuracy and efficiency via linear-complexity designs, their core module’s emphasis on global modeling often overlooks fine-grained local details, limiting their performance in small object detection. To address this, we propose EVMDet, a lightweight object detection framework tailored for small objects in complex scenes. Our Method introduces two training-only modules that incur no inference overhead: Historical-Aware Dynamic Filtering (HADF), which stabilizes feature selection through dynamic Top-K and cross-batch historical fusion; and Spatial-Aware Feature Enhancement (SAFE), which enhances fine-grained local semantics via multi-dimensional feature perception and dual-branch embedding. EVMDet achieves state-of-the-art (SOTA) performance in small object detection, running approximately 1.47 \(\times \) faster than the second-best model and consuming 1.18 \(\times \) less GPU memory. Experimental results on SODA-D and VisDrone2019 demonstrate its strong adaptability to high-resolution inputs and real-world deployment scenarios, outperforming prior methods in accuracy, throughput, and resource efficiency.