This paper presents an efficient early-out branch (EOBranch) integrated into YOLO-based object detection architectures to address the computational challenges inherent in edge-based traffic surveillance applications. By enabling the early exit for background images, the proposed EOBranch significantly reduces processing time and computational load without sacrificing detection accuracy. We evaluated our approach in YOLOv6 and YOLOv9, through a series of experiments that examine training strategies, optimal branch placement, and extended branch architectures. Experimental results reveal that, while a finetuning strategy delivers high early-exit performance, optimal branch placement is critical. Deeper placements significantly improved average precision, but are computationally more expensive. Placing an extended EOBranch earlier in the backbone achieved early-exit APs of 97.4% and 98.4% for YOLOv6 and YOLOv9, respectively. In particular, these optimized configurations led to reductions of processing time up to 46% for 24 h of traffic scene processing, highlighting the potential for substantial energy savings and enhancing the efficiency of traffic surveillance systems.