BEV object detection has advanced significantly recently. However, infrastructure perception focusing on roadside scenes is mainly underexplored, and most current detectors prefer to only utilize monocular camera, which hampers the perception in particular scenarios such as rain, high humidity, and uneven light conditions. To address this problem, we propose a distinctive and effective 3D object detection framework, dubbed BEVRoad. A lightweight spatial-channel adaptive fusion module (SCAFM) is designed for the impartial fusion of camera and LiDAR BEV features, greatly improving the representation capability of the model. Furthermore, to alleviate the blockage caused by the relative movement of objects under the road, we add a simple spatio-temporal network named TrajNet to perform temporal modeling on the BEV feature map and predict the target motion position, which achieves excellent performance improvements with slight computation cost compared to the single-frame baseline. Experimental results on DAIR-V2X demonstrate that BEVRoad achieves wonderful performance, including +11.09% for vehicle, +16.61% for pedestrian, and +6.64% for cyclist compared to BEVHeight.