<p>Cross-modal localization, utilizing only cameras and prior light detection and ranging (LiDAR) point cloud maps, achieves high localization accuracy at a low cost. The integration of semantic information can significantly enhance the accuracy at the cost of heavy computational load on optimization and huge semantic annotation on LiDAR point cloud maps. In this paper, we propose the SDA-Loc, a semantic cross-modal localization system that solely relies on visual semantic information, making our approach more streamlined compared to existing methods. We design a semantic-driven alignment algorithm that leverages visual semantic labels to perform different types of iterative closest point, allowing the system to better exploit the structural information represented by object semantics, thereby achieving accurate localization without the additional burden of point cloud annotation. Coupled with a designed dynamic error rejection mechanism, our approach effectively achieves a balance between accuracy and speed. The experiments conducted on the KITTI dataset demonstrate the competitive localization performance of our approach. Moreover, the experiment on outdoor campus dataset confirms that the proposed system can effectively mitigate the drift in visual localization under challenging lighting conditions, and proves the robustness of SDA-Loc when using poor LiDAR point cloud maps. The runtime analysis also shows that SDA-Loc strikes an excellent balance between localization accuracy and computational efficiency.</p>