Enhancing localization and mapping accuracy in constrained environments like parking lots is critical for autonomous driving. This paper introduces a novel visual-inertial Simultaneous Localization and Mapping (SLAM) approach tailored for automated valet parking (AVP). By incorporating semantic information such as various objects and markings found in parking lots, our method significantly enhances the robustness and precision of the localization process. These semantic features provide essential information for the automated parking system to understand the structure and rules of the parking environment, enabling more accurate navigation and decision-making. We developed a hybrid algorithm that integrates traditional key-point feature-based localization with semantic feature-based localization. The evaluations conducted in the CARLA simulator demonstrated a 54% reduction in position error compared to state-of-the-art methods, achieving an average trajectory error of 0.19 m. These advancements are vital for improving AVP system and facilitating the broader adoption of autonomous parking solutions. Future research will focus on scaling the approach to various urban environments and addressing challenges presented by dynamic conditions.