Construction sites are dynamic and unpredictable which often leads to various accidents. Current construction safety management, however, mostly relies on the experiences of safety teams, which is time-consuming and prone to error. Additionally, due to the dynamic nature of construction sites, safety inspection results will lack updated data as this inspection cannot be conducted every time. This study proposes an approach to identify and analyze construction site safety conditions automatically through the integration of computer vision and Building Information Modeling (BIM). This study utilizes the 3D semantic data from BIM and up-to-date information captured from CCTV as the data collector. By processing CCTV data, the site’s information can be obtained using computer vision in real time and stored in a database. Finally, BIM is utilized to visualize a hazard map to aid safety teams with visual information regarding hazardous work zones and workers safety status. Our developed system can provide updated information supplemented by 3D contextual data, thereby enabling proactive actions to prevent accidents.