In order to solve the problems of insufficient accuracy in evaluating the protective effect of movable guardrails on urban viaducts under dynamic traffic impact and low efficiency in simulating complex collision scenarios, this study proposed a hybrid deep learning framework. This paper designs a guardrail structure representation module based on SE(3)-Equivariant Graph Neural Network (SE3-GNN) to encode point cloud data with rigid transformation invariance and combine it with a physical constraint loss function to ensure momentum conservation. At the same time, a spatial–temporal convolutional ResNet50 (Spatial–Temporal Convolutional ResNet50) architecture is used to extract the spatial–temporal characteristics of the collision process. An adversarial training strategy is further introduced to generate synthetic data of extreme collision scenes through Wasserstein GAN (Generative Adversarial Network) to enhance the model’s generalization ability for long-tail distributions. The proposed hybrid framework has a MAE of 0.11 mm in predicting the maximum deflection of the guardrail, and the proposed framework outperforms the finite element method in terms of reasoning efficiency. The SHAP value analysis reveals that the collision angle and bolt preload are the key influencing factors. In the extreme scenario generalization ability test, the adversarial training strategy reduces the prediction error of the model by 42% in the 70 km/h side impact scenario. This study provides an efficient and accurate method for the safety assessment of movable guardrails on urban viaducts.