User Identity Linkage (UIL), which aims to connect individual identities across different platforms, has garnered increasing attention in recent years. Existing approaches mainly explore diverse types of user information (e.g., profiles, user-generated contents, and social connections) from social media platforms to tackle the task. Despite the great efforts made by these methods, their performances are still limited due to the following problems: 1) the neglect of video modality information; 2) the insufficient exploration of inter-modality information. To address the problems effectively, we propose a novel Hybrid Multi-modal Model (HMM) for the UIL task. Specifically, we first extract semantic features from the multi-modal posts of users. Then, a tailored attention mechanism is applied to adaptively measure the significances of these features. Next, we extract structural features from social connections, and unify them with semantic features into a hybrid network for joint learning. Finally, an adversarial learning strategy is adopted to enhance the performance of user identity linkage. Empirically, we evaluate the proposed model HMM on two real-world datasets, and the experimental results demonstrate its superiority.