In the field of intelligent recruitment, automated resume matching and non-contact interviews have significantly improved the efficiency of companies in finding suitable candidates. This corresponds to the techniques of person-job matching and AI interviews. However, current person-job matching methods lack substantial data support, while AI interview methods struggle to integrate deep information from multimodal data and provide comprehensive evaluations of candidates’ responses. To address these challenges, we propose a multimodal data-driven person-job evaluation model, comprising two key stages: a person-job matching method based on graph attention and a multimodal AI interview method. Using a dual-perspective graph neural network approach, we accomplish the screening of candidates and positions. In the second stage, we conduct a comprehensive evaluation of candidates’ interview performance based on text, audio, and image modalities, providing a more objective, consistent, and efficient interview assessment method. Experimental results demonstrate that our person-job matching method surpasses current popular techniques and effectively transfers features to the next stage. In our multimodal AI interview method, we achieve accurate scoring of candidate responses, assessment of intonation stress levels, and inference of their Big Five personality traits, comprehensively evaluating candidates from multiple perspectives. This confirms the superiority and efficiency of our approach.