Emotion recognition is a complex task, especially in the fusion of multimodal physiological signals. Effectively capturing the dynamic characteristics and cross-modal information of signals is a primary challenge. To address this, we propose an emotion recognition method based on a Cross-Modal Attention Transformer and Learning-Classification Adversarial Network (CAT-LCAN), which effectively integrates multiple physiological signals. Cross-subject experiments conducted on two publicly available datasets, DEAP and WESAD, show that CAT-LCAN significantly outperforms several state-of-the-art baseline models. This innovative approach offers new insights into cross-subject multimodal emotion recognition and holds substantial research and practical significance.