Emotion computing, a key research area in human computer interaction, has gained significant attention in recent years, particularly due to the potential of emotion recognition based on electroencephalogram (EEG) signals. Traditional convolutional neural networks (CNNs) for EEG analysis often struggle with low computational efficiency and channel dependency, which hinder the practical use and generalization of these models. To overcome these challenges, this study introduces a novel channel-temporal fusion attention convolutional neural network (CTFA-CNN). The proposed channel-temporal fusion module tackles computational redundancy and channel dependency by transforming tensor dimensions and sharing parameters. This greatly improves computational efficiency. Moreover, the module’s channel-independent feature removes the need for predefined brain region connections, enabling the model to learn dynamic associations across channels and time. Experimental results show that the model performs exceptionally well across five publicly available EEG datasets, achieving accuracy rates from 91.9% to 98.9% in binary classification tasks and from 93.0% to 98.2% in multi-class classification tasks. The channel-independent design enhances the model’s generalization ability, while the parameter sharing mechanism boosts efficiency, laying a strong foundation for future embedded deployment and advancing the practical application of affective computing technology.