Research on electroencephalogram (EEG) based emotion recognition has yielded advanced progress with the development of deep learning techniques. However, understanding EEG-based emotion recognition tasks from the perspective of deep learning models remains in its early stages. Some studies have pointed out that mainstream datasets usually assign the same label to a large number of samples collected under the same stimulus, and these discrete labels cannot well describe the emotion dynamics, resulting in insufficient information obtained by the models. Therefore, several previous works have introduced continuous labels, demonstrating both the feasibility of learning these labels and their effectiveness in enhancing models’ understanding of emotion recognition tasks. Nevertheless, these works either fail to demonstrate the beneficial effects or make incomplete use of the continuous labels. So in this work, we propose a multi-task emotion recognition model, which combines two emotion recognition tasks. Additionally, considering the advanced methods in other fields and the neglected information from stimuli that contribute to forming EEG datasets, we utilizes a pre-trained model to extract image features from the stimuli to assist in the training process. Our proposed model fully leverages all available labels and stimuli, achieving superior performance in both tasks compared to single-task models. In the emotion classification task proposed, our multi-task model achieves an accuracy of 86.08%, surpassing the performance of single-task models which achieve 84.60% and 83.13% with and without the use of continuous labels, respectively. These findings not only highlight the performance enhancement enabled by the multi-task approach but also advocate for the integration of pre-trained models and advanced methodologies from related fields into research on affective brain-computer interfaces, aiming to enhance the understanding of the complex human brain.