Unmasking Real vs. Fake Smiles Using EEG and Emotion-Aware Feature Fusion
摘要
Smiles do not always signify genuine positive feelings. Distinguishing real from fake smiles has become a critical research area, with existing methods such as facial expression analysis being susceptible to manipulation. This signals the need for manipulation-resistant modalities. Electroencephalography (EEG) is immune to such manipulations. Therefore, this study introduces a deep learning framework combining transfer learning and emotion-aware feature fusion to classify smile authenticity effectively. The proposed approach is built on the fact that smiles are interconnected with emotional states, which offer crucial contextual information for smile discrimination. The framework has three stages: first, an EEG-based emotion recognition model was developed using CNNs and topographical EEG mapping to capture spatiotemporal patterns associated with emotional states. Second, a smiles-specific CNN was designed to process EEG features tailored to smile classification. Third, transfer learning was employed to fine-tune the emotion recognition model for smile classification. To enhance feature integration, a cross-attention mechanism was applied, aligning emotion-specific and smile-specific features dynamically to capture cues of smile authenticity. The framework achieved an accuracy of 74.303%, significantly outperforming the baseline CNN (65.133%) and transfer learning without attention (71.494%). The correlation analysis between emotion states and smile classes revealed intriguing results: Happy emotions exhibited the highest correlation with real smiles (0.801), while disgust emotion had a strong positive correlation with fake smiles (0.698), suggesting that disgust is likely to manifest alongside fake smiles, which is potentially a form of masking or dissonance in social settings.