Facial and Vocal Cues in Daily Life: A Multimodal Framework for Behavioral Insight
摘要
This study proposes a multimodal framework for detecting cheerful affective states by integrating facial expression analysis and vocal signal processing. Facial features were extracted using the Facial Action Coding System (FACS). Cheerfulness was defined from AU06 (cheek raiser), AU12 (lip corner puller), and AU25 (lips part), while confidence was derived from AU05, AU06, AU07, AU12, and AU23. Fatigue was operationalized through PERCLOS, using AU45 (eye closure) as an indicator of vigilance loss. To validate these AU-based indices, Mel-Frequency Cepstral Coefficients (MFCCs) were computed from speech and tested for convergence with the facial composites. Results showed that cheerfulness aligned with MFCC7, MFCC12, and MFCC6, confidence with MFCC7 and MFCC12, and fatigue with MFCC3, MFCC10, MFCC11, and MFCC6. These findings suggest that cheerfulness and confidence exhibit distinct multimodal signatures, whereas fatigue primarily manifests as a facial expression. The integration of AU and MFCC features thus strengthens the reliability and ecological validity of affective assessment.