<p>Listeners recover speakers’ emotional state by integrating information from emotional semantics and the prosody accompanying sentence production. However, little attention has been paid to how the processing of emotional semantics and prosody is mediated by emotional dimensions (arousal and valence) and whether this relationship is mirrored at the level of the autonomic nervous system. In the present study, we compared the effects of emotional semantics, emotional prosody and their combination on behavioural responses and skin conductance responses (SCRs). Native French listeners (<i>N</i> = 77) judged the arousal and valence of utterances conveying emotions through semantics and/or prosody, while their SCRs were recorded. Our behavioural findings showed that prosody was more effective than semantics in transmitting emotional information to the listeners. Regarding SCRs, we found a significant effect of angry prosody in female listeners only, independently of whether it was combined with emotional or neutral semantics. Our work provides evidence for the perceptual saliency of emotional prosody and suggests a sex-based differentiation in emotional speech processing at the autonomic level. Finally, our findings align with the idea of emotional speech processing as a multistage process involving various processing levels, some of which may be driven by the autonomic nervous system and mediated by specific factors such as the emotional dimensions of stimuli and the sex of listeners.</p>