Estimating a person’s affective state through physiological signals offers non-invasive solutions for emotion-aware applications in daily life. Traditional approaches in this domain usually rely on hand-crafted features derived from physiological signals often limiting generalizability and scalability across users and contexts. This work presents a new hybrid deep and hand-crafted feature extraction framework, called DeepPhysioNet, designed to process univariate physiological signals collected via wearable sensors. This architecture combines deep representations learned through a one-dimensional convolutional neural network with a compact set of interpretable hand-crafted physiological metrics. Its dual-stream representation is used to train ensemble regression models for continuous estimation of arousal and valence dimensions. The proposed pipeline is validated on well known datasets using a leave-one-subject-out cross-validation strategy, simulating deployment on unseen users. Results demonstrate that DeepPhysioNet achieves superior performance compared to prior handcrafted methods, with the best models reaching an RMSE of 0.089 for arousal (Random Forest) and 0.053 for valence (Adaptive Boosting). These outcomes confirm the relevance of combining deep and domain-informed features for robust multimodal emotion recognition based on physiological signals opening the way to its applications in domains like, e.g., socially assistive robotics where emotion recognition plays a crucial role.