<p>Driver action recognition (DAR) is an important task in Advanced Driver Assistance Systems (ADAS), which suffers difficulties including subtle differences from intra-class and inter-class, complex background, vagaries of light, etc. Most previous methods are semantic-based or skeleton-based methods; the semantic-based methods cannot pay effective attention to driver action related area, while the skeleton-based methods exclude some important driver action related semantic information. To overcome these problems, we propose a novel Semantic-and-Skeleton Attention Hybrid Network (SSAH-Net), with two main parts including Skeleton and Semantic based Hybrid Network (SSH-Net) and Skeleton based Soft Spatial Attention Module (SSSA-Module). The SSSA-Module can intensify detail information of hands and head areas and suppress global background information to improve the DAR performance. SSH-Net combines skeleton and semantic based models to explore spatial semantic features and temporal characteristics. In addition, we introduce adaptive histogram equalization and transfer learning to deal with night images with low illumination and small amount. Our evaluation shows that our approach achieves better results than other state-of-the-art systems for driver action recognition on the real-world 3MDAD dataset; the accuracy on daytime and nighttime data are 88.50% and 78.51%, respectively.</p>