Human activity detection plays a significance role in surveillance system, security, human–computer interaction applications, and drivers’ behaviors detection, which exhibits its robust and suitability as well as cost-effectiveness. Conventional machine learning approaches have made a crucial progress in objects detection tasks based on handcrafted properties and statistical methods. However, these methods are restricted by manual features design. Deep learning models have talked such limitations through automatic deep features extraction layers, nevertheless its detection accuracy required enormous and annotated samples for training process. To handle these issues, transfer learning techniques transfer the knowledge gained from solving one problem and applied to a different, but related, problem. In this chapter, an improved structure of ResNet101 deep learning model is presented with transfer learning technique to eliminate the consuming time of computations and obtaining higher detection accuracy of driver’s behavior in the input images. The experimental results demonstrate higher detection accuracy of various driver behaviors which achieves 97%. The obtained results could be helpful for the objective evaluation of driving adeptness and thus upgrading driving safety.