Distracted driving is a pervasive problem that affects drivers of all ages and experience levels. Every year, a lot of humans lost their lives or suffered injuries due to road accidents. Distracted driving has emerged as one of the most significant causes of such accidents, and the problem is only getting worse. Hence, there is a necessity to develop a model that can categorize the behavior of driving in terms of texting while driving, eating behind the wheel, or adjusting the radio, any action that takes a driver’s attention away which can cause severe consequences. This inspired the development of proposed distraction detection system which can be used in automobiles as a real time application. This research aims to build and improve deep learning models and apply feature extractors to predict the distracted behaviors of drivers with the images taken by the vehicle’s dashboard camera. A dataset of 2D dashboard images is selected and categorized into different classes based on the driver’s behavior. An algorithm is then developed by preprocessing, classifying, and testing using different deep learning techniques such as, Convolutional Neural Network (CNN) and VGG16. From the experimental analysis, it is observed that the combination of Haralicks, Linear Binary Patterns (LBP), and Speeded-up Robust Features (SURF) based features with the modified VGG16 model outperformed the other techniques with 99.7% accuracy.