Nowadays, we are able to see a large number of vehicles on the road. Thus, enhancing road safety becomes equally important. With the advancement in computer vision, it has become easy to deal with such issues. In the majority of cases, accidents occur due to the drowsiness and inattention of the driver. In order to detect inattention, a dataset is prepared that includes 72.59% self-collected images and 27.41% publicly available images. Self-captured images contain a variety of Indian vehicles with drivers using different facial props. Three pre-trained models (ResNet50, InceptionV3, and VGG16) are tested on this dataset. A customized CNN is developed, which analyzes facial features like tiredness, eye closure, yawning, head tilt, and inattention of the driver on the road. The custom CNN outperformed established pre-trained models by achieving a testing accuracy of 81.73% and an ROC AUC of 0.82, demonstrating very good performance in challenging scenarios with varied lighting, facial obstructions, and camera angles.