Vehicle classification is one of the essential tasks in intelligent transportation, and the toll management system requires this compulsory for auditing purposes and various other analyses. This research aims to classify vehicles using a Convolutional Neural Network (CNN) with limited convolutional layers and medium-size training datasets with 12899 images in day and night and with front, side, rear, and crop views. The classification is done for five toll-paying vehicles: Cars, LCVs, Buses and trucks, MAVs, and OSVs, and three exempted from toll vehicles: Motorcycles, three-wheelers, and Tractors. With these eight diversified classes of vehicles, the experiment results showed a remarkable vehicle classification precision of 92%. The analysis of the classification precision shows that in the toll-paying category, the precision of the car/jeep/van class is highest, and the precision of Bus-Truck is the lowest among all the classes, while in non-toll, the paying category precision of the motorcycle is highest and the precision of the three-wheeler class is lowest.