The utilization of cognitive radio (CR) in vehicular ad hoc networks (VANETs), known as CR-VANET, helps to mitigate communication spectrum shortage inside the network. Optimal spectrum usage is necessary for VANET connections to enable applications that encompass entertainment, traffic management, vehicle security, etc. Therefore, a vehicle has to learn about the surrounding radio environment as fast as possible to get access to any vacant spectrum if there is no allocated spectrum available. Machine learning (ML) has emerged as a crucial technique for managing such circumstances. RL, a form of ML, is best for CR-VANET since it requires no environment model or training dataset. In spectrum sensing approaches, the combination of CR-VANET and Q-learning, a form of reinforcement learning, (CR-VANET-QL) is shown to be more efficient than the CR-VANET algorithm. In this paper, a simulation-based comparison of CR-VANET and CR-VANET-QL is performed in terms of spectrum usage for 20, 40, 60, 80, and 100 vehicles. Based on spectrum utilization during spectrum sensing, the simulated results show that CR-VANET-QL outperformed CR-VANET, increasing from 88 to 96.48% over 20 to 100 vehicles. Furthermore, an analysis is conducted on the average reward received for each channel.