The increasing reliance on photovoltaic systems (PVS) for sustainable energy production necessitates efficient monitoring methods to ensure optimal performance and timely fault detection. Infrared imaging has emerged as a vital tool for identifying anomalies, such as hotspots, cracking, and soiling, within solar panels. However, selecting an optimal classification approach remains a challenge due to the trade-offs between accuracy, computational efficiency, and real-time applicability. In this study, we propose a hybrid framework for defect detection in PVS using convolutional neural networks (CNNs) for feature extraction and Random Forest (RF), Decision Tree (DT), and LightGBM classifiers. Additionally, advanced CNN architecture (ResNet-50) is evaluated to benchmark his performance against traditional ML algorithms. The methodology is validated using an infrared image dataset captured via unmanned aerial vehicles (UAVs) equipped with midwave or longwave infrared cameras. Experimental results demonstrate that the hybrid framework achieves an optimal balance between accuracy and computational efficiency. The ResNet50-RF combination achieved a testing accuracy of 81% on the rotated dataset, outperforming traditional classifiers like RF, DT, and LightGBM in precision and recall, while ResNet showed superior accuracy (96.8%) at a higher computational cost. This comparative study provides valuable insights into the suitability of different ML and DL approaches for real-time fault diagnosis in PVS, highlighting the importance of balancing model complexity and deployment feasibility in industrial applications.