The significance of automatic defect detection in the photovoltaic (PV) industry has attracted a great attention in the last few years. This is due to the limitations of using the manual/visual inspection approaches and the growing production rates of PV modules. In this work, an automated method for identifying defects in PV modules using electroluminescence (EL) images is proposed. Two approaches were developed: the former utilized transfer-based learning, while the second employed a lightweight convolutional neural network architecture (L-CNN). These methods effectively identified defects in EL images and yielded promising performance when compared to existing research in the field. The L-CNN model exhibited a reduced amount of computational power and time and is capable of operating on a computer equipped with a single GPU. Furthermore, an online Data augmentation technique was implemented to improve model’s performance. Furthermore, the proposed methods utilized a proper class splitting of the PV cells that considered the characteristics of investigated mono-crystalline and poly-crystalline solar cell’s types. This refinement significantly enhanced the performance.