As modern software applications continue to expand in size and complexity, software reviews and rigorous testing become crucial for ensuring quality, particularly in the realm of software defect prediction (SDP). While machine-learning (ML) techniques applied to software metrics datasets are prevalent in software fault prediction, deep learning (DL) approaches remain underexplored and underutilized—particularly in the context of publicly available datasets. To investigate the potential of DL for defect prediction, this study applies four established DL techniques (stacked LSTM, DBN, DNN, and DNN+LSTM) alongside three cutting-edge, transformer-based models (TabPFN, TabNet, and FT-Transformer). To the best of the authors knowledge, this marks the first bench-marking of these transformer-based techniques for SDP. The objective of this study is the identification of defective instances using deep learning techniques as-well as comparing the performance of standard deep learning techniques with the transformer-based models. Seven models were evaluated across fourteen open-source software defect datasets from NASA’s MDP and PROMISE repositories. Furthermore, the Scott-Knott ESD test validated the effectiveness of the proposed methodology. The proposed models achieve significant improvements (16, 15, 14%) in AUC, F-Measure, and MCC, respectively, over prior DNN, LSTM, and DBN implementations and TabNet surpassed prior transformer implementations by 19% on similar datasets.