Software defect prediction (SDP) is crucial to producing superior software programs. It allows for early identification of defects so that timely action can be taken to correct them. Preliminary research in the field includes techniques based on machine learning that focus only on the simple code characteristics followed by deep learning (DL) models that also capture the semantics of the code. In order to solve SDP, practitioners have recently become interested in large language models (LLMs) because to their extensive usage. In the current study, we have proposed a model namely CodeBERT-DHG, that first extracts semantic information using CodeBERT, combines it with the simple code characteristics with an advanced merger mechanism, and collectively uses them to detect defects in the code. Experimental results of the proposed approach on multiple open-source datasets compared with the commonly used DL models signify its superiority. The findings of the work would be helpful in guiding future researchers and practitioners to best utilize the power of LLM in conjunction with the appropriate DL model for SDP.