Lung cancer is a malignant tumor with the highest mutation rate in the world, among which non-small cell lung cancer (NSCLC) has a very high mutation rate. In recent years, medical research has found that tumor mutation burden (TMB) can predict treatment of cancer, immunotherapy and chemotherapy. However, the traditional method of calculating TMB using gene prediction technology has the disadvantages of high detection cost, long cycle, and intensive sample interrogation. To solve the above problems, we propose a hybrid deep learning model (FCGA-Former) to automatically predict TMB predictions, aiming to save pathologists’ time. In order to solve the problem of semantic gap in medical images, the factor space is applied to the semantic embedding space, so that the high-level semantic space is consistent with the underlying image feature space, and the data features are directly related to the information expressed by the images. The lung adenocarcinoma histopathology image dataset was taken from the TCGA database and included 271 high TMB data and 66 low TMB data. Experimental results show that the maximum average area under the curve (AUC) of this model is 98.1%. FCGA-Former is discussed in the area of ​​other models in terms of interpretability as it provides more accurate results. The results of this study are of great significance in guiding lymph node treatment of NSCLC. This lays the foundation for deploying automatic classification decision-making systems in clinical applications and using deep learning technology to predict TMB.