The gear tooth surface roughness has a significant impact on the transmission performance, fatigue strength, and service life of the gear pair. At present, the tooth surface roughness detection method based on deep learning has shortcomings in the unified expression of global and local features on the gear surface image. To address this problem, this paper proposes a gear tooth surface roughness detection method with multi-level deep feature fusion, which uses the long-distance dependency of Transformer and the ResNet model combined with the CBAM attention mechanism. This method uses the dual-branch parallel DeiT-S and ResNet-18 models as the backbone network, and adds a feature fusion mechanism to the middle layer and output layer of the branch, which effectively strengthens the joint learning of the global and local deep features of the image. The model in this paper is pre-trained on the surface roughness dataset of plane milling workpieces, and then migrated to the gear tooth surface roughness dataset based on the clustering optimal transport (COT). The results show that the test accuracy of the model on the dataset is as high as 99.47%, which provides a new method for tooth surface roughness measurement.