Defect detection plays an important role in the industrial production and manufacturing sectors, directly impacting product quality and production costs. Most defect detection methods relying on RGB images often encounter challenges such as similarity between foreground and background. RGB-D image that introduces the depth channel can combine RGB and depth information to improve the defect detection performance. However, most RGB-D defect detection methods usually suffer from lower-quality depth images and can not fully take advantage of the two modalities. To tackle this problem, we propose an Adaptive Dual Attention Fusion Network (ADAFNet). ADAFNet is a three-branch decoder-encoder structure that extracts RGB and depth features and synchronously fuses them via a dual-attention dual-modality fusion (DADMF) module. DADMF first carries out dual-modality fusion on the channel and spatial attention features, respectively. Then it makes dual-attention fusion on the RGB and depth features, respectively. Finally, DADMF combines the resulting RGB and depth features using an adaptive weighting factor. Experiments conducted on the NEU RSDDS-AUG, NJU2K, and NLPR datasets demonstrate that our method outperforms other state-of-the-art approaches.