Pavement damage classification presents significant challenges due to the small size of damage regions, complex backgrounds, and high visual similarity among damage types. MambaOut combines high modeling efficiency and low computational complexity with a hierarchical feature extraction architecture, making it suited for image classification tasks. In this study we propose a novel classification model named Damage Detail Fusion MambaOut (DDFM) built upon the MambaOut. DDFM introduces two key modules: Content-Aware Mixer Feature Interaction (CAMFI), which enhances feature extraction by focusing on damage relevant details through content-aware attention; and Damaged Information Fusion (DIF), which integrates spatial, frequency, and semantic cues using a variance guided fusion mechanism to improve fine-grained recognition. Experimental evaluations on multiple datasets, including CQU-BPMDD, CQU-BPDD, and Crack500-PDD, demonstrate that DDFM outperforms both general purpose and task specific baseline models in terms of accuracy, robustness, and generalization. Notably, on the CQU-BPMDD dataset, DDFM achieves a 3.16% improvement in accuracy and a 4.16% increase in AUC compared to the MambaOut baseline, highlighting its effectiveness in pavement damage classification task.