Accurate early diagnosis of colorectal cancer (CRC) is crucial for patient survival. However, multi-modal analysis using Narrow Band Imaging (NBI) and Chromoendoscopy Imaging (CI) is often challenged by data scarcity and ineffective information fusion. In this paper, a novel unsupervised data augmentation method, called Pathology Information-Guided Diffusion Model (PIG-Diff), was proposed to address the issue of CRC data scarcity. In PIG-Diff, a General Segmentation Large Model (GSLM) is introduced to generate pathological segmentation masks, which are then combined with text prompts as dual priors to guide diffusion models in generating high-quality images. Moreover, to improve the classification accuracy of CRC, a Dual-branch Residual Wavelet Attention Network (DB-ResWANet) was proposed, which integrates a Wavelet Attention Module (WaveletAtt) for enhancing NBI feature details and a Spatial Feature Alignment Module (SFAM) for adaptive NBI-CI feature fusion. Furthermore, an auxiliary modality classification task is employed to enhance modality-aware representations. Experimental results on the CRC magnifying endoscopy dataset indicate that the image generation quality of the PIG-Diff surpasses that of other existing image generation methods. The DB-ResWANet method, which was trained with PIG-Diff augmented data also outperformed various representative medical image classification methods. Our code and dataset is available at https://github.com/helinrui/DB-ResWANet .