In the era of intelligence, visual image recognition technology still faces problems such as poor adaptability and insufficient interpretability of deep learning. This paper introduces an innovative color feature extraction framework, which effectively solves the limitations of existing technologies by integrating color theory and deep learning methods. It uses color space conversion technology to convert images to a perceptually uniform color space, constructs multidimensional color statistical features, and retains color distribution information. It designs a ResNet (Residual Neural Network) architecture combined with the SE (Squeeze-and-Excitation Attention Mechanism) attention mechanism, and enhances the interpretability of the model through feature visualization technology. Based on the dynamic feature fusion strategy of PPO (Proximal Policy Optimization) reinforcement learning, the adaptive weighted fusion of hand-designed features and deep learning features is realized. Each module adopts a phased training strategy, first independently optimized and then jointly fine-tuned, and finally forms an end-to-end color feature extraction framework. Experimental results show that the multi-scale feature extraction method based on Lab (CIE 1976 Lab* Color Space) color space can effectively capture the brightness and chromaticity information of the image, and the feature distribution of each channel is clearly identifiable. Grad-CAM visualization technology successfully reveals the model’s attention mechanism for key color areas, and the heat map shows that the network can accurately focus on the target color area. Through quantitative evaluation of the IoU indicator, it is confirmed that the features extracted by the proposed method are highly consistent with human visual cognition, providing reliable technical support for visual recognition tasks in complex scenes.