There is an issue with the incorrect assessment of outcomes, which is a major component of the reform of physical education in colleges and universities, which is the intelligent evaluation system of physical education instruction. The intelligent evaluation system of collegiate physical education is flawed and illogical, and traditional PE programs cannot fix it. A multi-scale feature fusion network approach is therefore suggested in this study for the purpose of analyzing the intelligent evaluation system in relation to optimization and innovation. To start, the intelligent assessment system uses learning theory to assess teachers, and then, to cut down on interference, the indications are categorized according to the needs of the system. The learning theory then develops a plan for an intelligent assessment system, conducts a thorough analysis of the system, and applies it to the physical education of college students. MATLAB simulations demonstrate that, when tested against certain evaluation criteria, the multi-scale feature fusion network approach outperforms conventional Phys Ed programs for college students in terms of accuracy and evaluation feature extraction time.