This paper proposes an evaluation algorithm of college teachers’ teaching ability based on deep learning, aiming to improve the objectivity and accuracy of the evaluation. The study first constructed an indicator system covering teaching process, teaching effect and teaching innovation, and designed an evaluation model integrating convolutional neural network and long short-term memory network, and optimized feature fusion through attention mechanism. The experimental results show that the model has excellent prediction performance under the complete indicator system, with a mean square error as low as 0.082 and a determination coefficient of 0.892, especially showing high accuracy in the group of high-ability teachers. Through visual analysis and performance comparison, the necessity of multidimensional indicator system and the advantages of deep learning methods in processing complex teaching data are verified, providing scientific technical support for the management of college teaching quality.