Fuzzing is an effective method for detecting bugs in deep learning (DL) libraries, which is critical for downstream applications. Large Language Models (LLMs) offer new opportunities to improve fuzzing. However, current LLM-based fuzzing methods face challenges in generating complex and error-prone API sequences for DL libraries. We introduce KL-RAG, a lightweight and versatile fuzzing framework for DL libraries, designed to overcome these challenges. Our key insight is that various DL libraries share commonalities, and numerous GitHub repositories contain extensive bug reports. These reports can guide the generation of API sequences for other libraries, improving their effectiveness. KL-RAG integrates knowledge-based generation and retrieval, collecting bug information from existing libraries to build a vulnerability knowledge base. When generating test code for other APIs, KL-RAG retrieves relevant vulnerability knowledge from the knowledge base and incorporates it into the prompt as context, helping LLMs generate high-quality, error-prone API sequences. We evaluate KL-RAG’s effectiveness through experiments conducted on three DL libraries. We have detected 23 bugs in the PaddlePaddle library, including 10 previously unknown bugs. The evaluation results show that KL-RAG significantly outperforms existing fuzzing methods in bug detection, code generation effectiveness, and API coverage.