The rapidly evolving biometric recognition technology has been widely applied in the detection and diagnosis of biological diseases. However, in the poultry domain, the scarcity of case information and the difficulty in data collection result in lower accuracy in disease diagnosis and treatment. Even though multi-modal large language models possess remarkable capabilities, they cannot be fully relied upon due to the hallucination problem. To address these issues, we propose PoultryAssistant, a novel approach that unifies LLMs with knowledge graphs to enhance diagnostic accuracy in poultry disease diagnosis. Specially, a poultry disease knowledge graph is constructed by utilizing entity-relation extraction algorithms. Then a strategy combining large language models and knowledge graphs is proposed to achieve diagnostic results through multi-source information interaction. Furthermore, extensive experiments on a poultry disease dataset demonstrate the effectiveness of our approach, marking a significant advancement in this field.