Plant diseases pose critical threats to global agricultural productivity and food security, incurring substantial economic losses annually. While traditional detection methods depend on labor-intensive manual observation, current deep learning approaches remain limited by reliance on unimodal visual data and struggle to provide interpret or actionable agricultural insights. To bridge this gap, we present AgriGPT, a comprehensive Visual-Language framework that synergistically integrates image analysis with textual context to enable precise and explainable detection of plant diseases. AgriGPT leverages ImageBind-Huge for robust cross-modal encoding and Vicuna-7B for generating human-readable diagnostic reports with domain-specific knowledge. A key innovation is our Learnable Clustering Module (LCM), which dynamically models normal plant feature distributions using Adaptive and Progressive Updating Mechanisms, enabling precise anomaly detection under data scarcity or class imbalance. By integrating an Encoding-Decoding Module, an LCM, and a Language Module, AgriGPT achieves state-of-the-art performance in localized anomaly localization, disease classification, and actionable treatment recommendations. Extensive experiments across multiple plant categories demonstrate improvements in Image-AUC ranging from 2.45% to 17.81% and in Pixel-AUC from 0.17% to 14.67% compared to existing models. To accelerate research in agricultural AI, we open source our code and a curated multifaceted dataset (covering nine plant and thirteen disease categories) at https://github.com/zzz123nnn/AgriGPT .