Transformer fault diagnosis (TFD) plays a critical role in the maintenance and management of power systems. Dissolved gases in insulating oil have emerged as critical indicators of transformer faults, as they effectively reveal the internal failures of power transformers. The primary methods for dissolved gas analysis (DGA) encompass knowledge-driven algorithms and data-driven algorithms. Despite their significant advancements, they are still subject to certain limitations. Specifically, knowledge-driven algorithms are constrained by their excessive reliance on predetermined rules, and the uncertainties inherent in industry situations inevitably diminish diagnostic accuracy. On the other hand, data-driven algorithms face challenges such as overfitting with limited samples and the difficulty in interpreting complex machine learning models. Therefore, we propose a transformer fault diagnosis algorithm based on knowledge distillation of large language models, aiming to mitigate the drawbacks of knowledge-driven and data-driven approaches. The main idea is to encode prior knowledge of large language models as prior fault distributions and employ label distribution learning techniques to integrate this prior knowledge with the fault classifier. Finally, extensive experimental results validate the effectiveness of our proposed method.