Classification models based on deep learning models have been widely used in fault diagnostic of nuclear power plants, while Softmax function is frequently employed in them to solve the multi-classification problem. However, those classification models are typically trained in a closed-world setting. It implies that both the test data and the training data are assumed to originate from the same distribution, which is referred to as the In-Distribution. But models deployed in real-world scenarios will inevitably encounter data that does not belong to the distribution, that is Out-of-Distribution data (OOD). Such an occurrence is prevalent in nuclear power plant fault diagnostic models, often characterized by overconfident classification of OOD, manifested as a high degree of confidence in results that may not accurately reflect their true nature. The issue has significant implications, as misclassification of OOD may result in erroneous decisions by nuclear power plant operators. This paper tests universal fault diagnostic models by simulating OOD samples that may occur in the actual operation of nuclear power plants. The results validate the overconfidence shown by the common CNN/BP fault diagnostic model in this case and provides an in-depth analysis of the potential risks and consequences. Furthermore, the importance of enabling models to recognize OOD data is highlighted, and potential solutions to mitigate model overconfidence are proposed as means to enhance the reliability of deep learning-based classification models in practice.