<p>Combining triplet knowledge from knowledge graphs with pre-trained language models can significantly enhance model performance. However, existing knowledge-enhanced pre-trained language models overlook the semantic relevance between the external knowledge and the original input, leading to knowledge noise where the inserted external knowledge distorts the input’s semantics. Moreover, existing knowledge-enhanced pre-trained language models aimed at improving natural language understanding are generally designed for broader tasks rather than specifically for text classification. We introduce ELITE, a novel approach for integrating knowledge into pre-trained language models for text classification. ELITE reduces knowledge noise by filtering out external knowledge with low relevance to the original input, assessing semantic relevance at the raw input level, and distinguishing between external knowledge and the original input during encoding. Furthermore, ELITE employs a graph neural network with a unique graph structure and an auxiliary mask prediction task to enhance the semantic representation of classification tokens, establishing itself as a promising solution for enhancing natural language understanding, especially in text classification tasks. Experimental results demonstrate that ELITE surpasses current state-of-the-art models in five Chinese natural language processing text classification tasks, improving accuracy by up to 2.0%. The source code and datasets used in this study are available at: <a href="https://github.com/Drew19980118/ELITE">https://github.com/Drew19980118/ELITE</a></p>