Exploring Methodologies for Computing Sentence Similarity in Natural Language Processing
摘要
The primary objective of this paper is to provide a thorough examination and comparative analysis of various methodologies for computing sentence similarity within the domain of natural language processing (NLP). By exploring a wide range of approaches—string-based, syntax-based, graph-based, transformer-based, and prompt engineering-based methods—the paper aims to evaluate the effectiveness and limitations of each technique. Specific methods discussed include Word2Vec, WordNet, GUSUM, BERT, SGPT, and AnglE. Our findings highlight the broader implications of semantic similarity over string-based techniques, with a notable shift towards Language Model Models (LLMs) alongside transformer dominance. Future research suggestions include developing algorithms for unsupervised sentence embeddings, handling datasets exceeding sequence length limits, and exploring context regularization methods. Analyzing challenges in sentence-based similarity via graph-based approaches and experimenting with prompt engineering techniques are also recommended, providing a roadmap for advancing sentence similarity computation in NLP.