Scalable Document Retrieval for Unstructured Data: A Hybrid Approach Using VSM and Single-Link Clustering
摘要
In industries like law, healthcare, and finance, unstructured text is being generated on day-to-day basis. Data like doctor’s notes, court cases, patient records, emails, and reports are continuously growing. Simple keyword searches to locate among such huge data corpus do not work well in efficient manner. Such searches often miss what the documents really mean and what it actually contains information about. To handle this situation, a hybrid approach was tested as a part of our research. It combines Single-Link Clustering with the Vector Space Model (VSM). The process details about algorithm in which documents and user queries are converted into vectors using Term Frequency-Inverse Document Frequency (TF-IDF). Then, cosine similarity is used to see how close a document is to the user’s query. Single-Link Clustering is very much useful because it doesn’t need a fixed number of clusters. It groups documents as it goes, based on similarity. This makes the system flexible. New documents or overlapping topics don’t break it. The hybrid method was compared to BM25 which is a standard keyword-based model. Metrics like Precision, Recall, F2 Score, MAP, and NDCG were used to verify system performance. Our research methodology shows how the hybrid system performed better. It ranked relevant documents higher and gave more reliable results. Clustering also uncovers connections that keywords alone might miss. There are situations where two different documents may discuss the same topic but with different set of words. But they have common information. Clustering can find links for such documents such that it improves search quality, especially when the dataset is big and complex. In short, combining Single-Link Clustering with VSM is effective. It works for messy and large datasets. It adapts to changes in the query as well as document corpus and gives results that are more meaningful. This is important in applications where accuracy of search results and relevance to user’s query have high importance.