Cosine Similarity Preserving Curse of Dimensionality Reduction for Managing Computational Complexity
摘要
High dimensional data in information retrieval and text mining is increasingly in demand. Information retrieval mechanisms in vector databases are performing well in managing high dimensional data. However, data sparsity remains a significant reoccurring issue forcing the data field to resort to traditional data retrieval, which eventually results in an increased curse of dimensionality. An elevated level of the curse of dimensionality eventually produces models that suffer from computational complexity. Data sparsity, being the root cause of the problem, occurs when data points are farther apart and therefore difficult to measure their similarity. It is on this premise that most conversational platforms are suffering from data hallucinations and unjustified responses throughout the search retrieval process. The curse of dimensionality has also been reoccurring in most high dimensional data retrieval systems despite the quite impressive performance of vectors. The prevalence of exponential increase in dimensions poses processing as well as analysis challenges which equates to computational complexity. To address this, reduction mechanisms are the core possible solutions. This research investigates the effectiveness of similarity metrics to aid dimensionality reduction, all in pursuit of managing computational complexity. We therefore designed a test plan to assess the suitability of various similarity metrics and compared them against core features. A total of 8 similarity metrics against 5 core features were evaluated through a matrix table, and the results indicated that cosine similarity is most suitable for solving the dimensionality issues and lower computational complexity.