Outlier Selection with Local Intrinsic Dimensionality and Visualization Using t-SNE
摘要
The k-Nearest Neighbor (k-NN) graph is an essential technique in data mining, machine learning, and computer vision for identifying local data patterns; however, its efficacy is significantly hindered in high-dimensional datasets due to the challenges of the curse of dimensionality, which includes issues like distance concentration and heightened computational requirements. To mitigate these challenges, various strategies, including dimensionality reduction and sophisticated graph construction methods, have been proposed. Outlier detection, vital in domains such as fraud detection and healthcare, similarly faces obstacles in high-dimensional contexts, leading to the development of diverse statistical and learning-based methodologies. This research aims to enhance k-NN graph construction and outlier detection by utilizing sparsification and subspace selection techniques, which effectively capture both local and global outliers through precise estimation of intrinsic dimensionality in localized areas. Furthermore, the study underscores the significance of visualizing high-dimensional data in reduced dimensions to enhance human interpretability, proposing an algorithm that leverages sparsified data representations to improve the accuracy of outlier detection and provide a more efficient framework for high-dimensional data analysis.