The k-Nearest Neighbor (k-NN) graph is an essential technique in data mining, machine learning, and computer vision for identifying local data patterns; however, its efficacy is significantly hindered in high-dimensional datasets due to the challenges of the curse of dimensionality, which includes issues like distance concentration and heightened computational requirements. To mitigate these challenges, various strategies, including dimensionality reduction and sophisticated graph construction methods, have been proposed. Outlier detection, vital in domains such as fraud detection and healthcare, similarly faces obstacles in high-dimensional contexts, leading to the development of diverse statistical and learning-based methodologies. This research aims to enhance k-NN graph construction and outlier detection by utilizing sparsification and subspace selection techniques, which effectively capture both local and global outliers through precise estimation of intrinsic dimensionality in localized areas. Furthermore, the study underscores the significance of visualizing high-dimensional data in reduced dimensions to enhance human interpretability, proposing an algorithm that leverages sparsified data representations to improve the accuracy of outlier detection and provide a more efficient framework for high-dimensional data analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Outlier Selection with Local Intrinsic Dimensionality and Visualization Using t-SNE

  • Dharamsotu Bheekya,
  • Salman Abdul Moiz,
  • C. Raghavendra Rao

摘要

The k-Nearest Neighbor (k-NN) graph is an essential technique in data mining, machine learning, and computer vision for identifying local data patterns; however, its efficacy is significantly hindered in high-dimensional datasets due to the challenges of the curse of dimensionality, which includes issues like distance concentration and heightened computational requirements. To mitigate these challenges, various strategies, including dimensionality reduction and sophisticated graph construction methods, have been proposed. Outlier detection, vital in domains such as fraud detection and healthcare, similarly faces obstacles in high-dimensional contexts, leading to the development of diverse statistical and learning-based methodologies. This research aims to enhance k-NN graph construction and outlier detection by utilizing sparsification and subspace selection techniques, which effectively capture both local and global outliers through precise estimation of intrinsic dimensionality in localized areas. Furthermore, the study underscores the significance of visualizing high-dimensional data in reduced dimensions to enhance human interpretability, proposing an algorithm that leverages sparsified data representations to improve the accuracy of outlier detection and provide a more efficient framework for high-dimensional data analysis.