Anomaly detection is crucial for identifying unusual patterns that deviate from expected behavior, enabling timely responses to potential threats, fraud, or system failures. However, the prevalence of high-dimensional data poses substantial challenges to anomaly detection, often leading to performance deterioration due to increased feature complexity. Traditional methods are ineffective in coping with high dimensionality, hindering their anomaly detection capabilities. This study introduces a novel approach, particle swarm optimization with isolation forest (PSOIF), designed to enhance anomaly detection in high-dimensional data. PSOIF utilizes the particle swarm optimization to select relevant features and optimize the threshold value of the isolation forest algorithm, thereby improving its performance. Comprehensive experiments are conducted on seven high-dimensional datasets with four state-of-the-art comparison methods. The outcomes, measured across three distinct performance metrics, demonstrate PSOIF’s overall effectiveness and superiority in managing high-dimensional anomaly detection tasks.