Hybrid Prediction Framework Using Novel Stability-enhanced Dynamic Thresholding Feature Selection and Artificial Intelligence Methods for Financial Market Trend Prediction
摘要
Financial markets are characterised by volatility, non-stationarity, and high-dimensional data, making accurate daily price direction prediction challenging. The abundance of technical indicators derived from historical price and volume series often introduces redundancy and noise, which degrades model performance and generalisability. Effective feature selection is therefore essential to isolate informative features and improve forecasting precision. This study developed a hybrid framework for daily price direction prediction in global equity indices. The framework incorporated feature expansion through rolling-window technical indicators, novel feature selection via Stability-Enhanced Dynamic Thresholding (SEDT), and multi-model prediction. SEDT aggregated stable importance scores over iterations using a linear Support Vector Classifier and applied a dynamic percentile thresholding to select adaptive, noise-reduced subsets. The methodology was applied to six major market indices: Financial Times Stock Exchange 100 (FTSE), Shanghai Stock Exchange (SSE), Standard and Poor 500 (S&P 500), Deutscher Aktienindex (DAX), New York Stock Exchange (NYSE), and Hang Seng Index (HSI). Eight classifiers were evaluated: Logistic Regression (LR), Random Forest (RF), Support Vector Machine (SVM), Extreme Gradient Boosting (XGB), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Convolutional Neural Network (CNN), with hyperparameter optimisation via grid search. Time-series cross-validation and hold-out splits (70/30, 80/20, 90/10) were used, assessing accuracy and F1-score against baselines: Plain (no selection), Least Absolute Shrinkage and Selection Operator (LASSO), Boruta, and Recursive Feature Elimination with Cross-Validation (RFECV). Results showed SEDT consistently outperformed baselines across all indices and models, with (XGB) achieving the highest metrics when using SEDT filtered features. Wilcoxon signed-rank tests confirmed statistically significant improvements in most cases. SHAP analysis revealed market-specific feature importance, dominated by momentum and oscillator indicators. Trading simulations on the 90/10 split using long-short strategies from (XGB) signals demonstrated SEDT’s economic value through superior total returns, lower volatility, shallower maximum drawdowns, and higher Sharpe ratios. These findings have implications for finance by providing a reliable feature selection approach that enhances predictive models and supports profitable trading strategies. The framework offers traders and investors a tool for better-informed decisions in volatile equity markets, contributing to improved risk management and maximum return generation.