<p>This paper undertakes an extensive study to search for empirical evidence of directional predictability and profitability on an aggregate stock market index by applying supervised machine learning (ML) algorithms to a large set of financial variables, technical indicators, and price patterns to generate predictions [of the moving direction of future asset prices] that can then be used to identify buying and selling opportunities. We use symmetric and asymmetric loss function to train (and both statistical and economic scoring functions to cross-validate) a ML algorithm. We also extend the bootstrap Reality Check (RC) procedure to formally compare the performance of trading methods. The trading strategy using one-period ahead ML forecasts can generate higher annualized returns than the buy-and-hold strategy on average when transaction cost is low and there is no strong upward momentum in the market. These average annualized excess returns (i.e., the average annualized returns of our strategy in excess of those from buying and holding the aggregate market index) are statistically significant. Most positive annualized excess returns are realized during trading sessions with low price/high volatility. However, the trading strategy using multiple-days ahead forecasts can become less profitable. We also find that economically motivated scoring functions (such as the <i>correlation between equity curve and perfect profit (CECPP)</i>, the <i>Calmar ratio</i>, or the <i>Sharpe ratio</i>) can generate more profitable trading strategies than the others. Several candlestick chart patterns have a strong predictive power that can be effectively leveraged by Random Forest to increase the annualized excess return compared to using only financial variables and technical indicators as predictors.</p>