Convolutional neural networks have become core parts of many modern chess engines. CNNs with their ability to predict the next move from the current board configuration can be utilized as a replacement for heuristic functions intended to prune the game tree. This prediction can however serve as a standalone chess engine. This paper studies the ability to predict the best move by searching the game tree with a comparison to CNN prediction. Particularly, this paper reveals some limitations of CNNs, specifically ResNet models, to predict the best move in certain scenarios. This paper also proposes a method for measuring chessboard complexity, which can be very helpful for the identification of board configurations, where CNNs struggle with prediction.