In recent years, a variety of computer vision methods have been proposed for hand pose estimation. The reported error rates on public benchmark datasets are often quite low, in the order of a few millimeters per keypoint. However, such accuracies are obtained by experimenting with specific datasets. In this paper, we pose the question of whether these accuracies are reliable estimates of performance over general real-world data, outside of the datasets that were used. We propose a method for assessing whether a test set has realistic or unrealistic similarities to a training set. In principle, test results are representative of real-world performance if the test cases are randomly and independently sampled from the space of all possible cases that the system could be applied to. Conversely, test results cannot be trusted as representative of real-world performance if the test images are unrealistically similar to the training set, more similar than we would expect truly random test cases to be. In our experiments, we find that such an unrealistic degree of similarity indeed exists in three public benchmark datasets, namely the NYU Hand Pose dataset, the MSRA dataset, and the BigHand dataset. We believe that the approach proposed in this paper can inform how to interpret the reported accuracies of state-of-the-art hand pose estimation systems, and can be a valuable guideline towards constructing less biased and more representative datasets in the future.