The paper studies the hardness of the Shift Minimization Personnel Task Scheduling Problem (SMPTSP). For this purpose, a new dataset of SMPTSP instances are introduced. The new instances are clearly harder and more diverse than the current ones. The performance of the two best performing algorithms on the current and on the new benchmark instances are published. The study includes two sets of hardness indicators: basic instance characteristics and performance of heuristic methods. Conclusions are drawn at two levels: simple visual observations and statistical methods. The statistical methods used are discriminant analysis, support vector machines and logistic regression. The study reveals a number of interesting benchmark instances that, when carefully examined, could provide statistically significant hardness indicators. We encourage the research community to do in-depth study on the structural properties of the instances based on graph theory.