Measuring cognitive gains in a multi-skill training game is challenging, especially with dynamic difficulty levels that must be factored into metric design. This exploratory study aims to investigate whether Item Response Theory (IRT) can be used to model the mission difficulty and scale the performance score of participants’ attentional abilities to reflect their training gains in an attentional training game, Skylar’s Run. Properties of EEG-based attention performance were investigated. We modeled the participants’ focused and sustained attention scores during gameplay that are physiologically measured every 1/10th of a second. Using IRT, we calculated difficulty weights for 15 missions and scaled the performance scores accordingly to reflect the inherent difficulty across the missions. We also accounted for EEG data variability. Lastly, we validated the scaled performance scores by fitting regression models and found that training duration had a marginally significant positive association with the performance score of sustained attention but not focused attention. These results provide evidence of the viability of using IRT to consider variability in physiological measures of attention as well as in game difficulty in complex cognitive training games.