Reviewing software changes is a critical activity that helps prevent the introduction of defects, ultimately saving development time and reducing costs. Just-in-time defect prediction has emerged as a promising approach to support this process by estimating the likelihood of defects in newly submitted commits. Effort-aware evaluations were proposed to better manage developers’ limited time and to analyze the applicability of defect prediction approaches. However, current effort-aware approaches neglect the time-dependent nature of software engineering when evaluating the performance and rank commits to find most defective commits with limited effort. This reveals a gap between the evaluation of models in research and their application in practice, where a timely decision for every single commit is needed. To assess the impact of effort-aware evaluations on the applicability of defect prediction approaches, we quantify their limitations and provide insights into the implications for future research. Our findings show that effort-aware metrics can overestimate the proportion of defects identified within a limited inspection budget by up to 55% when accounting for realistic, time-sensitive review processes.