An assumption-free, disease-agnostic pathology detector and segmentor is often regarded as one of the holy grails in medical image analysis. Building on this concept, un- or weakly supervised anomaly localization approaches have gained popularity. These methods aim to model normal or healthy samples using data and then detect deviations (i.e., abnormalities). However, as this is an emerging field situated between image segmentation and out-of-distribution detection, most approaches have adapted their evaluation setups and metrics from either of these areas. Consequently, they may have overlooked peculiarities inherent to anomaly localization. In this paper, we revisit the anomaly localization setup, analyze commonly used metrics, introduce alternative metrics inspired by instance segmentation, and compare these metrics across various settings and algorithms. We contend that the choice of metric is use-case dependent, but the SoftInstanceIoU and other object-based metrics show significant promise for future applications.