There is a growing disconnect between the theory and practice of anomaly detection in time series, especially in respect of the metrics employed. Many studies borrow well-known metrics from adjacent fields or employ flawed metrics that prove to be wholly unsuitable for the task at hand. We begin addressing this issue by formulating a set of basic assumptions that apply to most applications, and demonstrate that virtually all widespread metrics violate these assumptions. Thereafter, we introduce three new metrics: one for evaluating entire anomaly detection pipelines, one for benchmarking anomaly scoring models, and one designed to be flexible enough for more demanding use cases. All of these metrics are relatively interpretable, computationally feasible, and closely aligned with desirable behaviour in practice.