In this paper we study the Hilbert–Schmidt norm of time-frequency localization operators \(L_{\Omega } :L^2(\mathbb {R}^d) \rightarrow L^2(\mathbb {R}^d)\) , with Gaussian window, associated with a subset \(\Omega \subset \mathbb {R}^{2d}\) of finite measure. We prove, in particular, that the Hilbert–Schmidt norm of \(L_\Omega \) is maximized, among all subsets \(\Omega \) of a given finite measure, when \(\Omega \) is a ball and that there are no other extremizers. Actually, the main result is a quantitative version of this estimate, with sharp exponent. A similar problem is addressed for wavelet localization operators, where rearrangements are understood in the hyperbolic setting.