In this paper, we recast the variational formulation corresponding to the single layer boundary integral operator \(\varvec{{\text {\textsf{V}}}}\) for the wave equation as a minimization problem in \(\varvec{L^2(\Sigma )}\) , where \(\varvec{\Sigma := \partial \Omega \times (0,T)}\) is the lateral boundary of the space-time domain \(\varvec{Q:= \Omega \times (0,T)}\) . For discretization, the minimization problem is restated as a mixed saddle point formulation. Unique solvability is established by combining conforming nested boundary element spaces for the mixed formulation such that the related bilinear form is discrete inf-sup stable. We analyze under which conditions the discrete inf-sup stability is satisfied, and moreover, we show that the mixed formulation provides a simple error indicator, which can be used for adaptivity. We present several numerical experiments showing the applicability of the method to different time-domain boundary integral formulations used in the literature.