We describe the atypical fluctuations of the ground state energy of the random elastic manifold, a disordered model defined on a lattice of linear size L with internal dimension \(0\le d<4\) embedded in a medium of dimension \(N\gg 1\) . The ground-state energy results from a competition between confinement, elasticity and disorder. We obtain an exact description of the large deviation rate function with speed \(NL^d\) and its different phases, corresponding to different patterns of replica symmetry breaking (RSB). Our results show that the ground-state energy satisfies a central limit theorem and we obtain an explicit expression for the rescaled variance. In the (massless) limit of zero confinement, this variance vanishes for short-range disorder and the ground-state energy displays super-concentration. From our results on the large deviation function, we characterise explicitly the left tail of the distribution of the typical fluctuations of the ground state energy. It displays an exponential tail for a one step RSB pattern while for a full RSB pattern it decays super-exponentially with a non trivial exponent \(\xi \) that we compute explicitly.