<p>Ultrafast magnetization dynamics has the potential to spark a new era of information technology by harnessing the interaction between spin and light. Recent studies indicate a complex interplay between the microscopic spin and band structure dynamics and the magneto-optical signals, which are typically exploited in experiments. Using a kinetic model based on microscopic Boltzmann collision integrals, we demonstrate that the spin dynamics are intrinsically energy-dependent due to the non-equilibrium dynamics of carrier distributions and the band structure. We find that the energy-resolved spin dynamics of unoccupied states (holes) differs strongly from those of the electrons. This has major implications for the magneto-optical response, which we reveal to be determined by hole dynamics on early timescales and by electron dynamics on longer timescales.</p>