We study the long time evolution of the position–position correlation function \(C_{\alpha ,N}(s,t)\) for a harmonic oscillator (the probe) interacting via a coupling \(\alpha \) with a large chain of N coupled oscillators (the heat bath). At \(t=0\) the probe and the bath are in equilibrium at temperature \(T_P\) and \(T_B\) , respectively. We show that for times t and s of the order of N, \(C_{\alpha ,N}(s,t)\) is very well approximated by its limit \(C_{\alpha }(s,t)\) as \(N\rightarrow \infty \) . We find that, if the frequency \(\Omega \) of the probe is in the spectrum of the bath, the system appears to thermalize, at least at higher order in \(\alpha \) . This means that, at order 0 in \(\alpha \) , \(C_\alpha (s,t)\) equals the correlation of a probe in contact with an ideal stochastic thermostat, that is forced by a white noise and subject to dissipation. In particular we find that \(\lim _{t\rightarrow \infty } C_\alpha (t,t)=T_B/\Omega ^2\) while that \(\lim _{\tau \rightarrow \infty } C_\alpha (\tau ,\tau +t)\) exists and decays exponentially in t. Notwithstanding this, at higher order in \(\alpha \) , \(C_{\alpha }(s,t)\) contains terms that oscillate or vanish as a power law in \(|t-s|\) . That is, even when the bath is very large, it cannot be thought of as a stochastic thermostat. When the frequency of the bath is far from the spectrum of the bath, no thermalization is observed.