This study advances material classification using Spectral Sub-Surface Scattering ( \(\mathcal {S}^{4}\) ) measurements. While spectrum and subsurface scattering measurements have individually been used in material classification, we argue that the strong spectral dependence of subsurface scattering lends itself to highly discriminative features. However, obtaining \(\mathcal {S}^{4}\) measurements requires a time-consuming hyperspectral scan. We avoid this by showing that a carefully chosen 2D projection of the \(\mathcal {S}^{4}\) point spread function is sufficient for material estimation. We also design and implement a novel imaging setup, consisting of a point illumination and a spectrally-dispersing camera, to make the desired 2D projections. Finally, through comprehensive experiments, we demonstrate the superiority of \(\mathcal {S}^{4}\) imaging over spectral and sub-surface scattering measurements for the task of material classification.