3D semantic occupancy prediction provides fine-grained 3D scene understanding by densely predicting occupancy status and semantic labels. However, dense voxel-based methods are inefficient, as the most voxels remain empty. Recent sparse approaches such as OPUS leverage learnable queries to directly predict occupied locations, but they can struggle to capture fine volumetric structural details. Meanwhile, 3D Gaussian Splatting has emerged as a compact, detail-preserving representation. To this end, this paper proposes ScaffoldOcc, a hybrid sparse point-anchored structural 3D Gaussian representation for hierarchical semantic occupancy prediction. First, multi-scale image features are extracted and used to update a set of learnable queries to encode the global scene. Next, these queries predict a set of sparse anchor points and their associated features, each representing a local region. Around each anchor, we distribute a cluster of local 3D Gaussians to capture fine-grained volumetric details. These Gaussians form a compact structural representation from which semantic occupancy is inferred. Additionally, we introduce a hierarchical multi-layer decoding strategy that progressively learns scene representations and adaptively focuses on complex regions in a coarse-to-fine manner. Experiments on Occ3D-nuScenes demonstrate that ScaffoldOcc surpasses prior methods with less overhead.