This article offers a detailed analysis of systematic site effects derived from the residuals of ground-motion simulations, concentrating on small magnitude (3.5 ≤ MW ≤ 5.0) point source ruptures recorded in New Zealand (NZ). A large portion of the total uncertainty arises from within-event residuals, emphasizing the necessity for an in-depth study of various sedimentary basins and sites that contribute to this uncertainty. To address this issue, the study meticulously examines site-to-site residuals from nationwide simulations, aiming to quantify the modeling errors associated with different site effects. The primary objective is to enhance physics-based simulations in NZ. The analysis employs physical approaches such as geomorphic categorization and site characterization parameters, combined with data-driven clustering methods, to study site-to-site residuals across 212 sites nationwide. Using these influencing attributes for different site types, empirical adjustment factors are developed and applied to the simulation results.