<p>Carbonaceous mudstone spoil, a granular soil derived from loose waste dumps, exhibits uniform particle sizes and poor gradation, rendering it prone to static liquefaction. A comprehensive series of drained and undrained triaxial compression and extension tests were conducted to investigate the influence of gradation and particle size on the shear behavior of this material. Five distinct gradations were examined, encompassing three uniformity coefficients and three mean particle sizes. The results revealed that gradation and particle size significantly affect critical state parameters: soils with poorer gradations and smaller mean particle sizes exhibited elevated and steeper critical state lines (CSLs) in <i>e–p</i> space, alongside reduced critical stress ratios (<i>M</i>). In contrast, the slope of the instability line, defined by peak shear stresses, remained unaffected by gradation or particle size. Notably, the instability line of triaxial extension tests did not cross the origin, indicating a susceptibility to liquefaction. During undrained shearing, poorer gradations correlated with diminished dilation, where porewater pressure initially surged to high levels but subsequently exhibited negligible increase. A linear relationship was identified between the maximum porewater pressure ratio and the brittleness index, which quantifies post-instability shear strength reduction. These trendlines were parallel but offset between compression and extension conditions. This study quantifies the effects of gradation and particle size on the critical state, dilatancy, instability, and liquefaction potential of carbonaceous mudstone spoil, providing insights critical for practices such as landslide risk assessment.</p>