<p>Math anxiety may shape learning engagement in complex, nonlinear ways, yet most studies assume linear effects and often aggregate anxiety facets. Using an online survey of Japanese high school students (<i>n</i> = 574), I examined whether mathematics learning anxiety and mathematics evaluation anxiety showed distinct, potentially nonlinear associations with behavioural, emotional, and cognitive engagement. Generalized additive models with penalized smooths were estimated for each engagement dimension, adjusting for gender and socioeconomic status, and were compared with linear and quadratic alternatives. Behavioural engagement showed a U-shaped association with learning anxiety and a positive association with evaluation anxiety that appeared to level off at higher anxiety. In contrast, emotional and cognitive engagement tended to be lower as evaluation anxiety increased, although model-fit differences relative to simpler specifications were small; learning anxiety was not a significant predictor of these internal dimensions. Although model fit was modest, the results indicate that the anxiety–engagement association may vary in functional form across anxiety facets and engagement dimensions, cautioning against interpreting “more anxiety” as uniformly implying “less engagement.” By separating learning from evaluation anxiety and modeling effects flexibly, this study identifies descriptive, hypothesis-generating patterns—including a visible–internal divergence—that provide a descriptive basis for theory refinement and for generating more targeted hypotheses about engagement under evaluative threat.</p>