This study explores the cognitive and neural mechanisms underlying math avoidance in individuals with high math anxiety (HMA), a pattern contributing to reduced practice and poor performance. Using an approach-avoid conflict paradigm and both general linear mixed model and Hierarchical Drift Diffusion Model (HDDM) regression analyses, we found their avoidance behavior is primarily driven by heightened sensitivity to task difficulty, rather than reward sensitivity. Task difficulty sensitivity also mediated the link between math anxiety and avoidance tendency. Neuroimaging revealed distinct activation in the ventral valuation network (e.g., nucleus accumbens, hippocampus) and cognitive control regions (e.g., precuneus, mid-cingulate cortex, temporo-parietal junction) in HMA individuals. Functional connectivity among these regions effectively distinguished HMA from low math anxiety participants. Additionally, activations in the hippocampus, mid-cingulate cortex, and posterior insula mediated the relationship between math anxiety and avoidance. These findings highlight the cognitive and neural bases of math avoidance and may inform targeted interventions.