This study is a secondary analysis of the data collected in a randomized control trial study with sixth graders with mathematics learning difficulties (MLDs). We explored the relation between mathematics anxiety and mathematics performance for students with and without MLD. We further explored whether mathematics vocabulary and computational fluency mediate the relation between mathematics anxiety and mathematics performance, and whether such relations differ for students with and without MLD. A group of U.S. sixth graders (N = 245) with (n = 65) and without MLD (n = 180) were assessed on mathematics anxiety, mathematics vocabulary, computational fluency, and mathematics competence. Results indicated that for students without MLD, mathematics anxiety had a direct negative effect on mathematics vocabulary and computational fluency, which had a direct positive effect on mathematics competence. Mathematics vocabulary and computational fluency fully mediated the relation between mathematics anxiety and mathematics competence for the no-MLD group. For students with MLD, only mathematics vocabulary had a significant direct effect on mathematics competence. Our findings suggest that the nature of the relation between mathematics anxiety and mathematics competence differs by MLD status, and students with MLD may require different types of intervention to improve mathematics competence.