Urban and rural children have different levels of performance in arithmetic processing. This study investigated whether such a residence difference can be explained by phonological processing. A total of 1,501 Chinese primary school students from urban and rural areas were recruited to complete nine cognitive tasks: two in arithmetic performance (subtraction and multiplication), one in phonological processing (word rhyming), two in quantity processing (numerosity and number comparison), one in visuo-spatial processing (mental rotation), as well as a mathematical reasoning task (number-series completion) as a contrast with arithmetic processing and two general cognitive tasks (choice reaction time and Raven's progressive matrices). The results showed that Chinese urban children had superior performance in arithmetic, which was eliminated after controlling for phonological ability, but not after controlling for quantity or visuo-spatial performance or general cognitive factors. Urban children also showed better performance in mathematical reasoning than did rural children, but residence difference in mathematical reasoning was not eliminated by any of the cognitive factors. The above result suggests that phonological processing explains the relationship between residence and arithmetic performance.