In research on math self-concept (MSC) formation, very few studies have juxtaposed the effects of math grades, math ability, school-average math grades, and school-average math ability. However, these factors are important in enabling Taiwanese senior school students to achieve proper MSC and choose a suitable academic track. Thus, the present study aimed to model the relationship of these factors with MSC. The study also examined the gender effect and school location (rurality) effect. The sample contained 1,196 10th-grade students from 31 senior high schools in Taiwan. A doubly latent multilevel linear model was adopted to account for both sampling and measurement errors. The results suggested that math grades relative to the school-average math grade are key in MSC formation. There were school location and gender effects but no significant interaction between gender and school-level math grade or ability. These effects are discussed, as are the implications in practical settings.