This article explores the power of language to either include or exclude certain groups of students from genuine opportunities for mathematical sense-making. The substantial increase worldwide in the number of students learning mathematics through a language other than their primary language makes this a particularly urgent issue. This paper focuses on the South African situation, where, because English is widely perceived as the language of opportunity, it is, by grade 4, overwhelmingly the chosen language of learning and teaching. The epistemological and pedagogical consequences of this choice are evidenced in the poor performance of the country's students on national and international assessments of mathematical proficiency. Drawing on research literature around language immersion education models and the extent to which these align with certain key principles of second language acquisition, this position paper motivates for a stronger and more sustained commitment to providing students, particularly those from marginalized and vulnerable communities, with opportunities for becoming both bilingual and biliterate. Empirical data from two South African grade 4 mathematics classrooms are used to illuminate aspects of the mathematical sense-making challenges students and their teachers face without such commitment.