Multiple studies have reported mathematics underachievement for students who are deaf, but the onset, scope, and causes of this phenomenon remain understudied. Early language deprivation might be one factor influencing the acquisition of numbers. In this study, we investigated a basic and fundamental mathematical skill, automatic magnitude processing, in two formats (Arabic digits and American Sign Language number signs) and the influence of age of first language exposure on both formats by using two versions of the Number Stroop Test. We compared the performance of individuals born deaf who experienced early language deprivation to that of individuals born deaf who experienced sign language in early life and hearing second language learners of ASL. In both formats of magnitude representation, late first language learners demonstrated overall slower reaction times. They were also less accurate on incongruent trials but performed no differently from early signers and second language learners on other trials. When magnitude was represented by Arabic digits, late first language learners exhibited robust Number Stroop Effects, suggesting automatic magnitude processing, but they also demonstrated a large speed difference between size and number judgments not observed in the other groups. In a task with ASL number signs, the Number Stroop Effect was not found in any group, suggesting that magnitude representation might be format-specific, in line with the results from several other languages. Late first language learners also demonstrate unusual patterns of slower reaction time for neutral rather than incongruent stimuli. Together, the results show that early language deprivation affects the ability to automatically judge quantities expressed both linguistically and by Arabic digits, but that it can be acquired later in life when language is available. Contrary to previous studies that find differences in speed of number processing between deaf and hearing participants, we find that when language is acquired early in life, deaf signers perform identically to hearing participants.