This paper reports a study designed to examine the neuronal correlates for comprehending the signs of American Sign Language representing numerals in deaf signers who acquired Japanese Sign Language as their first language. The participants were scanned by functional magnetic resonance imaging (fMRI) twice on the day of the experiment. The results of the measurements revealed that upon learning that the signs actually have numeric meaning, a network of brain areas is activated immediately. Many of these areas have been previously implicated in numerical processing. The similar neural network of brain regions responsible for numerical processing exists on a nonlinguistical basis and works to retrieve arithmetic facts from presented linguistic material regardless of the mode of the language.