The current study investigated the mechanism of language switching in unbalanced visual unimodal bilinguals as well as balanced and unbalanced bimodal bilinguals during a picture naming task. All three groups exhibited significant switch costs across two languages, with symmetrical switch cost in balanced bimodal bilinguals and asymmetrical switch cost in unbalanced unimodal bilinguals and bimodal bilinguals. Moreover, the relative proficiency of the two languages but not their absolute proficiency had an effect on language switch cost. For the bimodal bilinguals the language switch cost also arose from modality switching. These findings suggest that the language switch cost might originate from multiple sources from both outside (e.g., modality switching) and inside (e.g., the relative proficiency of the two languages) the linguistic lexicon.