Listeners have been shown to judge second language (L2) users more negatively than L1 speakers on measures of language proficiency and even on personal qualities. However, less is known about what factors affect listener judgment which this study seeks to explore. In the study, one first language (L1) English speaker recorded her spontaneous speech (i.e. the baseline recording). Three other speakers of English (with Northern American, moderate Chinese, and heavy Chinese accents) read aloud two scripts controlled for content and language (one grammatical and one ungrammatical), resulting in six scripted recordings. Six-hundred and eighty-seven listeners with mixed L1s were randomly allocated into six groups. Each group listened to the same baseline recording and a different scripted recording. They evaluated the speakers' proficiency on 9-point scales and reported their background information via an online questionnaire. Results suggested that speakers' accents negatively influenced listeners' proficiency judgment. However, a threshold was observed regarding this influence; specifically, the moderately-accented speaker was not perceived negatively compared to L1 speakers. Multiple regression showed that listeners' accent familiarity and linguistic training were positively associated with their rating severity, meaning that these variables have the potential to affect listeners' perceptual bias against L2 users.