The purpose of this case study is twofold: it (1) explores multilingual raters' judgements regarding the accentedness and comprehensibility of second language speech and (2) examines how the raters' language backgrounds influence their judgements. In this study, six multilingual Singaporean raters judged the accentedness and comprehensibility of 50 unfamiliar accented speech samples produced by Japanese learners of English with different proficiency levels. In order to investigate rater judgement, the rating scores were subjected to a multifaceted Rasch analysis. A questionnaire and an interview elicited the raters' retrospective reports on their language backgrounds at three time points in their life (when they were 5, 11, and 21 years old). The results suggested that the raters' language backgrounds, notably the proficiency balance between multiple languages in early childhood, are related to their rating judgement.