The Assessment of Language Competence (ALC) certificates is an annual, international testing program developed by the Australian Council for Educational Research to test the listening and reading comprehension skills of lower to middle year levels of secondary school. The tests are developed for three levels in French, German, Italian and Japanese, and at two levels in Chinese and Indonesian. There is a mixture of target language and English questions in the Level 2 and 3 tests. Some teachers have raised this as a concern in the belief that all questions should only be offered in English for the sake of fairness. Their view is that the tests are unduly difficult when they are designed with questions in the language. Arising from this concern, the aim of the research to be reported in this paper was to investigate the effects of the language of the question on student performance. We drew on data from a trial test and a final listening test, a questionnaire administered to students to gauge their perceptions of the tests and an examination of public documentation about the ALC. For the statistical analysis, we used Item Response Theory for calibrating items and for comparing item difficulty estimates, and fit statistics to verify how well items with English and target language question formats worked together. We found that where the questions involved listening for simple, explicitly stated information, students found the items in the target language relatively easier. In questions that required students to listen for global meaning, language choice either did not matter or tended to favour items in English rather than the target language. Furthermore, each of the six tests with some items in English and others in the target language showed a high level of reliability and fit to the single latent scale, indicating that items were functioning consistently regardless of the language of the test question. (Contains 1 table, 5 figures and 3 notes.)