This paper compares direct measures of content knowledge for students studying Chemistry through English Medium Instruction (EMI) and Japanese Medium Instruction (JMI). It also examines a range of contextual differences between the two mediums of instruction. A total of 53 undergraduate students undertook pre-post course content tests at a university in Japan. These measurable test outcomes were triangulated with data from semi-structured interviews with eight EMI and five JMI students and two content teachers as well as field notes from ten EMI and ten JMI lecture observations. While the quantitative results revealed no statistically significant differences in the post-test scores between the two courses, the qualitative data illuminated a huge range of context-dependent differences which shed light on possible explanations of this non-significant result. Findings revealed that while there were unique challenges only experienced by EMI students, many of the students navigated their EMI studies successfully through a combination of effective institutional and academic language support, teacher support and -- not least -- collaborative peer networks. This comparative study of an EMI and non-EMI course contributes to the currently limited research evidence on medium of instruction differences on content learning outcomes.