This article investigates the relative effectiveness of online and paper-based formative assessment in the learning of English as a second language. For this study, 74 Japanese university second language (L2) students took online quizzes, while 71 were given paper-based quizzes to complete. The quiz content was drawn from learning material found in the class textbook. Both groups were assigned the exercises as homework. At the end of the 15-week study period, both groups were administered a summative assessment that also covered learning points from the textbook. After controlling for the learners' L2 proficiency through their TOEIC test scores, the performance of the two groups on the formative quiz was compared through correlation and regression analysis. While there was found to be a statistically significant relationship between the online formative assessment and summative exam scores, this was not the case for the paper-based homework. When compared through an analysis of covariance, there was found to be a significant interaction between group and homework on exam scores. The online group was more affected by their homework score than the pencil and paper group. The findings from this study have implications for teachers, learners and curriculum developers.