Large-scale online learning environments such as MOOCs provide an opportunity to evaluate the efficacy of learning strategies in an informal learning context with a diverse learner population. Here, we evaluate the extent to which retrieval practice -- recognized as one of the most effective learning strategies -- facilitates long-term knowledge retention (and thus learning outcomes) among MOOC learners using an instructional intervention. We observed no effect on learning outcomes and high levels of treatment non-compliance. In addition, we conducted a series of exploratory studies into long-term recall of knowledge acquired in MOOCs. Surprisingly, both passing and non-passing learners scored similarly on a knowledge post-test, retaining approximately two-thirds of what they learned over the long term.