Recommender-driven online learning systems (ROLS) are at the forefront of new computer-based learning. They incorporate machine learning to allow learning-by-doing, generating personalized recommendations in the process. This article describes the evaluations of a new type of online learning systems, ROLS. This evaluation was carried out in three phases using a design science research approach. In Phase I, building the ROLS prototype validated the conceptual framework used. In Phase II, building an instantiation of ROLS to teach Structured Query Language (SQL), SQL-with-Ease, validated the ROLS prototype. In Phase III, a laboratory experiment evaluated learning outcomes from using SQL-with-Ease compared with two other traditional forms of learning. A set of qualitative interviews carried out with learners soon after using the system confirmed that the system was effective. They indicated that more work on fine-tuning recommendations generated by the system could further improve learner satisfaction. The key implication for practitioners is that ROLS have the potential to improve learning outcomes significantly. Implications for researchers are that evaluations of ROLS, which include formative and summative evaluations, are essential to improve their performance and that developing innovative approaches to evaluation can advance these learning technologies.