As higher education institutions develop fully online course programs to provide better access for the non-traditional learner, there is increasing interest in identifying students who may be at risk of attrition and poor performance in these online course programs. In our study, we investigate the effectiveness of an online orientation course in improving student retention in an online college program. Using student activity data from the orientation course, Engage, we make use of machine learning methods to develop prediction models of whether students will be retained and continue to register for program-specific courses in the eVersity program. We then discuss the implications of our findings on improvements that may be made to the existing orientation course to improve student retention in the program. [For the full proceedings, see ED596512.]