This study employed propensity score matching and regression analysis to determine whether or not a retention scholarship pilot program at a mid-size public research university was effective at increasing first-year retention. The scholarship pilot did not require any application process for students but rather used a predictive logistic regression model for identifying recipients and automatically awarded students $5,000. The scholarship was only awarded for the students' sophomore year. The award was not continued as students persisted to their junior and senior years. Findings demonstrated that students who did not receive the scholarship were two times more likely to depart from the institution at the end of their first year compared with scholarship recipients. Longitudinally, nonscholarship recipients were 1.4 times more likely to depart at the end of their sophomore year.