This study demonstrates the importance of Rasch Measurement Theory (RMT) in program evaluation when outcome measures need to be constructed from scratch. The paper introduces typical measure validation methods presented in program evaluation texts and discusses room for improvement. The study then illustrates how the seamless transitions from program theory, to ethnographic observation, to item generation, to measure analysis, to effectsize estimation are all possible under one framework using Rasch modelling. The program evaluated in the present study focuses on increasing retention of students in the biology major at a public university. The results presented balance the pragmatics and technical requirements of program evaluation. We show how this approach can align with Robert Stake's call for valuing in program evaluation.