In this study, we outlined some of the obstacles associated with screening and identifying high-quality online adjunct instructor candidates and described the steps that we took to improve our screening and hiring process. We discussed issues associated with collecting and using performance ratings in the context of a remote screening and hiring process. We also demonstrated a novel application of the Many Facet Rasch (MFR) model to ratings of online adjunct instructor candidate performance. The MFR model was useful for validating the results of our processes improvements. Investing in streamlining and focusing our Candidate Assessment Course (CAS) yielded screening results that were more valid and reliable while requiring fewer administrative resources.