Cognitive pretesting (CP) is an interview methodology for pretesting the validity of items during the development of self-report instruments. This article reports on the development and evaluation of a systematic method to rate self-report item validity performance utilizing CP interview text data. Five raters were trained in the application of that system, resulting in acceptable to substantial levels of interrater agreement. Results from this study suggest that excellent reliability can be achieved in the analysis of CP data. Guidelines for systematically rating the qualitative data collected using CP methods are provided. Future research should focus on empirical demonstrations of how such rating procedures can lead to improvements in the validity of self-report instruments. (Contains 3 tables and 1 figure.)