Response Process Validity (RPV) reflects the degree to which items are interpreted as intended by item developers. In this study, teacher responses to constructed response (CR) items to assess pedagogical content knowledge (PCK) of middle school mathematics teachers were evaluated to determine what types of teacher responses signaled weak RPV. We analyzed 38 CR pilot items on proportional reasoning across up to 13 middle school mathematics teachers per item. By coding teacher responses and using think-alouds, we found teachers' responses deemed indicative of low item RPV often had one of the following characteristics: vague answers, unanticipated assumptions, a focus on unintended topics, and paraphrasing. To develop a diverse pool of items with strong RPV, we suggest it is helpful to be aware of these symptoms, use them to consider how to improve items, and then revise and retest items accordingly. [This paper was published in: "Proceedings of the forty-fourth annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education," edited by A. E. Lischka et al., Middle Tennessee State University, 2022, pp. 671-75.]