Social, emotional, and behavioral (SEB) instruments are currently used in schools to screen, refer, and progress monitor students. Although many of these instruments have demonstrated strong technical adequacy, there has been far less examination of their consequential validity--that is, positive or negative intended and unintended consequences of measure use. A stated purpose of SEB instruments is to facilitate equitable assessment practices; therefore, examining consequential validity is needed. In this study, we examined the unintended negative consequences of one existing instrument: the Direct Behavior Ratings-Single Item Scales (DBR-SIS). We investigated unintended negative consequences by examining variation in the types of evidence teachers use to justify different students' DBR-SIS scores. Participants included twenty-eight teachers (13 elementary, 15 secondary) who watched standardized video clips in which student actors engaged in a variety of behaviors. Using a verbal protocol procedure, we had participants rate the behavior of four focal students using the DBR-SIS while explaining how they arrived at each focal student's score. Using conventional content analysis, we found that teachers' justifications often did not align with definitions provided on the instrument. In these cases, teachers justified scores using labels they applied to students, references to classroom experience, personal expectations for student behavior, instructor redirection, comparisons to other students, and misapplied definitions. Justifications were not consistently applied across students by race and gender; Black students were generally described more harshly than White students. We discuss the potential social consequences of these results and implications for teachers' professional learning.