Generalizability theory (g-theory) was used, as an alternative to classical test theory, to evaluate measurement error in a behaviorally anchored role-play measure, highlighting the usefulness of this theory in instrument development. G-theory partitions an observed score into the universe score and error scores associated with separate sources of error variation, and enables the researcher to make decisions about instrument development and validation not afforded by classical test theory. Forty-four female patients at an eating disorders clinic and a sample of 44 young women without eating disorders were recruited for the study. Participants completed questionnaires about demographics and eating disorders and then responded to vignettes about problem family situations. These responses were rated by four raters as part of the development of the behavioral role-play measure of social competence, and g-theory was used to assess agreement among the four raters. Traditional approaches to assessing reliability were compared to generalizability theory to validate the developed instrument. Although four raters were used, results suggest that a single rater could produce a reliable score. Results also show that using g-theory in addition to classical test theory to assess the psychometric properties of the instrument offers a measurement approach that benefits researchers and clinicians. (Contains 5 tables and 28 references.) (SLD)