In circumstances where judges are required to make ratings of performance, it is usually required to have two or more raters who are trained to agree on independent ratings of the same performance. It is suggested that such a requirement may produce the paradox of attenuation associated with item analysis, in which too high a correlation between items, while enhancing reliability, decreases validity. With an empirical example of ratings of levels of social understanding it is shown how information is lost as judges agree more and more over a series of ratings which follow discussions of the previous set of ratings. Pursuing the item analysis analogy, it is demonstrated how it might be more appropriate to handle systematic differences among judges using a person-item response theory approach. (Author)