Criteria for the selection of judges (evaluators) for setting item-based standards involved in tests for which cutting scores must be established are investigated. Focus is on cases in which test standards are based on specialists' judgments concerning the difficulty of test items in tests used to determine who will be awarded a diploma, admitted to a program, certified as competent, or licensed to practice. The literature on the nature of expertise describe the qualifications that are required of judges assigned to apply item-based standard-setting procedures. Qualifications for judges of items for tests used to select applicants to teacher education programs, grant initial teacher certificates, and grant high school diplomas are briefly outlined. Post hoc analyses of judges are preferable to competency testing of judges. An item response model outlined by Van der Linden (1982) and an alternative approach involving interrater comparisons outlined by Jaeger (1988) are offered as possible means of post hoc analysis. Problems associated with determination of a sufficient number of judges, including issues involving standard errors of mean and measurement, are outlined. In general, judges should be selected through procedures that permit generalization of their collective recommendations to well-defined populations. The numbers of judges selected should be sufficient to provide precise estimation of the standard that would be recommended by an entire population of judges. (TJH)