Federal policies (e.g., 2002 reauthorization of the Elementary and Secondary Education Act [ESEA] and the American Recovery and Reinvestment Act) posit that teacher quality is a potential leverage point for improving student achievement (U.S. Department of Education, 2010). Moreover, in the Race to the Top competition, teacher effectiveness must be based, in part, on teacher performance measured by classroom observations. This has driven many districts to adopt teacher classroom observation rubrics to meet the Race to the Top requirement. Without clear guidance on how to rate teachers and without proper calibration activities, scores on these rubrics can become upwardly biased, leading to an inability to distinguish among teachers at different performance levels (see, for example, Weisberg, Sexton, Mulhern, & Keeling, 2009). When a rubric is used inconsistently, teachers may not receive useful feedback, and the rubric could lack teacher buy-in, resulting in views that the evaluation does not provide credible information. In partnership with district central office staff, a study was conducted in a set of 20 elementary and middle schools in the western United States to investigate the use of a classroom observation rubric within the context of a district's pilot teacher performance evaluation system. The district wanted to know whether peer raters would use the observation rubric differently compared with principals, whether interrater reliability differed by rater type, and whether district-selected raters, on average, were more or less lenient than principals in rating other teachers. Supplemental tables and figures are appended.