Recent research on multiple measures of teaching effectiveness has redefined the role of in-classroom observations in teacher evaluation systems. In particular, most states now mandate that teachers are observed on multiple occasions during the school year, and it is increasingly common that multiple raters are utilized across the different rating occasions. In-classroom observations are typically conducted using a rating rubric. However, related research has found that many rubrics measure multiple dimensions of instructional quality, suggesting that teachers' practices are not well described in terms of a total score. Halpin & Kieffer (2015) argued for the use of latent class analysis (LCA) as a means of capturing the multidimensional features of rating rubrics, while also providing the standard error of measurement for each teacher, and item-level diagnostic information that can be used as the basis of feedback to educators and for professional development. The main purpose of the present research is to develop a multilevel extension of the LCA methodology described by Halpin & Kieffer (2015). For a given rating rubric, the multilevel LCA approach is specifically intended to answer the following questions: (1) How reliably (precisely) is a teacher's teaching ability measured during any single observation session?; (2) How consistently does a teacher perform over observation sessions?; and (3) For a given teacher, how many observation sessions are required before his/her teaching ability has been measured with a desired level of precision? The last question in particular has relevance for policy, in that multi-rater systems can place heavy financial demands on school districts in terms of deploying a sufficient number of trained raters to meet required number of observation sessions per teacher. The proposed methodology allows for decisions about the required number of observations to be made on a teacher-by-teacher basis, and to be informed by the data collected from each teacher. Two figures are appended.