Classroom observation is an important component of teacher evaluation systems. Most states are implementing systems that assign a composite score to each teacher based on weights assigned to several different measures. Policy discussions often address this weighting, with many states adopting formulas with high weights for the summative scores from observations conducted by school principals or other administrators. Given the weighting of this one measure, it is important to ensure the validity of observation rubrics and equitability of the resulting teacher rankings. This paper addresses the problem of observation scores being affected by characteristics of the students in the class being taught, and explores the problem in two phases: (1) the demonstration of an alternative to the common (often implicit) assumption that the components or elements of the observation score are measuring a single underlying concept and all have the same relevance to any personnel decision that is to be based on the evaluation score; and (2) showing how the multifaceted nature of observations can be used to better understand how observation scores are affected by class characteristics. Most observation rubrics in wide use, such as the Framework for Teaching (FFT), have been designed, and are used as universal instruments. They are applied without any modifications in classrooms at different grades levels, in different subjects, and with students of widely different abilities, backgrounds, and resources. This implicit assumption of instrument invariance is questionable. The nature of the invariance may be different for different components of the instrument. The goal of the analyses in this report is to provide a stronger basis for making observations a useful part of teacher evaluation by addressing these facets of variability. Adapting the underlying meaning of instrument categories to specifics of various classrooms may require more experience than can be obtained in the course of a single academic study or in one or two rounds of annual observation for evaluation purposes. [This paper was presented at the Annual Meeting of the Association for Education Finance and Policy (AEFP) on February 27, 2015, in Washington, DC.]