This study assesses the reliability of two observation rubrics, one in math and the other in science, and documents how the rubric data were used to inform a teacher education program. Classroom observations are typically considered essential for assessing teaching practice, yet many popular observation frameworks, while comprehensive in aim, do not appropriately capture key features of teaching valued by teacher education programs. Many of these tools do not attend to issues of equity, humanizing pedagogy, and thus, social justice. We report on the development of two observation rubrics--secondary math and science--that embody the aims and values of our teacher education program, specifically, equity and humanizing pedagogy, and the results of our examination of the reliability of ratings of teaching practice generated using these rubrics. We discuss the various sources of measurement error and the implications for further developing and using the observation rubric in our program.