Research Findings: This study explores the inter-rater reliability of WaKIDS, Washington State's kindergarten entry assessment (KEA). Specifically, we analyze (1) the extent to which teachers' assessments are in agreement with a master code, (2) how often inaccurate assessment decisions lead to misidentification of school readiness, and (3) whether teacher characteristics influence the accuracy of teachers' assessment decisions. We created four video portfolios of students with diverse identities and skill levels and asked 54 kindergarten teachers across Washington State to evaluate each portfolio using an online survey version of the WaKIDS assessment. Data were analyzed with calculation of agreement percentages, cut-point score analysis, and multiple linear regression analysis. Results indicate that inter-rater reliability varies by developmental domain and is weaker for dual language learners and children with developmental delays. In addition, teachers' professional experiences are significant predictors of agreement level. Practice or Policy: These results suggest a need for targeted professional development experiences for educators and further quantitative and qualitative investigation of kindergarten entry assessments. Ongoing evaluation and improvement of assessment tools across developmental domains and for diverse learners is crucial in order to strengthen the potential of KEAs for supporting all children's learning and growth.