Background: Education decision makers are increasingly expected to use evidence to inform their actions. However, the majority of educational interventions have not yet been studied and it is challenging to produce high quality research evidence quickly enough to influence policy questions. Aims and objectives: We set out to gather evidence on the…

The Pediatric Symptom Checklist-17 (PSC-17) is a popular screening instrument used by parents and clinicians to assess children's behavioral functioning. However, more schools are examining the potential of the PSC-17 as part of a Multi-Tier System of Support framework. To investigate the potential of the PSC-17 in the schools, a sample of 1,779…

This article explores how the analysis of inter-rater discourse can be used to support collective reflective practice in second language (L2) assessment. To demonstrate, a focused case of the discourse between two experienced language teachers as they negotiate assessment decisions on L2 written texts is presented. Of particular interest was the…

A mixed methods action research study was designed to answer three research questions based on inter-rater reliability (IRR) in compliance calls for transition at a state education agency, perceived confidence levels in making and discussing compliance calls, and perceived confidence in sharing transition resources. An innovation based on…

Rater-mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater-mediated assessments using three distinct models. The first model is the observation…

Rater-mediated assessments require the evaluation of the accuracy and consistency of the inferences made by the raters to ensure the validity of score interpretations and uses. Modeling rater response processes allows for a better understanding of how raters map their representations of the examinee performance to their representation of the…

The assessment of English speaking in EFL environments can be inherently subjective and influenced by various factors beyond linguistic ability, including choice of assessment criteria, and even the rubric type. In classroom assessment, the type of rubric recommended for English speaking tasks is the analytical rubric. Driven by three aims, this…

Background/purpose: Generative artificial intelligence (GenAI) is often promoted as a transformative tool for assessment, yet evidence of its validity compared to human raters remains limited. This study examined whether an AI-based rater could be used interchangeably with trained faculty in scoring complex coursework. Materials/methods:…

This study compared the statistical properties of four job analysis task survey response scale types: criticality, difficulty in learning, importance, and frequency. We used nine job analysis studies spanning two fields, medical imaging and allied health professionals, to compare the job analysis scales in terms of variability and interrater…

Research Findings: This study explores the inter-rater reliability of WaKIDS, Washington State's kindergarten entry assessment (KEA). Specifically, we analyze (1) the extent to which teachers' assessments are in agreement with a master code, (2) how often inaccurate assessment decisions lead to misidentification of school readiness, and (3)…

There is growing interest in using measures of teacher applicant quality to improve hiring decisions, but the statistical properties of such measures are poorly understood. We present evidence on structured ratings solicited from teacher applicants' references. We find that the reference ratings capture only one underlying dimension of applicant…

We report on a study bearing implications for ethical learning in schoolchildren during social interaction. The study was conducted as part of a project aimed at promoting ethical learning of socially-oriented values within the context of dialogic education. 172 fourth graders from 7 classes participated in an 8-session series designed to foster…

The aim of teaching observations and post observation feedback in higher education is to support teachers to reflect on and improve their teaching. Yet, our understanding of tutors' (observers') and teachers' (observees') capacities for capitalising on these feedback opportunities is limited and there is little empirically derived advice for…

The observation systems being used in all 50 states require administrators to learn to accurately and reliably score their teachers' instruction using standardized observation systems. Although the literature on observation systems is growing, relatively few studies have examined the outcomes of trainings focused on developing administrators'…

In this study, it was aimed to develop a rubric for evaluating the critical writing skills of primary school fourth-grade students. The research was designed in quantitative research method and scanning design. The study group of the research consists of 215 students studying in the fourth grade of five different primary schools determined by the…