Rater-mediated activities in educational research occur when an expert judge or rater utilizes an instrument to judge persons or items and generates scale scores. Scale scores are from a subjective judgment and must undergo a quality control measure called rating quality. Rating quality in this study is broadly defined as the extent to which…

Reliability that can be proved by numeric indicators in quantitative studies has become a very discussible issue. The reason for this is to be thought that in qualitative researches, reliability is not based on positive perspective and those forming reliability criteria is difficult. However, for testing the reliability of a qualitative study or…

While skin color is relevant and important in the Latinx community, as it is associated with colorism, little is known about how often it is measured or the best way to measure it. This article presents results from two studies examining these key concerns in three prominent journals, where Latinx research is typically published (i.e., the…

There has recently been a surge of interest in improving the replicability of second language (L2) research. However, less attention is paid to replicability in the context of L2 meta-analyses. I argue that conducting interrater reliability (IRR) analyses is a key step toward improving the replicability of L2 meta-analyses. To that end, I first…

Purpose: Analysis of connected speech in the field of adult neurogenic communication disorders is essential for research and clinical purposes, yet time and expertise are often cited as limiting factors. The purpose of this project was to create and evaluate an automated program to score and compute the measures from the Quantitative Production…

Teacher preparation programs are increasingly expected to use data on preservice teacher (PST) skills to drive program improvement and provide targeted supports. Observational ratings are especially vital, but also prone to measurement issues. Scores may be influenced by factors unrelated to PSTs' instructional skills, including rater standards.…

Systematic reviews of single-case experimental research (SCER) in special education often use the What Works Clearinghouse (WWC) Standards to assess the methodological rigor of studies within a given literature base. While significant changes were made between the two most recent versions of the WWC standards, no research to date has evaluated the…

This study explores vocal development as an early marker of autism, focusing on canonical babbling rate and onset, typically established by 7 months. Previous reports suggested delayed or reduced canonical babbling in infants later diagnosed with autism, but the story may be complicated. We present a prospective study on 44 infants later diagnosed…

We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found…

Although there are claims that ChatGPT, an AI-based language model, is capable of assessing the writing of L2 learners accurately and consistently in the classroom, a number of recent studies have shown discrepancies between AI and human raters. Furthermore, there is a lack of studies investigating the intrareliability of ChatGPT scores.…

Rivet Education developed a Professional Learning Partner Guide (PLPG), a searchable database of learning providers with expertise in the adoption and implementation of High-Quality Instructional Materials (HQIM), in response to the lack of resources and guidance available to support districts with evaluation of professional learning (PL)…

Background/purpose: This study addresses the challenge of engaging students in meaningful assessment processes within mathematics education, particularly in pre-university preparatory courses. Peer assessment, a growing pedagogical practice, can enhance learning outcomes, promote critical thinking, and improve collaborative skills. However,…

This study examined the new self-report version of the Strengths and Difficulties Questionnaire (SDQ-S), SDQ-Kids, in primary school children regarding internal consistency, teacher-child agreement, and validity. Data from 2,655 children in Grades 1 to 3 and their teachers were analyzed. Children completed SDQ-Kids, previously piloted (n = 896),…

Self-assessment and peer assignment have clear advantages for the training of responsible, critical, and reflective professionals. In recent years, self and peer evaluation have also been shown to be even more effective than lecturer evaluation when we assure anonymity through online platforms learning tools. Therefore, self and peer assessments…

For autism spectrum disorder (ASD) in adults there are several diagnostic instruments available with a need for consideration of the psychometric properties. This study aimed to conduct a first psychometric evaluation of a new diagnostic ASD instrument, the NIDA (Dutch Interview for Diagnostic assessment of ASD in adults) in 90 adult males without…