Purpose: The current standard for clinical diagnosis of childhood apraxia of speech (CAS) is expert clinician judgment. The psychometric properties of this standard are not well understood; however, they are important for improving clinical diagnosis. The purpose of this study is to determine the extent to which experts agree on the clinical…

Background: The concept of reliability is central to conducting--and understanding--research in Psychology. Students' understanding of concepts are strengthened when they learn by applying concepts. Objective: This article describes initial evidence of an activity for teaching reliability. Method: Students watched a short video of a staged bank…

Despite a tremendous increase in the use of video for conducting research in classrooms as well as preparing and evaluating teachers, there remain notable challenges to using classroom videos at scale, including time and financial costs. Recent advances in artificial intelligence could make the process of analyzing, scoring, and cataloguing videos…

The Canadian Agility and Movement Skill Assessment (CAMSA) is a valid and reliable circuit-based test of motor competence which can be used to assess children's skills in a live or recorded performance and then coded. We aimed to analyze the intrarater reliability of the CAMSA scores (total, time, and skill score) and time measured, by comparing…

The Occupation-Centered Intervention Assessment (OCIA) was developed as a reflective tool for students to improve their comprehension of occupation-centered practice. Finding new and innovative ways to incorporate occupation-centered assignments can serve as a strategy to develop student integration of occupation-centered practice and allow…

The Play Cycle Observation Method (PCOM) is an observational tool developed to focus on the process of play and has shown good reliability when watching videos of children playing. This study piloted use of the PCOM in 'real time' in a pre-school setting where 3-year-old children play. The results from two independent observers not familiar with…

Several observation protocols in different theoretical frameworks and components have been designed and validated by teacher trainers and professional development providers to capture and categorize observational data on the characteristics and level of inquiry in science practices. However, certain constraints limit their wide use, such as the…

The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a…

Thin slices ratings (i.e., ratings based on first impressions) have yielded intriguingly accurate results in various domains. Among other, researcher have applied the thin slices technique to assess instructional quality, showing that teacher-student interactions can be reliably inferred by just very short snippets of classroom instruction. The…

The assessment of English speaking in EFL environments can be inherently subjective and influenced by various factors beyond linguistic ability, including choice of assessment criteria, and even the rubric type. In classroom assessment, the type of rubric recommended for English speaking tasks is the analytical rubric. Driven by three aims, this…

Evaluation of music performance in competitive contexts often produces discrepancies between the expert judges. These discrepancies can be reduced by using appropriate rubrics that minimise the differences between judges. The objective of this study was the design and validation of an analytical evaluation rubric, which would allow the most…

This study aims to explore the validity of measuring joint attention through gaze coordination in computer-supported collaborative learning (CSCL) research. Gaze coordination, aligning visual attention in social contexts, aids comprehension and communication. Many CSCL researchers use gaze coordination to gauge joint attention quantitatively.…

The goal of this paper is twofold. First, the paper clarifies and elaborates on an important theoretical construct called orientation with respect to understanding in mathematics, which denotes the degree to which students exhibit an inclination towards and demonstrate an earnest concern for understanding in mathematical learning. Second, the…

This paper discusses the challenges and lessons learned from conducting observations to measure the quality of classroom practice for a large-scale study of elementary teachers' mathematics instruction. Specifically, this paper shares our process for obtaining valid data for quality of elementary mathematics instruction; what we learned can inform…

This study utilized a quantitative research approach with a survey design to determine whether a sample of K-8 principals in the state of Illinois, that rated teachers in a set of classroom instructional videos, exhibited inter-rater reliability since they all received the Illinois state-approved training and certification for teacher evaluators.…