Teacher evaluation systems often include classroom observations in which raters use rating scales to evaluate teachers' effectiveness. Recently, researchers have promoted the use of multifaceted approaches to investigating reliability using Generalizability theory, instead of rater reliability statistics. Generalizability theory allows analysts to…

Using a newly created teacher evaluation instrument, Inter-rater Reliability (IRR) analyses were conducted on four teacher videos as a means to establish instrument reliability. Raters included 42 principals and assistant principals in a southern US school district. The videos used spanned the teacher quality spectrum and the IRR findings across…

Background/purpose: ChatGPT, a recent form of AI-based language model, have garnered interest among people from diverse backgrounds with its immersive capabilities. Using ChatGPT to support or generate scientific research has also created an ongoing debate over its advantages versus risks. The present study aimed to conduct an AI-enabled research…

This study employed a mixed-methods approach to examine how rater performance develops during a semester-long rater certification program for an English as a Second Language (ESL) writing placement test at a large US university. From 2016 to 2018, we tracked three groups of novice raters (n = 30) across four rounds in the certification program.…

This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on "G studies"--a method of breaking down measurement variance--and "D studies"--a predictive study of the impact on reliability when…

Background/Context: Recently there has been increased emphasis on disaggregating students' experiences collected through surveys based on student identities (Arredondo, 2016). Yet demographic data collection is problematic by nature, often reducing the complexity of an individual's racial, gender, or sexual identity to a single category. Racial…

This study explored the impact of mild bilateral or unilateral hearing loss on auditory, social, and behavior skills in early school-aged children. Thirty-two children (aged 5-9 years) were evaluated with parent and teacher questionnaires. Most outcomes were within the range of expected scores. However, functional auditory skills were below…

The aim of this study is to develop and apply a rubric to evaluate the solutions proposed for questions about electromagnetic induction belonging to university second year pre-service teachers. In this study which has pretest-posttest quasi-experimental design with control group, teaching of the topic of electromagnetic induction was applied to…

Because many websites claim to provide information about evidence-based practices (EBPs), consumers must know the information and practices are based upon quality research. Practitioners may intend to locate trustworthy online sources providing EBPs, but if those sources are not easy to navigate and lack implementation resources (i.e., are…

The purpose of this study is to develop and validate instrument-specific rating scales to evaluate the classroom performances of middle and high school students. The study is guided by the following research questions: 1. What does Rasch measurement analysis reveal about the psychometric properties (i.e., validity and reliability) of items,…

Purpose: Recording young children's vocalizations through wearables is a promising method to assess language development. However, accurately and rapidly annotating these files remains challenging. Online crowdsourcing with the collaboration of citizen scientists could be a feasible solution. In this article, we assess the extent to which citizen…

As an innovative evaluation tool, peer assessment is essential in Massive Open Online Courses (MOOCs). In both formative and summative peer assessments in MOOCs, providing reliable feedback is crucial in enhancing learning outcomes. Peer assessment has been highlighted as a reliable tool in both traditional classrooms and small-scale online…

Identifying and describing children's play behaviours is an important component of evaluating child development. The Behaviour Mapping Schedule is a direct observational tool which aims to describe and quantify children's play behaviours but is yet to undergo reliability testing. This study aimed to determine the intra- and inter-rater reliability…

The purpose of the present study was to examine the relationship between the PEAK Direct Training Pre-assessment (PEAK-DT-PA) and the PEAK Generalization Pre-assessment (PEAK-G-PA) with the corresponding indirect assessments based on parent and therapist reports. Participants were administered the PEAK-DT-PA and PEAK-G-PA. Parents and therapist…

Purpose: While flexible endoscopic evaluation of swallowing (FEES) is a common clinical procedure used in the head and neck cancer (HNC) population, extant outcome measures for FEES such as bolus-level penetration-aspiration and residue scores are not well suited as global patient-level endpoint measures of dysphagia severity in cooperative group…