Although AI transformer models have demonstrated notable capability in automated scoring, it is difficult to examine how and why these models fall short in scoring some responses. This study investigated how transformer models' language processing and quantification processes can be leveraged to enhance the accuracy of automated scoring. Automated…

The author provides information about reliability and areas that educators should examine in determining if an assessment is consistent and trustworthy for use, and how it should be interpreted in making decisions about students. Reliability areas that are discussed in the column include internal consistency, test-retest or stability, inter-scorer…

Objective: To measure the interrater reliability of assessing the frequency of vegetable intake using mobile photos and descriptions. Design: Repeated measures design. Setting: A Midwestern university. Participants: Undergraduate students (N = 165). Measurable Outcome/Analysis: Number of times each of these vegetable subgroups were consumed daily:…

Double-scoring constructed-response items is a common but costly practice in mixed-format assessments. This study explored the impacts of Targeted Double-Scoring (TDS) and random double-scoring procedures on the quality of psychometric outcomes, including student achievement estimates, person fit, and student classifications under various…

The agreement between a wearable inertial sensor (GYKO, G) and the force platform (P) was assessed by evaluating "test-retest" and "inter-rater reliability." Thirty-eight subjects were enrolled; the selected indices of balance were investigated over foot positions and (un)stable conditions. Intraclass correlation coefficient…

Recent curricular interventions in engineering education have focused on encouraging students to develop an entrepreneurial mindset (EM) to equip them with the skills needed to generate innovative ideas and address complex global problems upon entering the workforce. Methods to evaluate these interventions have been inconsistent due to the lack of…

Globalization and increases in the numbers of English language learners have led to a growing demand for English proficiency assessments of spoken language. In this paper, we describe the development of an automatic pronunciation scorer built on state-of-the-art deep neural network models. The model is trained on a bespoke human-rated dataset that…

As ChatGPT continues to reshape student engagement and instructional design, it is crucial to examine its practical implications. This study aims to evaluate the effectiveness of ChatGPT3.5 and ChatGPT4 as potential automated essay scoring (AES) systems. Fifty authentic, student-written annotated bibliographies were evaluated by three human raters…

This study compares how the generative AI (GenAI) large language model (LLM) ChatGPT performs in grading university exams compared to human teachers. Aspects investigated include consistency, large discrepancies and length of answer. Implications for higher education, including the role of teachers and ethics, are also discussed. Three…

Background: Individuals with dementia have communication limitations resulting from cognitive impairments that define the syndrome. Whereas there are numerous cognitive assessments for individuals with dementia, there are far fewer communication assessments. The Profiling Communication Ability in Dementia (P-CAD) was developed to address this gap.…

Despite the widespread use of school-based universal screening systems for social, emotional, and behavioral risk, limited research has examined discrepancies in ratings provided by teachers and their secondary students. Using the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS; teacher report) and mySAEBRS (student report) scores…

Purpose: Assessing social communication and measuring its changes among young autistic children presents significant challenges, particularly when tracking intervention effects within short timeframes. Existing measures, mostly validated in Western contexts, may not be suitable for culturally diverse populations. Addressing this gap, the Social…

Background: The need for the adaptation of instruments to other native languages to promote culture-fairness framed this study. Aim: This article reports on the adaptation of the locally developed Emotional Social Screening Tool for School Readiness (E3SR-R) into isiXhosa. Setting: This adaptation study was conducted in South Africa. Methods: The…

This meta-analytic study investigates the impact of online peer assessment on academic achievement in higher education. By synthesizing 20 effect sizes, we provide a comprehensive understanding of how online peer assessment influences student learning outcomes. The findings reveal a statistically significant positive effect (Hedges's g = 0.672),…

Scoring mathematics exam papers accurately is vital for fostering students' engagement and interest in the subject. Incorrect scoring practices can erode motivation and lead to the development of false self-confidence. Therefore, the implementation of appropriate scoring methods is essential for the success of mathematics education. This study…