Doctors' interactional competencies play a crucial role in patient satisfaction, well-being, and compliance. Accordingly, it is in medical schools' interest to select candidates with strong interactional abilities. While Multiple Mini Interviews (MMIs) provide a useful context to assess such abilities, the evaluation of candidate performance…

This study examined the consistency between instructor ratings of learner-generated summaries and those estimated by a large language model (LLM) on summary content checklist items designed for undergraduate second language (L2) writing instruction in Japan. The effects of the LLM prompt design on the consistency between the two were also explored…

The International English Language Testing System (IELTS) is a high-stakes exam where Writing Task 2 significantly influences the overall scores, requiring reliable evaluation. While trained human raters perform this task, concerns about subjectivity and inconsistency have led to growing interest in artificial intelligence (AI)-based assessment…

In this study, various statistical indexes of agreement were calculated using empirical data from a group of evaluators (n = 45) of early childhood teachers. The group of evaluators rated ten fictitious teacher profiles using the North Carolina Teacher Evaluation Process (NCTEP) rubric. The exact and adjacent agreement percentages were calculated…

Spinal alignment can be assessed without radiation using three-dimensional ultrasound imaging (3DUS). Reliable measurements could inform the ideal arm position for scoliosis radiographs. This study determined the inter-evaluator reliability of axial vertebral rotation (AVR) measurements and sagittal curve angles in healthy females from 3DUS spinal…

Background: People with aphasia (PWA) typically exhibit deficits in spoken discourse. Discourse analysis is the gold standard approach to assess language deficits beyond sentence level. However, the available discourse assessment tools are biased towards English and European languages and Western culture. Additionally, there is a lack of consensus…

This meta-analytic study investigates the impact of online peer assessment on academic achievement in higher education. By synthesizing 20 effect sizes, we provide a comprehensive understanding of how online peer assessment influences student learning outcomes. The findings reveal a statistically significant positive effect (Hedges's g = 0.672),…

We aimed to calculate interrater reliability of the Test of Gross Motor Development--Third Edition (TGMD-3) after raters reached a consensus regarding measurement criteria. Three raters measured the fundamental movement skills of 25 children on the TGMD-3 at two different times: (a) once when simply following the measurement criteria in the TGMD-3…

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance…

Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to…

In this collective essay the authors consider the nature and consequences of reading and researching across difference in an international and intergenerational team, whose core members are focused on understanding how curriculum operates and the nature of textbook representation of diversity in Mainland China, Hong Kong, Taiwan, and Macau.…

Purpose: The aim of this study was to describe changes in speech intelligibility and interrater and intrarater reliability of naive listeners' ratings of words produced by young children diagnosed with childhood apraxia of speech (CAS) over a period of motor-based intervention (dynamic temporal and tactile cueing [DTTC]). Method: A total of 120…

In this study, it was aimed to develop a rubric to evaluate the creative story writing skill levels of seventh grade secondary school students. The research was designed in quantitative research method and survey model. In the research, convenience sampling technique was used and 270 students studying at the seventh grade level of secondary school…

In response to the call for promoting three-dimensional science learning (NRC, 2012), researchers argue for developing assessment items that go beyond rote memorization tasks to ones that require deeper understanding and the use of reasoning that can improve science literacy. Such assessment items are usually performance-based constructed…

Cohen's kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss' generalized kappa. Fleiss' generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among…