The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance…

For decades, evaluators have noted that it is difficult for stakeholders to accept negative evaluation results (Carter, 1971; Taut & Brauns, 2003). There is a need for additional research on evaluation to better understand when and why stakeholders reject negative or critical evaluation findings. Drawing on social identity theory (SIT), the…

Business Model Canvas (BMC) is a business model that must be mastered by students in the Bioentrepreneurship course as an initial provision for entering the entrepreneurial world, while in compiling Business Model Canvas (BMC) systematic thinking skills are needed. This study aims to provide an assessment instrument to measure students' system…

Autistic youth are more likely to experience depression than their non-autistic peers, yet research on risk and protective factors to depression in this population is limited. Behavioral activation (i.e., prioritizing and engaging in meaningful activities), including social activities, is an important mechanism in the pathway to depression in…

Early literacy screening tools are often a critical component of schools' intervention frameworks. As the use of screening tools becomes more prevalent for identifying students who may be at risk for learning difficulties and in need of additional instruction, it is important to continuously evaluate teachers' implementation of these screening…

Turnover in the teacher workforce imposes significant costs to schools, both in terms of student achievement and the time and expense required to recruit and train new staff. This paper examines the potential for structured ratings of teacher applicants, solicited from their professional references, to inform hiring decisions through the selection…

The agreement between raters is examined within the scope of the concept of "inter-rater reliability". Although there are clear definitions of the concepts of agreement between raters and reliability between raters, there is no clear information about the conditions under which agreement and reliability level methods are appropriate to…

Purpose: Secretion bubbling on the superior aspect of the velopharyngeal (VP) valve typically occurs with a small VP opening during production of oral pressure consonants. The use of high-speed nasopharyngoscopy has shown correlation between the bubbling frequency and the acoustics captured with the nasal microphone of the nasometer. The purpose…

Purpose: Information on reliability of outcome measures used to assess the effectiveness of interventions in dysphagia rehabilitation is lacking, particularly when used by different research groups. Here, we report on reliability of the penetration- aspiration scale (PAS) and temporal and clearance measures, determined using videofluoroscopy.…

Creativity research crucially relies on creativity evaluations by external raters, but it is not clear what properties characterize good raters. In the present study, we investigated whether rater personality and rater creativity are related to discernment (i.e., the ability to distinguish creative from uncreative responses) when evaluating…

Parents offer unique information and are recognized as the main informants about children's social emotional functioning across time and situations. Inter-parental agreement was analysed with 100 preschoolers (M[subscript age] = 54.94 months) being rated independently by their mothers and fathers, using the Portuguese version of the Preschool and…

Purpose: Postswallow voice abnormality is often assumed to indicate the presence of material in the laryngeal airway (MIA), but prior research has not shown definitive evidence of a causal relationship. This study investigated if endoscopically confirmed MIA (prandial material or secretions) generates specific voice quality attributes that can be…

The current review examined the use of self-report measures in autistic individuals in the context of psychiatric assessments. It focused on inter-rater agreement, internal consistency, test-retest reliability, and criterion validity with clinical diagnoses. It also gathered information on constructs measured, the nature of the samples, and the…

Background: The concept of reliability is central to conducting--and understanding--research in Psychology. Students' understanding of concepts are strengthened when they learn by applying concepts. Objective: This article describes initial evidence of an activity for teaching reliability. Method: Students watched a short video of a staged bank…

Fitting wrongly specified models to observed data may lead to invalid inferences about the model parameters of interest. The current study investigated the performance of the posterior predictive model checking (PPMC) approach in detecting model-data misfit of the hierarchical rater model (HRM). The HRM is a rater-mediated model that incorporates…