This paper reports on a case study which identifies and illustrates sources of difference in agreement in relation to reliability in a context of quantitative content analysis of a transcript of an online asynchronous discussion (OAD). Transcripts of 10 students in a month-long online asynchronous discussion were coded by two coders using an…

The standardization of ADHD ratings in adults is important given their differing symptom presentation. The authors investigated the agreement and reliability of rater standardization in a large-scale trial of atomoxetine in adults with ADHD. Training of 91 raters for the investigator-administered ADHD Rating Scale (ADHDRS-IV-Inv) occurred prior to…

Among the many possible aspects to assessing writing, one of the most problematic is coherence. The problems with marking coherence arise because it is by nature subjective. However, the reasonable probability of several readers reaching a consensus concerning the coherence of a text suggests that it may be possible to assign relatively reliable…

This article contends that the modern descendant of B. F. Skinner's experimental analysis of behavior, "behavior analysis," and as well his 1957 masterwork "Verbal Behavior," have rarely if ever been seriously contemplated by applied linguists for possible contributions to the field. Rather, a pat literature of dismissal has developed that…

A brief instrument to assess risk of injury was applied retrospectively for 2 years and prospectively for 1 year to all people living in a large ICF/MR. Results suggest that the percentage of people who experienced an injury significantly increased across the levels of increasing risk indicated by the assessment. Furthermore, people who…

The research literature reveals that there are many factors that influence the consistency of assessors' or examiners' judgements. One issue that has not been considered is whether National Vocational Qualifications assessors' consistency of judgement is affected by different types of evidence. In this article, 15 Customer Service and 12 Assessor…

Purpose: To explore the utility of time-interval analysis for documenting the reliability of coding social communication performance of children in classroom settings. Of particular interest was finding a method for determining whether independent observers could reliably judge both occurrence and duration of ongoing behavioral dimensions for…

Background: From 2002 onwards, initiatives and first steps for the project International Comparative Analysis of Learning and Teaching (ICALT) have been taken by the inspectorates of education in England, Flanders (Belgium), Lower Saxony (Germany) and The Netherlands. The inspectorates of education in these European countries reviewed the results…

Autism susceptibility genes likely have effects on continuously distributed autism-related traits, yet few measures of such traits exist. The Broader Phenotype Autism Symptom Scale (BPASS), developed for use with affected children and family members, measures social motivation, social expressiveness, conversational skills, and flexibility. Based…

The authors report the development and initial psychometric evaluation of gender-specific brief screening instruments to identify undetected psychiatric impairment on incarceration. Women and men completed the Correctional Mental Health Screen (CMHS), a 56-item screen derived from validated measures. Representative subsamples completed structured…

This article empirically validates an existing content analysis scheme and addresses a main concern of researchers about text-based, online transcripts in the form of code-recoding by mapping our scheme to the practical inquiry, cognitive presence model's five phases directly to realise higher-order thinking or critical thinking aspects for our…

This action research study investigated the impacts of teaching strategies based on the balanced literacy approach on story grammar acquisition of three Turkish students with hearing loss. Data were collected from students' work, archival information, interviews and observations during lessons which were audio- and video-recorded. The pre- and…

The current research was conducted to investigate the validity of automated essay scoring (AES) by comparing group mean scores assigned by an AES tool, IntelliMetric [TM] and human raters. Data collection included administering the Texas version of the WriterPlacer "Plus" test and obtaining scores assigned by IntelliMetric [TM] and by…

The use of online rater self-training is growing in popularity and has obvious practical benefits, facilitating access to training materials and rating samples and allowing raters to reorient themselves to the rating scale and self monitor their behaviour at their own convenience. However there has thus far been little research into rater…

Behaviorally Anchored Rating Scales (BARS) were developed to score responses from a previously designed police written communication test that lacked reliability. Rating scales for each of the 9 dimensions of the test consisted of the scale definition and a 5-point continuum, with the scores of 5, 3, and 1 defined by specified behavioral…