Forced-choice questionnaires can prevent faking and other response biases typically associated with rating scales. However, the derived trait scores are often unreliable and ipsative, making interindividual comparisons in high-stakes situations impossible. Several studies suggest that these problems vanish if the number of measured traits is high.…

The Survey of Physics Reasoning on Uncertainty Concepts in Experiments (SPRUCE) was designed to measure students' proficiency with measurement uncertainty concepts and practices across ten different assessment objectives to help facilitate the improvement of laboratory instruction focused on this important topic. To ensure the reliability and…

The State of Washington requires school districts to file court petitions on students with excessive unexcused absences. The "Washington Assessment of Risks and Needs of Students" (WARNS), a self-report screening instrument developed for use by high school and juvenile court personnel in such situations, purports to measure six facets of…

Situational Judgement Questions (SJQs) measuring process quality were included in the OECD Starting Strong Teaching and Learning International Survey 2018 (TALIS Starting Strong 2018) to address concerns of self-report bias in large-scale international surveys. These SJQs provide the staff in early childhood education and care with situations…

The long-term goal of the State Systemic Improvement Plan (SSIP) and other federal and state early intervention and early childhood education initiatives is improved child and family outcomes. States play a critical role in supporting practitioners in the use of evidence-based practices to improve child and family outcomes. When practitioners…

Objective: To develop a retail grocery instrument with weighted scoring to be used as an indicator of the food environment. Participants/Setting: Twenty six retail food stores in low-income areas in California. Intervention: Observational. Main Outcome Measure(s): Inter-rater reliability for grocery store survey instrument. Description of store…

Research Findings: This study examined the validity and reliability of the Classroom Assessment Scoring System (CLASS; R. C. Pianta, K. M. La Paro, & B. K. Hamre, 2008) in Finnish kindergartens. A pair of trained observers used the CLASS to observe 49 kindergarten teachers (47 female, 2 male) on two different days. Questionnaires measuring…

Author replies to article TM 500 559. (MB)

A two-stage testing procedure, a routing test followed by one of several alternative second-stage tests, is studied in the situation where the purpose is measurement, not classification. Models are developed, examined, and compared with conventional tests and up-and-down procedures. (DG)

Measurement problems which jeopardize the reliability and validity of competency-based writing assessments are analyzed. Methods to stabilize rating criteria and readers' application of them are necessary. Most writing assessment programs use guidelines from norm-referenced test methodology. Use of this method of criteria application based on…

Weighting test scores by appropriateness of confidence, has almost without exception raised the reliability of test scores. Greater gains appear to occur for the less reliable tests, but that is at least partly because the more reliable a test is to begin with, the more difficult it is to improve it. If confidence-testing allows us to weight…

A procedure for determining the reliability of an examinee knowing k out of n possible multiple choice items given his or her performance on those items is presented. Also, a scoring procedure for determining which items an examinee knows is presented. (Author/JKS)

When raters assign different scores to a performance task, a method for resolving rating differences is required to report a single score to the examinee. Recent studies indicate that decisions about examinees, such as pass/fail decisions, differ across resolution methods. Previous studies also investigated the interrater reliability of…

This literature review summarizes what is currently known about the agreement among six measures of writing skills. Three of these methods involve the application of human judgment in scoring or rating a piece of writing: holistic, analytical, and primary trait scoring. Two methods involve anatomical or taxonomic analysis of a piece of writing:…