In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze…

In programming, one problem can usually be solved using different logics and constructs but still producing the same output. Sometimes students get marked down inappropriately if their solutions do not follow the answer scheme. In addition, lab exercises and programming assignments are not necessary graded by the instructors but most of the time…

The Creative Solution Diagnosis Scale (CSDS) is a 30-item scale based on a core of four criteria: Relevance & Effectiveness, Novelty, Elegance, and Genesis. The CSDS offers potential for the consensual assessment of functional product creativity. This article describes an empirical study in which non-expert judges rated a series of mousetrap…

The reliability of data is a critical issue in decision-making for practitioners in the school. Percent Agreement and Cohen's kappa are the two most widely reported indices of inter-rater reliability, however, a recent Monte Carlo study on the reliability of multi-category scales found other indices to be more trustworthy given the type of data…

To find out whether the education community shares a collective understanding about how students should be evaluated, we surveyed 202 educators (from all grade levels) and scientists attending assessment workshops (Pennsylvania, California, and Massachusetts) or judging a national student competition (Washington, DC). The educators and scientists…

Across the nation, states and districts are in the process of building better teacher evaluation systems that not only identify highly effective teachers but also systematically provide data and feedback that can be used to improve teacher practice. "A Practical Guide to Designing Comprehensive Teacher Evaluation Systems" is a tool…

Many researchers and clinicians believe that the therapeutic relationship is essential in bringing about clinical change. Empirical research to support this contention is scarce in part due to the difficulty of specifying and measuring theoretically derived mechanisms of change and the important dimensions of the client-therapist relationship.…

The Autism Observation Scale for Infants (AOSI) was developed to detect and monitor early signs of autism as they emerge in high-risk infants (all with an older sibling with an autistic spectrum disorder). Here we describe the scale and its development, and provide preliminary data on its reliability. Inter-rater reliability both for total scores…