While the field advocates for a diversified and comprehensive professional role (National Association of School Psychologists, 2020), school psychologists have long spent most of their time in assessment-related activities (Farmer et al., 2021), averaging about eight cognitive evaluations monthly (Benson et al., 2020). Assessment practices have…

In Part 1 of this series, the authors briefly reviewed some challenges inherent in using standardized tools with students who are not well represented in norming data. To help readers clearly conceptualize the framework steps, the authors present two case studies that showcase how a nonstandardized approach to assessment can be individualized to…

Medical schools are required to assess and evaluate their curricula and to develop exam questions with strong reliability and validity evidence, often based on data derived from statistically small samples of medical students. Achieving a large enough sample to reliably and validly evaluate courses, assessments, and exam questions would require…

The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal…

While score comparability across test forms typically relies on common (or randomly equivalent) examinees or items, innovations in item formats, test delivery, and efforts to extend the range of score interpretation may require a special data collection before examinees or items can be used in this way--or may be incompatible with common examinee…

Test input has often been taken as a given in test design practice. Nearly all guides for test designers provide extensive coverage of how to design test items but pay little attention to test input. This paper presents the case that test input plays a crucial role in designing tests of soft skills that have rarely been assessed in existing tests.…

There are many valuable sources of evaluation data, including -- but not limited to -- professional learning participants. In the authors' work on leadership development and organizational learning for Tulsa Public Schools in Oklahoma, they regularly ask educators to share feedback and perceptions of usefulness of their professional learning. The…

There has been increasing interest in applying integrative data analysis (IDA) to analyze data across multiple studies to increase sample size and statistical power. Measures of a construct are frequently not consistent across studies. This article provides a tutorial on the complex decisions that occur when conducting harmonization of measures…

The UK census authorities have proposed guidance for the 2021 census indicating that the sex question may be answered according to subjective gender identity. This raises issues about the measurement of sex and gender identity which other data collection exercises are also contending with. This paper addresses the questions that have arisen…

In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically defensible function that adequately summarizes both individual change as well as the average response. The module…

In this digital ITEMS module, Nikole Gregg and Dr. Brian Leventhal discuss strategies to ensure data visualizations achieve graphical excellence. Data visualizations are commonly used by measurement professionals to communicate results to examinees, the public, educators, and other stakeholders. To do so effectively, it is important that these…

Every day, people use data to make decisions that affect their personal and professional lives, trusting that the data are correct. Many times, however, the data are inaccurate, as a result of a flaw in the design or methodology of the survey used to collect the data. Researchers agree that only questions that are clearly worded, unambiguous, free…

The Domain Five Observation Instrument (DFOI) is a competency-based observation instrument recommended for sport leaders or researchers who wish to evaluate coaches' instructional behaviors. The DFOI includes 10 behavior categories and four timed categories that encompass 34 observable instructional benchmarks outlined in domain five of the…

"Data-drive instruction can distort the way reading is taught, harming the students who need high-quality instruction the most," Susan B. Neuman concludes from her research team's two years of observation in nine low-income New York City schools. She describes how some students are reminded that they are "failures" every day by…

TIMSS (Trends in International Mathematics and Science Study) is a long-standing international assessment of mathematics and science at the fourth and eighth grades that has been collecting trend data every four years since 1995. About 70 countries use TIMSS trend data for monitoring the effectiveness of their education systems in a global…