Dynamic measurement modeling (DMM) is a recent framework for measuring developing constructs whose manifestation occurs after an assessment is administered (e.g., learning capacity). Empirical studies have suggested that DMM may improve consequential validity of test scores because DMM learning capacity estimates were shown to be much less related…

Data literacy, or students' abilities to understand, interpret, and think critically about data, is an increasing need in K-16 science education. Ocean Tracks College Edition (OTCE) sought to address this need by creating a set of learning modules that engage students in using large-scale, professionally collected animal migration and physical…

Reliability is the consistency of a set of scores that are designed to measure the same thing. Reliability is a statistical property of scores that must be demonstrated rather than assumed.

What follows is a practical guide for establishing the validity of a survey for research purposes. The motivation for providing this guide is our observation that researchers, not necessarily being survey researchers per se, but wanting to use a survey method, lack a concise resource on validity. There is far more to know about surveys and survey…

In April 2021, The Association for Assessment and Research in Counseling Executive Council commissioned a time-referenced task group to revise the Responsibilities of Users of Standardized Tests (RUST) Statement (3rd edition) published by the Association for Assessment in Counseling (AAC) in 2003. The task group developed a work plan to implement…

Chalmers recently published a critique of the use of ordinal a[alpha] proposed in Zumbo et al. as a measure of test reliability in certain research settings. In this response, we take up the task of refuting Chalmers' critique. We identify three broad misconceptions that characterize Chalmers' criticisms: (1) confusing assumptions with…

Validity is broadly defined as how well something measures what it's supposed to measure. The reliability and validity of scores from assessments are two concepts that are closely knit together and feed into each other.

The Social and Emotional Learning Program Quality Assessment (SEL PQA) is an efficient and effective, research-validated observational tool for assessing adult youth leader practices that support social and emotional learning. Intended to be used for programs serving young people in grades K-12, the instrument builds on the research-validated…

Many large universities, community colleges and some smaller four-year colleges are turning to hybrid or online instruction for remedial and entry level mathematics courses, often assessed using online exams in a proctored computer lab environment. Faculty face the task of choosing questions from a publisher's text bank with very little, if any,…

Background/Context: Pass/fail standardized exams frequently selectively rescore failing exams and retest failing examinees. This practice distorts the test score distribution and can confuse those who do analysis on these distributions. In 2011, the Wall Street Journal showed large discontinuities in the New York City Regent test score…

The digital transition in educational testing has introduced many new opportunities for technology to enhance large-scale assessments. These include the potential to collect and use log data on test-taker response processes routinely, and on a large scale. Process data has long been recognised as a valuable source of validation evidence in…

As the research base for school crisis intervention and prevention expands, the need for well-developed tools to assess school readiness in the event of a crisis increases. This paper describes how the Comprehensive Crisis Plan Checklist (CCPC) was updated to reflect advances in crisis management and crisis planning. An extensive literature search…

Generalizability theory is a modern, powerful, and broad framework used to assess the reliability, or dependability, of measurements. While there exist classic works that explain the basic concepts and mathematical foundations of the method, there is currently a lack of resources addressing computational resources for those researchers wishing to…

One of the areas subsumed under the data-based decision making and accountability practice identified in the National Association of School Psychologists' (NASP) "Model for Integrated School Psychological Services" is to collect information on psychological and educational variables to make decisions at a number of levels of service…

This inquiry is focused on three indicators of the precision of measurement--conditional on fixed values of ?, the latent variable of item response theory (IRT). The indicators that are compared are (1) The traditional, conditional standard errors, s(eX|?) = CSEM; (2) the IRT-based conditional standard errors, s[subscript irt](eX|?)=C[subscript…