For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

Ensuring a credible literature base is essential for all research fields. One element of this relates to the replicability of published work, which is the probability that the results of an original study would replicate in an independent investigation. A critical feature of replicable research is that the sample size of a study is sufficient to…

In a 1984 essay, Benjamin Bloom, an educational psychologist at the University of Chicago, asserted that tutoring offered "the best learning conditions we can devise" and that tutors could raise student achievement by two full standard deviations--or, in statistical parlance, two "sigmas." The influence of Bloom's two-sigma…

Economic efficiency demands accurate assessment of individual ability for selection purposes. This study investigates Classical Test Theory (CTT) and Item Response Theory (IRT) for estimating true ability and ranking individuals. Two Monte Carlo simulations and real data analyses were conducted. Results suggest a slight advantage for IRT, but…

Multisite cluster randomized trials (MCRTs), in which, the intermediate-level clusters (e.g., classrooms) are randomly assigned to the treatment or control condition within each site (e.g., school), are among the most commonly used experimental designs across a broad range of disciplines. MCRTs often align with the theory that programs are…

Analysis of the differential treatment effects across targeted subgroups and contexts is a critical objective in many evaluations because it delineates for whom and under what conditions particular programs, therapies or treatments are effective. Unfortunately, it is unclear how to plan efficient and effective evaluations that include these…

The population relationship between coefficient alpha and scale reliability is studied in the widely used setting of unidimensional multicomponent measuring instruments. It is demonstrated that for any set of component loadings on the common factor, regardless of the extent of their inequality, the discrepancy between alpha and reliability can be…

Imputation or likelihood-based approaches to handle missing data assume the data are missing completely at random (MCAR) or missing at random (MAR). However, little research has examined the missingness pattern before using these imputation/likelihood methods. Three missingness mechanisms -- MCAR, MAR, and not missing at random (NMAR) -- can be…

To date, there have been four responses (Dumas & Edelsbrunner, "Educational Psychology Review," 35, 48, 2023; Grosz, "Educational Psychology Review," 35, 57, 2023; Mayer, "Educational Psychology Review," 35, 64, 2023; Zitzmann et al., "Educational Psychology Review," 35, 65, 2023) to the Brady et al.…

Recruiting participants for studies of early-life longitudinal development is challenging, often resulting in practical upper bounds in sample size and missing data due to attrition. These factors pose risks for the statistical power of such studies depending on the intended analytic model. One mitigation strategy is to increase measurement…

Evidence-based health interventions are frequently translated into real-world settings where practical needs drive changes to intervention protocols. Due to logistical and resource constraints, these naturally arising adaptations are rarely assessed for comparative effectiveness using a randomized trial. Nevertheless, when observational data are…

Several marginal effect size (ES) statistics suitable for quantifying the magnitude of differential item functioning (DIF) have been proposed in the area of item response theory; for instance, the Differential Functioning of Items and Tests (DFIT) statistics, signed and unsigned item difference in the sample statistics (SIDS, UIDS, NSIDS, and…

Meta-analysis of diagnostic test accuracy (DTA) is a powerful statistical method for synthesizing and evaluating the diagnostic capacity of medical tests and has been extensively used by clinical physicians and healthcare decision-makers. However, publication bias (PB) threatens the validity of meta-analysis of DTA. Some statistical methods have…

For large-scale assessments, data are often collected with missing responses. Despite the wide use of item response theory (IRT) in many testing programs, however, the existing literature offers little insight into the effectiveness of various approaches to handling missing responses in the context of scale linking. Scale linking is commonly used…

Classical evaluation methods, assessments, exams, and so forth accentuate the perception of one against all, professor versus learners. Including students in the assessment process, allows transforming the professor from an opponent to a critical friend, with the role of helping students to recognize both their strengths and weaknesses. However,…