Sensitivity analyses can inform evidence-based education policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster's (2019) Coefficient of Proportionality (COP). Oster's COP leverages changes in estimated effects and R[superscript 2] when observed…

The purpose of this paper is to elaborate and comment on some themes related to construct validity, construct choice, and construct respecification in the Campbellian validity system. It is argued that Reichardt's [(2008, November). An alternative to the Campbellian conceptualization of validity. (Paper presentation). Evaluation 2008 Conference,…

Pearson's correlation is widely used to test for an association between two variables and also forms the basis of several multivariate statistical procedures including many latent variable models. Spearman's [rho] is a popular alternative. These procedures are compared with ranking the data and then applying the inverse normal transformation, or…

When no prior information of differential item functioning (DIF) exists for items in a test, either the rank-based or iterative purification procedure might be preferred. The rank-based purification selects anchor items based on a preliminary DIF test. For a preliminary DIF test, likelihood ratio test (LRT) based approaches (e.g.,…

Generalizability theory is a modern and powerful framework for conducting reliability analyses. It is flexible to accommodate both random and fixed facets. However, there has been a relative scarcity in the practical literature on how to handle the fixed facet case. This article aims to provide practitioners a conceptual understanding and…

Data pooling is a powerful strategy in empirical research. However, combining multiple datasets often results in a large amount of missing data, as variables that are not present in some datasets effectively contain missing values for all participants in those datasets. Furthermore, data pooling typically leads to a mix of continuous and…

This paper proposes an extension of linear equating that may be useful in one of two fairly common assessment scenarios. One is where different students have taken different combinations of test forms. This might occur, for example, where students have some free choice over the exam papers they take within a particular qualification. In this…

Current research on intergenerational mobility (IGM) is informed by "statistical" approaches based on log-level regressions, whose "economic" interpretations remain largely unknown. We reveal the subjective value-judgments in them: they are represented by weighted-sums (or aggregators) over heterogeneous groups, with…

Large-scale assessments are rich sources of data that can inform a diverse range of research questions related to educational policy and practice. For this reason, datasets from large-scale assessments are available to enable secondary analysts to replicate and extend published reports of assessment results. These datasets include multiple imputed…

Causal mediation analysis has gained increasing attention in recent years. This article guides empirical researchers through the concepts and challenges of causal mediation analysis. I first clarify the difference between traditional and causal mediation analysis and highlight the importance of adjusting for the treatment-by-mediator interaction…

School start regulations allocate children born immediately either side of a given date to different life paths: those slightly older starting school a full year earlier. School effectiveness literature exploits this to estimate causal effects described as 'the absolute effect of schooling' or 'the effect of an additional year's schooling', using…

We consider the cascading effects of researcher decisions throughout the process of quantifying racially polarized voting (RPV). We contrast three methods of estimating precinct racial composition, Bayesian Improved Surname Geocoding (BISG), fully Bayesian BISG, and Citizen Voting Age Population (CVAP), and two algorithms for performing ecological…

Background and Context: Data science and statistics are used across a broad spectrum of professions, experience levels and programming languages. The popular scientific computing languages, such as Matlab, Python and R, were organized without using empirical methods to show evidence for or against their design choices, resulting in them feeling…

Visual analysis is the primary methodology used to determine treatment effects from graphed single-case design data. Previous studies have demonstrated mixed findings related to interrater agreement between both expert and novice visual analysts, which represents a critical limitation of visual analysis and supports calls for also presenting…

This article presents methods for using extant data to improve the properties of estimators of the standardized mean difference (SMD) effect size. Because samples recruited into education research studies are often more homogeneous than the populations of policy interest, the variation in educational outcomes can be smaller in these samples than…