Sensitivity analyses can inform evidence-based education policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster's (2019) Coefficient of Proportionality (COP). Oster's COP leverages changes in estimated effects and R[superscript 2] when observed…

Background: The generalizability of a study refers to the extent to which the results and inferences from a sample apply to individuals in a larger target population of inference (Shadish et al., 2002). In practice, the strongest tool to facilitate generalizations is random or probability sampling, which is rare in educational studies (Olsen et…

I. Purpose: There are instances where equivalences rather than differences are of interest (e.g., Lakens, 2017). For instance, researchers may be interested in testing whether two versions of interventions are equivalent or whether an alternative but more cost-efficient program can provide similar outcomes (Lakens et al., 2018; Schuirmann, 1987).…

The purpose of this paper is to elaborate and comment on some themes related to construct validity, construct choice, and construct respecification in the Campbellian validity system. It is argued that Reichardt's [(2008, November). An alternative to the Campbellian conceptualization of validity. (Paper presentation). Evaluation 2008 Conference,…

Pearson's correlation is widely used to test for an association between two variables and also forms the basis of several multivariate statistical procedures including many latent variable models. Spearman's [rho] is a popular alternative. These procedures are compared with ranking the data and then applying the inverse normal transformation, or…

Background/Context: Single-case experimental designs (SCEDs) play an important role in evaluating interventions in psychological, educational, and behavioral research. Unlike between-subjects designs, SCEDs involve a small number of cases whose responses to controlled experimental conditions are measured repeatedly over time. The evaluation of…

When no prior information of differential item functioning (DIF) exists for items in a test, either the rank-based or iterative purification procedure might be preferred. The rank-based purification selects anchor items based on a preliminary DIF test. For a preliminary DIF test, likelihood ratio test (LRT) based approaches (e.g.,…

Generalizability theory is a modern and powerful framework for conducting reliability analyses. It is flexible to accommodate both random and fixed facets. However, there has been a relative scarcity in the practical literature on how to handle the fixed facet case. This article aims to provide practitioners a conceptual understanding and…

It is an important point to test the homogeneity of variances in statistical methods such as the t-test or F-test used to make comparisons between groups. An erroneous decision regarding the homogeneity of variances will affect the test to be selected and thus lead to different results. For this reason, there are many tests for homogeneity of…

Matching methods are widely used in educational research to estimate causal effects when randomization is not feasible. These techniques pair treated units (such as schools receiving an intervention) with similar control units based on observable characteristics. However, current statistical inference procedures for these methods can produce…

Background: Effectiveness studies that ignore costs can lead to misguided policies. For instance, reducing class sizes gained traction due to positive outcomes on student achievement (Konstantopoulos & Li, 2012; Krueger, 1999), yet cost-effectiveness studies revealed the high-cost relative to its modest benefits (Brewer et al., 1999). More…

The difference in differences design is widely used to assess treatment effects in natural experiments or other situations where random assignment cannot, or is not, used (see, e.g., Angrist & Pischke, 2009). The researcher must make important decisions about which comparisons to make, the measurements to make, and perhaps the number of…

Data pooling is a powerful strategy in empirical research. However, combining multiple datasets often results in a large amount of missing data, as variables that are not present in some datasets effectively contain missing values for all participants in those datasets. Furthermore, data pooling typically leads to a mix of continuous and…

This paper proposes an extension of linear equating that may be useful in one of two fairly common assessment scenarios. One is where different students have taken different combinations of test forms. This might occur, for example, where students have some free choice over the exam papers they take within a particular qualification. In this…

Current research on intergenerational mobility (IGM) is informed by "statistical" approaches based on log-level regressions, whose "economic" interpretations remain largely unknown. We reveal the subjective value-judgments in them: they are represented by weighted-sums (or aggregators) over heterogeneous groups, with…