Sensitivity analyses can inform evidence-based education policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster's (2019) Coefficient of Proportionality (COP). Oster's COP leverages changes in estimated effects and R[superscript 2] when observed…

Background: The generalizability of a study refers to the extent to which the results and inferences from a sample apply to individuals in a larger target population of inference (Shadish et al., 2002). In practice, the strongest tool to facilitate generalizations is random or probability sampling, which is rare in educational studies (Olsen et…

The purpose of this paper is to elaborate and comment on some themes related to construct validity, construct choice, and construct respecification in the Campbellian validity system. It is argued that Reichardt's [(2008, November). An alternative to the Campbellian conceptualization of validity. (Paper presentation). Evaluation 2008 Conference,…

Pearson's correlation is widely used to test for an association between two variables and also forms the basis of several multivariate statistical procedures including many latent variable models. Spearman's [rho] is a popular alternative. These procedures are compared with ranking the data and then applying the inverse normal transformation, or…

Background/Context: Single-case experimental designs (SCEDs) play an important role in evaluating interventions in psychological, educational, and behavioral research. Unlike between-subjects designs, SCEDs involve a small number of cases whose responses to controlled experimental conditions are measured repeatedly over time. The evaluation of…

When no prior information of differential item functioning (DIF) exists for items in a test, either the rank-based or iterative purification procedure might be preferred. The rank-based purification selects anchor items based on a preliminary DIF test. For a preliminary DIF test, likelihood ratio test (LRT) based approaches (e.g.,…

Generalizability theory is a modern and powerful framework for conducting reliability analyses. It is flexible to accommodate both random and fixed facets. However, there has been a relative scarcity in the practical literature on how to handle the fixed facet case. This article aims to provide practitioners a conceptual understanding and…

It is an important point to test the homogeneity of variances in statistical methods such as the t-test or F-test used to make comparisons between groups. An erroneous decision regarding the homogeneity of variances will affect the test to be selected and thus lead to different results. For this reason, there are many tests for homogeneity of…

The difference in differences design is widely used to assess treatment effects in natural experiments or other situations where random assignment cannot, or is not, used (see, e.g., Angrist & Pischke, 2009). The researcher must make important decisions about which comparisons to make, the measurements to make, and perhaps the number of…

Data pooling is a powerful strategy in empirical research. However, combining multiple datasets often results in a large amount of missing data, as variables that are not present in some datasets effectively contain missing values for all participants in those datasets. Furthermore, data pooling typically leads to a mix of continuous and…

This paper proposes an extension of linear equating that may be useful in one of two fairly common assessment scenarios. One is where different students have taken different combinations of test forms. This might occur, for example, where students have some free choice over the exam papers they take within a particular qualification. In this…

Current research on intergenerational mobility (IGM) is informed by "statistical" approaches based on log-level regressions, whose "economic" interpretations remain largely unknown. We reveal the subjective value-judgments in them: they are represented by weighted-sums (or aggregators) over heterogeneous groups, with…

Background/Context: Single-case experiment designs (SCEDs) are experimental designs in which a small number of cases are repeatedly measured over time, with manipulation of baseline and intervention phases. Because SCEDs often rely on direct behavioral observations, count data are common. To account for both the clustering and the non-normal…

Large-scale survey assessments (LSAs) such as NAEP, TIMSS, PIRLS, IELS, and NAPLAN produce plausible values of student proficiency for estimating population statistics. Plausible values are imputed values for latent proficiency variables. While prominently used for LSAs, they are applicable to a wide range of latent variable modelling contexts…

Large-scale assessments are rich sources of data that can inform a diverse range of research questions related to educational policy and practice. For this reason, datasets from large-scale assessments are available to enable secondary analysts to replicate and extend published reports of assessment results. These datasets include multiple imputed…