Sensitivity analyses can inform evidence-based education policy by quantifying the hypothetical conditions necessary to change an inference. Perhaps the most prevalent index used for sensitivity analyses is Oster's (2019) Coefficient of Proportionality (COP). Oster's COP leverages changes in estimated effects and R[superscript 2] when observed…

This article presents a step-by-step guide to using R and SPSS to bootstrap exam questions. Bootstrapping, a versatile nonparametric analytical technique, can help to improve the psychometric qualities of exam questions in the process of quality assurance. Bootstrapping is particularly useful in disciplines such as medical education, where student…

Large language models (LLMs) provide cost-effective but possibly inaccurate predictions of human behavior. Despite growing evidence that predicted and observed behavior are often not "interchangeable," there is limited guidance on using LLMs to obtain valid estimates of causal effects and other parameters. We argue that LLM predictions…

Most randomized field experiments experience some attrition. Moreover, the extent of attrition may differ by treatment condition in systematic, non-random ways, biasing estimates of treatment effects and contributing to invalid inferences. We address concerns about non-random attrition by quantifying the conditions necessary in the attritted data…

Knowledge tracing (KT) for evaluating students' knowledge is an essential task in personalized education. More and more researchers have devoted themselves to solving KT tasks, e.g., deep knowledge tracing (DKT), which can capture more sophisticated representations of student knowledge. Nonetheless, these techniques ignore the reconstruction of…

In causal mediation studies that decompose an average treatment effect into indirect and direct effects, examples of post-treatment confounding are abundant. In the presence of treatment-by-mediator interactions, past research has generally considered it infeasible to adjust for a post-treatment confounder of the mediator-outcome relationship due…

Beginning with debates about the effects of smoking on lung cancer, sensitivity analyses characterizing the hypothetical unobserved conditions that can alter statistical inferences have had profound impacts on public policy. One of the most ascendant techniques for sensitivity analysis is Oster's (2019) coefficient of proportionality, which…

Accounting for model misspecification in Bayesian structural equation models is an active area of research. We present a uniquely Bayesian approach to misspecification that models the degree of misspecification as a parameter--a parameter akin to the correlation root mean squared residual. The misspecification parameter can be interpreted on its…

Missing data in scientific research go hand in hand with assumptions about the nature of the missingness. When dealing with missing values, a set of beliefs has to be formulated about the extent to which the observed data may also hold for the missing parts of the data. It is vital that the validity of these missingness assumptions is verified,…

Bayesian methods seem a natural choice for combining sources of evidence in meta-analyses. However, in practice, their sensitivity to the choice of prior distribution is much less attractive, particularly for parameters describing heterogeneity. A recent non-Bayesian approach to fixed-effects meta-analysis provides novel ways to think about…

Recent critiques of physics education research (PER) studies have revoiced the critical issues when drawing causal inferences from observational data where no intervention is present. In response to a call for a "causal reasoning primer" in PER, this paper discusses some of the fundamental issues in statistical causal inference. In…

Pseudocontingencies are inferences of correlations between variables, like two options and two outcomes, drawn on the basis of their skewed base rates covarying across a third variable (e.g., two contexts). Here, we investigated the effect of pseudocontingency inference on choice behavior. When choices between two options are not based on the…

The standard methodology for many large-scale assessments in education involves regressing latent variables on numerous contextual variables to estimate proficiency distributions. To reduce the number of contextual variables used in the regression and improve estimation, we propose and evaluate principal component analysis on the covariance matrix…

Recently, there has been an increased interest in the associations between theory of mind (ToM), the ability to infer others' mental states, and academic functioning in middle childhood including scientific reasoning. This study follows this trend and addresses two gaps in the literature. First, it examines whether individual differences in ToM…

When fitting unidimensional item response theory (IRT) models, the population distribution of the latent trait ([theta]) is often assumed to be normally distributed. However, some psychological theories would suggest a nonnormal [theta]. For example, some clinical traits (e.g., alcoholism, depression) are believed to follow a positively skewed…