In any meta-analysis, it is critically important to report the dispersion in effects as well as the mean effect. If an intervention has a moderate clinical impact "on average" we also need to know if the impact is moderate for all relevant populations, or if it varies from trivial in some to major in others. Or indeed, if the…

We study the external validity of instrumental variable estimation. The key assumption we impose for external validity is conditional external unconfoundedness among compliers, which means that the treatment effect and target selection are independent among compliers conditional on covariates. We study this assumption with a case study about the…

Large language models (LLMs) provide cost-effective but possibly inaccurate predictions of human behavior. Despite growing evidence that predicted and observed behavior are often not "interchangeable," there is limited guidance on using LLMs to obtain valid estimates of causal effects and other parameters. We argue that LLM predictions…

The study provides a detailed methodological approach, cross-industry standard process for data mining, for predicting at-risk students with an imbalanced class. The objective was to identify the best machine learning model for predicting students at risk of failing the course during weeks 2-8 of the semester. We encountered issues in the dataset,…

We propose new ways of robustifying goodness-of-fit tests for structural equation modeling under non-normality. These test statistics have limit distributions characterized by eigenvalues whose estimates are highly unstable and biased in known directions. To take this into account, we design model-based trend predictions to approximate the…

What does it mean for a model to be a better model? One conceptualization, indeed a common one in Educational Data Mining, is that a better model is the one that fits the data better, that is, higher prediction accuracy. However, oftentimes, models that maximize prediction accuracy do not provide meaningful parameter estimates, making them less…

Random encouragement designs evaluate treatments that aim to increase participation in a program or activity. These randomized controlled trials (RCTs) can also assess the mediated effects of participation itself on longer term outcomes using a complier average causal effect (CACE) estimation framework. This article considers power analysis…

Over the last decade, there has been a 10-fold increase in the number of published systematic reviews of prevalence. In meta-analyses of prevalence, the summary estimate represents an average prevalence from included studies. This estimate is truly informative only if there is no substantial heterogeneity among the different contexts being pooled.…

For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both time-consuming and limiting: even the largest human coding efforts are typically constrained to…

Linguistic phenomena (e.g., words and syntactic structure) co-occur with a wide variety of meanings. These systematic correlations can help readers to interpret a text and create predictions about upcoming material. However, to what extent these correlations influence discourse processing is still unknown. We address this question by examining…

Random encouragement designs are randomized controlled trials (RCTs) that test interventions aimed at increasing participation in a program or activity whose take up is not universal. In these RCTs, instead of randomizing individuals or clusters directly into treatment and control groups to participate in a program or activity, the randomization…

Nonresponse bias is a widely prevalent problem for data on education. We develop a ten-step exemplar to guide nonresponse bias analysis (NRBA) in cross-sectional studies and apply these steps to the Early Childhood Longitudinal Study, Kindergarten Class of 2010-2011. A key step is the construction of indices of nonresponse bias based on proxy…

The purpose of this dissertation is to develop principles and strategies for and identify limitations of multisite cluster randomized trials in the context of partially and fully nested designs. In the first study, I develop principles of estimation, sampling variability, and inference for studies that leverage multisite designs within the context…

This study elaborates on the pivotal roles of mathematical and statistical models in data-driven predictions in an integrated STEM context using the case of Year 4 students: (?) "a descriptive means" to describe the features of trends and variability of data and (?) "an explanatory means" to explain causal relationships behind…

Determining the number of factors in exploratory factor analysis is arguably the most crucial decision a researcher faces when conducting the analysis. While several simulation studies exist that compare various so-called factor retention criteria under different data conditions, little is known about the impact of missing data on this process.…