As evidence from evaluation and experimental studies continue to influence decision and policymaking, applied researchers and practitioners require tools to derive valid and credible inferences. Over the past several decades, research in causal inference has progressed with the development and application of propensity scores. Since their…

First-year seminars (FYS) comprise one of 11 researched interventions in postsecondary education known as High-Impact Practices, but few rigorous studies report significantly high impacts. This study examined a FYS employing propensity score matching to link cases and controls in a quasi-experimental design. One semester later cumulative grade…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

In the United States high school students may take college level courses that count for credit in both high school and college, often called dual enrollment. Students may take courses at their high school from certified instructors or at a local college. We investigate possible differences between these two groups of students using student level…

Artificial neural networks (NN) can help researchers estimate propensity scores for quasi-experimental estimation of treatment effects because they can automatically detect complex interactions involving many covariates. However, NN is difficult to implement due to the complexity of choosing an algorithm for various treatment levels and monitoring…

In order to ensure content validity by covering a broad range of content domains, the testing times of some educational large-scale assessments last up to a total of two hours or more. Performance decline over the course of taking the test has been extensively documented in the literature. It can occur due to increases in the numbers of: (a)…

The reliability of a test score is usually underestimated and the deflation may be profound, 0.40 - 0.60 units of reliability or 46 - 71%. Eight root sources of the deflation are discussed and quantified by a simulation with 1,440 real-world datasets: (1) errors in the measurement modelling, (2) inefficiency in the estimator of reliability within…

Using Coarsened Exact Matching (CEM), this analysis examined the impact of Zearn Math across 7,116 matched pairs of students in 31 Louisiana parishes. Students who completed an average of 3+ Zearn Math lessons per week during the 2021-2022 school year were compared to similarly matched peers who completed less than 1 lesson per week. Findings…

Although Goodman-Kruskal gamma (G) is used relatively rarely it has promising potential as a coefficient of association in educational settings. Characteristics of G are studied in three sub-studies related to educational measurement settings. G appears to be unexpectedly appealing as an estimator of association between an item and a score because…

As colleges and universities grapple with uncertainty around current and future enrollment as well as increasingly vocal questions about the value of postsecondary education, it is critically important that institutions ascertain and invest in the elements of campus learning and engagement that add value to the undergraduate experience. This study…

Although educational research and evaluation generally occur in multilevel settings, many analyses ignore cluster effects. Neglecting the nature of data from educational settings, especially in non-randomized experiments, can result in biased estimates with long-term consequences. Our manuscript improves the availability and understanding of…

Testing programs are confronted with the decision of whether to report individual scores for examinees that have engaged in rapid guessing (RG). As noted by the "Standards for Educational and Psychological Testing," this decision should be based on a documented criterion that determines score exclusion. To this end, a number of heuristic…

When constructed-response items are administered for a second time, it is necessary to evaluate whether the current Time B administration's raters have drifted from the scoring of the original administration at Time A. To study this, Time A papers are sampled and rescored by Time B scorers. Commonly the scores are compared using the proportion of…

The literature showing that subscores fail to add value is vast; yet despite their typical redundancy and the frequent presence of substantial statistical errors, many stakeholders remain convinced of their necessity. This article describes a method for identifying and reporting unexpectedly high or low subscores by comparing each examinee's…

Purpose: This study builds a stochastic model of a discrete-time Markov chain (DTMC) that fits well with a dataset of professional playing records. Methods: The point-by-point dataset of Men's single matches played in the Association of Tennis Professionals (ATP) tour from 2011 to 2015 is analyzed. A long-debated assumption on the…