The nonequivalent groups with anchor test (NEAT) design is widely used in test equating. Under this design, two groups of examinees are administered different test forms with each test form containing a subset of common items. Because test takers from different groups are assigned only one test form, missing score data emerge by design rendering…

As evidence from evaluation and experimental studies continue to influence decision and policymaking, applied researchers and practitioners require tools to derive valid and credible inferences. Over the past several decades, research in causal inference has progressed with the development and application of propensity scores. Since their…

First-year seminars (FYS) comprise one of 11 researched interventions in postsecondary education known as High-Impact Practices, but few rigorous studies report significantly high impacts. This study examined a FYS employing propensity score matching to link cases and controls in a quasi-experimental design. One semester later cumulative grade…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

The current literature on test equating generally defines it as the process necessary to obtain score comparability between different test forms. The definition is in contrast with Lord's foundational paper which viewed equating as the process required to obtain comparability of measurement scale between forms. The distinction between the notions…

In the United States high school students may take college level courses that count for credit in both high school and college, often called dual enrollment. Students may take courses at their high school from certified instructors or at a local college. We investigate possible differences between these two groups of students using student level…

Artificial neural networks (NN) can help researchers estimate propensity scores for quasi-experimental estimation of treatment effects because they can automatically detect complex interactions involving many covariates. However, NN is difficult to implement due to the complexity of choosing an algorithm for various treatment levels and monitoring…

In order to ensure content validity by covering a broad range of content domains, the testing times of some educational large-scale assessments last up to a total of two hours or more. Performance decline over the course of taking the test has been extensively documented in the literature. It can occur due to increases in the numbers of: (a)…

Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact…

The reliability of a test score is usually underestimated and the deflation may be profound, 0.40 - 0.60 units of reliability or 46 - 71%. Eight root sources of the deflation are discussed and quantified by a simulation with 1,440 real-world datasets: (1) errors in the measurement modelling, (2) inefficiency in the estimator of reliability within…

Using Coarsened Exact Matching (CEM), this analysis examined the impact of Zearn Math across 7,116 matched pairs of students in 31 Louisiana parishes. Students who completed an average of 3+ Zearn Math lessons per week during the 2021-2022 school year were compared to similarly matched peers who completed less than 1 lesson per week. Findings…

Although Goodman-Kruskal gamma (G) is used relatively rarely it has promising potential as a coefficient of association in educational settings. Characteristics of G are studied in three sub-studies related to educational measurement settings. G appears to be unexpectedly appealing as an estimator of association between an item and a score because…

Out of over 1.9 million students in the 2018 ACT-tested graduating class, 44% took the full ACT® test at least twice in hopes of improving their scores. Camara and Allen (2017) support this practice, confirming that factors such as time in the classroom between test administrations are associated with increases in ACT Composite scores from the…

As colleges and universities grapple with uncertainty around current and future enrollment as well as increasingly vocal questions about the value of postsecondary education, it is critically important that institutions ascertain and invest in the elements of campus learning and engagement that add value to the undergraduate experience. This study…

Although educational research and evaluation generally occur in multilevel settings, many analyses ignore cluster effects. Neglecting the nature of data from educational settings, especially in non-randomized experiments, can result in biased estimates with long-term consequences. Our manuscript improves the availability and understanding of…