Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to…

Quantitative education research is often perceived to be "objective" or "neutral." However, quantitative research has been and continues to be used to perpetuate inequities; these inequities arise as both intended effects and unintended side effects of traditional quantitative research. In this review of the literature, we…

Education policy makers must make decisions under uncertainty. Thus, how they think about risks has important implications for resource allocation, interventions, innovation, and the information that is provided to the public. In this policy brief we illustrate how the standard of evidence for making decisions can be quite inconsistently applied,…

The purposes of this manuscript are to identify common statistical mistakes in sport management, and to provide scholars with suggestions on how to develop and improve the quality of quantitative research. We have reviewed articles published from 2001 to 2017 in the "Journal of Sport Management," "Sport Management Review,"…

Multivariate analysis of variance (MANOVA) is a statistical method used to examine group differences on multiple outcomes. This article reports results of a review of MANOVA in gifted education journals between 2011 and 2017 (N = 56). Findings suggest a number of conceptual and procedural misunderstandings about the nature of MANOVA and its…

Missing data are a persistent problem in psychological research. Peer nomination data present a unique missing data problem, because a nominator's nonparticipation results in missing data for other individuals in the study. This study examined the range of effects of systematic nonparticipation on the correlations between peer nomination data when…

This letter documents some problems in Ancaiani et al. (2015). Namely the evaluation of concordance, based on Cohen's kappa, reported by Ancaiani et al. was not computed on the whole random sample of 9,199 articles, but on a subset of 7,597 articles. The kappas relative to the whole random sample were in the range 0.07-0.15, indicating an…

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects…

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The 'number needed to disturb' a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or 'effect' size disappears, minus the number of…

Traditions of statistical significance testing in second language (L2) quantitative research are strongly entrenched in how researchers design studies, select analyses, and interpret results. However, statistical significance tests using "p" values are commonly misinterpreted by researchers, reviewers, readers, and others, leading to…

The educational literature, the popular press, and educated laypeople have all echoed a conclusion from the book "Academically Adrift" by Richard Arum and Josipa Roksa (which has now become received wisdom), namely, that 45% of college students showed no significant gains in critical thinking skills. Similar results were reported by…

Recently, Stephen Gorard has outlined strong objections to the use of significance testing in social research. He has argued, first, that as the samples used in social research are almost always non-random it is not possible to use inferential statistical techniques and, second, that even if a truly random sample were achieved, the logic behind…

The effective evaluation of correctional programs is critically important. However, research in corrections rarely allows for the randomization of offenders to conditions of the study. This limitation compromises internal validity, and thus, causal conclusions can rarely be drawn. Increasingly, researchers are employing propensity score matching…

Given the American Psychological Association's strong recommendation to always report effect sizes in research, scholars have a responsibility to provide complete information regarding their findings. The purposes of this study were to (a) determine the frequencies with which different effect sizes were reported in published, peer-reviewed…

The authors of the study "The Long-Term Impact of Teachers" claim that their study shows that increases in teacher value-added lead to significant and lasting increases in test scores and significant increases in income that will last throughout adulthood. Instead, I show that these claims are false because they are contradicted by the…