Drasgow, Levine, and Zickar (1996) suggested a statistic based on the Neyman-Pearson lemma (e.g., Lehmann & Romano, 2005, p. 60) for detecting preknowledge on a known set of items. The statistic is a special case of the optimal appropriateness indices of Levine and Drasgow (1988) and is the most powerful statistic for detecting item…

In late November 2020, there was a flurry of media coverage of two companies' claims of 95% efficacy rates of newly developed COVID-19 vaccines, but information about the confidence interval was not reported. This paper presents a way of teaching the concept of hypothesis testing and the construction of confidence intervals using numbers announced…

In narrative synthesis of evidence, it can be the case that the only quantitative measures available concerning the efficacy of an intervention is the direction of the effect, that is, whether it is positive or negative. In such situations, the sign test has been proposed in the literature and in recent Cochrane guidelines as a way to test whether…

This article presents two alternative methods to null hypothesis significance testing (NHST) for improving inferences from underpowered research designs. Post hoc design analysis (PHDA) assesses whether an NHST analysis generating null findings might otherwise have had sufficient power to detect effects of plausible magnitudes. Bayesian analysis…

Although the null hypothesis significance testing procedure is problematic, many still favor the use of "p"-values as indicating the state of evidence against the model used to generate the "p"-value. From this perspective, "p"-values benefit science; or would benefit science if used correctly. In contrast, the novel…

Purpose: Null hypothesis significance testing is commonly used in audiology research to determine the presence of an effect. Knowledge of study outcomes, including nonsignificant findings, is important for evidence-based practice. Nonsignificant "p" values obtained from null hypothesis significance testing cannot differentiate between…

According to Wollack and Schoenig (2018), score differencing is one of six types of statistical methods used to detect test fraud. In this paper, we suggested the use of Bayes factors (e.g., Kass & Raftery, 1995) for score differencing. A simulation study shows that the suggested approach performs slightly better than an existing frequentist…

Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

Background: Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs)…

Education researchers frequently have to choose between statistical models for their data, and in many cases the candidate models or parameters can be listed in a sequence, m=1,...,M, from less preferable choices to more. For instance, in choosing a bandwidth for regression discontinuity designs, researchers would favor the largest possible…

Structural equation modeling is an ideal data analytical tool for testing complex relationships among many analytical variables. It can simultaneously test multiple mediating and moderating relationships, estimate latent variables on the basis of related measures, and address practical issues such as nonnormality and missing data. To test the…

The natural experiment proposed in this article extracts three stories from boxes of "100 paper clips". The activity requires students to apply three lessons from inferential statistics, starting with a hypothesis test and including confidence intervals as well as tolerance intervals.

BASIE is a framework for interpreting impact estimates from evaluations. It is an alternative to null hypothesis significance testing. This guide walks researchers through the key steps of applying BASIE, including selecting prior evidence, reporting impact estimates, interpreting impact estimates, and conducting sensitivity analyses. The guide…

Formal empirical assessments of replication have recently become more prominent in several areas of science, including psychology. These assessments have used different statistical approaches to determine if a finding has been replicated. The purpose of this article is to provide several alternative conceptual frameworks that lead to different…

Under the generalizability-theory (G-theory) framework, the estimation precision of variance components (VCs) is of significant importance in that they serve as the foundation of estimating reliability. Zhang and Lin advanced the discussion of nonadditivity in data from a theoretical perspective and showed the adverse effects of nonadditivity on…