I discuss three common practices that obfuscate or invalidate the statistical analysis of randomized controlled interventions in applied linguistics. These are (a) checking whether randomization produced groups that are balanced on a number of possibly relevant covariates, (b) using repeated measures ANOVA to analyze pretest-posttest designs, and…

As result replicability is essential to science and difficult to achieve through external replicability, the present paper notes the insufficiency of null hypothesis statistical significance testing (NHSST) and explains the bootstrap as a plausible alternative, with a heuristic example to illustrate the bootstrap method. The bootstrap relies on…

This study is an evaluation of the behavior of the Liu-Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu-Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no…

The aim of the current study is to validate the invariance of major profile patterns derived from multidimensional scaling (MDS) by bootstrapping. Profile Analysis via Multidimensional Scaling (PAMS) was employed to obtain profiles and bootstrapping was used to construct the sampling distributions of the profile coordinates and the empirical…

This report provides estimates of criminal incidents that occur at school. Incident-level data were obtained from the National Crime Victimization Survey (NCVS), the nation's primary source of information on criminal victimization and criminal incidents in the United States. The NCVS collects demographic information on respondents in the NCVS…

This paper discusses the effect of clustering on statistical tests and illustrates this effect using classroom environment data. Most classroom environment studies involve the collection of data from students nested within classrooms and the hierarchical nature to these data cannot be ignored. In particular, this paper studies the influence of…

The estimate-and-subtract method for eliminating threats to validity is described. It is argued that the method is superior to the use of no-difference findings for this purpose. Two ways of improving the no-difference findings are presented. (TJH)

This paper examines the estimation of two-stage clustered RCT designs in education research using the Neyman causal inference framework that underlies experiments. The key distinction between the considered causal models is whether potential treatment and control group outcomes are considered to be fixed for the study population (the…

The use of multiple comparisons in analysis of variance (ANOVA) is discussed. It is argued that experimentwise Type I error rate inflation can be serious and that its influences are often unnoticed in ANOVA applications. Both classical balanced omnibus and orthogonal planned contrast tests inflate experimentwise error to an identifiable maximum.…

Four things are recommended to minimize the influence or importance of statistical significance testing. Researchers must not neglect to add "statistical" to significant and could interpret results before giving p-values. Effect sizes should be reported with measures of sampling error, and replication can be built into the design. (SLD)

In mean and covariance structure analysis, the chi-square difference test is often applied to evaluate the number of factors, cross-group constraints, and other nested model comparisons. Let model M[a] be the base model within which model M[b] is nested. In practice, this test is commonly used to justify M[b] even when M[a] is misspecified. The…

A Monte Carlo approach was used to examine bias in the estimation of indirect effects and their associated standard errors. Results illustrate the adverse effects of nonnormality on the accuracy of significance tests in latent variable models estimated using normal theory maximum likelihood statistics. (SLD)

Some of the methods originally used to find relationships between reliability and power associated with a single measurement are extended to difference scores. Results, based on explicit power calculations, show that augmenting the reliability of measurement by reducing error score variance can make significance tests of difference more powerful.…

Often, all of the variables in a model are latent, random, or subject to measurement error, or there is not an obvious dependent variable. When any of these conditions exist, an appropriate method for estimating the linear relationships among the variables is Least Principal Components Analysis. Least Principal Components are robust, consistent,…

The use of stepwise regression has been criticized for both interpretive misunderstandings and statistical aberrations. A major statistical problem with stepwise regression and other procedures that involve multiple significance tests is the inflation of the Type I error rate. General approaches to control the family-wise error rate such as the…