Quantitative results from empirical studies are common in the field of Scholarship of Teaching and Learning (SoTL), but it is important to remain aware of what the results from our studies can, and cannot, tell us. Oftentimes studies conducted to examine teaching and learning are constrained by class size. Small sample sizes negatively influence…

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects…

The authors of the study "The Long-Term Impact of Teachers" claim that their study shows that increases in teacher value-added lead to significant and lasting increases in test scores and significant increases in income that will last throughout adulthood. Instead, I show that these claims are false because they are contradicted by the…

Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power…

This report presents findings on crime and violence in U.S. public schools, using data from the 2015-16 School Survey on Crime and Safety (SSOCS:2016). First administered in school year 1999-2000 and repeated in school years 2003-04, 2005-06, 2007-08, 2009-10, and 2015-16, SSOCS provides information on school crime-related topics from the…

The present paper aims to review two motivations to conduct "what if" analyses using Excel and "R" to understand the statistical significance tests through the sample size context. "What if" analyses can be used to teach students what statistical significance tests really do and in applied research either prospectively to estimate what sample size…

The t-test is one of the most commonly used significance tests to assess whether the means of two groups are statistically significantly different from each other. The use of t-test has become a natural choice, and rarely practitioners question its appropriateness. This article reviews and discusses t-test's value in providing a rough comparison…

This study is an evaluation of the behavior of the Liu-Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu-Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no…

Researchers conducting studies involving individuals with exceptionalities are often prevented from involving large numbers of participants in their study samples. When this is the case, some say significant correlations are likely to replicate because the relation between two variables must be robust enough to be detected even with low…

This study compared the accuracies of four differential item functioning (DIF) estimation methods, where each method makes use of only one of the following: raw data, logistic regression, loglinear models, or kernel smoothing. The major focus was on the estimation strategies' potential for estimating score-level, conditional DIF. A secondary focus…

The recent study of Oshima, Raju, and Nanda proposes the item parameter replication (IPR) method for assessing statistical significance of the noncompensatory differential item functioning (NCDIF) index within the differential functioning of items and tests (DFIT) framework. Previous Monte Carlo simulations have found that the appropriate cutoff…

This paper reviews methodological issues related to the use of P less than 0.05 in health behavior research and suggests how application and presentation of statistical significance may be improved. Assessment of sample size and P less than 0.05, the file drawer problem, the Law of Large Numbers and the statistical significance arguments in…

This report describes patterns of continuity and change over time in four areas of the transition to adulthood among young adults as measured 2 years after their senior year of high school. The four areas are postsecondary enrollment, labor force roles, family formation, and civic engagement through voting or military service. The analysis…

This report provides estimates of criminal incidents that occur at school. Incident-level data were obtained from the National Crime Victimization Survey (NCVS), the nation's primary source of information on criminal victimization and criminal incidents in the United States. The NCVS collects demographic information on respondents in the NCVS…

This investigation reports on the cross-cultural equivalence testing of the Self-Description Questionnaire II (short version; SDQII-S) for Indigenous and non-Indigenous Australian secondary student samples. A variety of statistical analysis techniques were employed to assess the psychometric properties of the SDQII-S for both the Indigenous and…