In the present Monte Carlo simulation study, the authors compared bias and precision of 7 sampling error corrections to the Pearson r[superscript 2] under 6 x 3 x 6 conditions (i.e., population ρ values of 0.0, 0.1, 0.3, 0.5, 0.7, and 0.9, respectively; population shapes normal, skewness = kurtosis = 1, and skewness = -1.5 with kurtosis =…

Investigated how dissimilar in composition and variability samples inducting reliability coefficients from prior studies were from the cited prior samples from which coefficients were generalized. Results from 20 articles show that citing reliability coefficients from prior studies as the basis for concluding new scores are reliable is only…

Three problems with stepwise research methods are explored. Computer packages may use incorrect degrees of freedom in stepwise computations. In addition, stepwise methods do not identify correctly the best variable set of a given size. A third problem is that stepwise methods tend to capitalize on sampling error. (SLD)

Responds to Biskin's comments (this issue) on the significance test controversy. Highlights areas of agreement (importance of replication evidence, importance of effect sizes) and disagreement (influence of sample size, evaluation of populations vs. samples, significance of Carver's article). Includes further recommendations for reporting research…

Multivariate methods are being used with increasing frequency in educational research because these methods control "experimentwise" error rate inflation, and because the methods best honor the nature of the reality to which the researcher wishes to generalize. This paper: explains the basic logic of canonical analysis; illustrates that…

A Monte Carlo study involving 1,000 random samples from each of 64 different population matrices investigated bias in both canonical correlation and redundancy coefficients. Results indicate that the Wherry correction provides a reasonable solution to this problem and that canonical results are not as biased as has been believed. (TJH)

In the present study Monte Carlo methods were employed to evaluate the degree to which canonical function and structure coefficients may be differentially sensitive to sampling error. Sampling error influences were investigated across variations in variable and sample (n) sizes, and across variations in average within-set correlation sizes and in…

Stepwise methods are frequently employed in educational and psychological research, both to select useful subsets of variables and to evaluate the order of importance of variables. Three problems with stepwise applications are explored in some detail. First, computer packages use incorrect degrees of freedom in their stepwise computations,…

Use of the bootstrap method in a canonical correlation analysis to evaluate the replicability of a study's results is illustrated. More confidence may be vested in research results that replicate. (SLD)

Canonical correlation analysis is a powerful statistical method subsuming other parametric significance tests as special cases, and which can often best honor the complex reality to which most researchers wish to generalize. However, it has been suggested that the canonical correlation coefficient is positively biased. A Monte Carlo study…

Three criticisms of overreliance on results from statistical significance tests are noted. It is suggested that: (1) statistical significance tests are often tautological; (2) some uses can involve comparisons that are not completely sensible; and (3) using statistical significance tests to evaluate both methodological assumptions (e.g., the…

Conventional statistical significance tests do not inform the researcher regarding the likelihood that results will replicate. One strategy for evaluating result replication is to use a "bootstrap" resampling of a study's data so that the stability of results across numerous configurations of the subjects can be explored. This paper…

Too few researchers understand what statistical significance testing does and does not do, and consequently their results are misinterpreted. This Digest explains the concept of statistical significance testing and discusses the meaning of probabilities, the concept of statistical significance, arguments against significance testing,…

Effect sizes have been increasingly emphasized in research as more researchers have recognized that: (1) all parametric analyses (t-tests, analyses of variance, etc.) are correlational; (2) effect sizes have played an important role in meta-analytic work; and (3) statistical significance testing is limited in its capacity to inform scientific…