The level of standardized effect sizes obtained by chance and the use of significance tests to guard against spuriously high standardized effect sizes were studied. The concept of the "protected effect size" is also introduced. Monte Carlo methods were used to generate data for the study using random normal deviates as the basis for sample means to be compared using one-way analysis of variance. Standardized effect sizes were generated for 5000 replications within each combination of number of groups from 2 to 10 and sample sizes from 5 to 100 in steps of 5, resulting in 900,000 total replications. Results indicate that the significance test does provide some protection against judging spuriously high standardized effect size values as being meaningful. Applying a statistical significance test does substantially reduce the proportion of standardized effect sizes that achieve small, medium, or large criteria levels by chance. Perhaps the arguments should not be about using either statistical significance testing or effect sizes. Instead, the focus should be on finding ways to combine the philosophies and methods of both to make decisions about group differences. (Contains 5 figures. 6 tables, and 36 references.) (SLD)