Despite recent emphasis on the use of randomized control trials (RCTs) for evaluating education interventions, in most areas of education research, observational methods remain the dominant approach for assessing program effects. Over the last three decades, the within-study comparison (WSC) design has emerged as a method for evaluating the…

The identification of evidence-based practices continues to provoke issues of disagreement across multiple fields. One area of contention is the role of single-subject design (SSD) research in providing scientific evidence. The debate about SSD's utility centers on three issues: sample size, effect size, and serial dependence. One potential…

With growing popularity of single subject design as a method to evaluate the efficacy of interventions, it is important to ensure that the analyses of these methods are rigorous and reliable. The purpose of this case study is to discuss the measures used to evaluate the efficacy of interventions in single subject design studies in the fields of…

In some instances, intentionally or not, study designs are such that there is clustering in one group but not in the other. This paper describes methods for computing effect size estimates and their variances when there is clustering in only one group and the analysis has not taken that clustering into account. The authors provide the effect size…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

This methodological brief introduces the readers to the propensity score matching method, which can be used for enhancing the validity of causal inferences in research situations involving nonexperimental design or observational research, or in situations where the benefits of an experimental design are not fully realized because of reasons beyond…

A common mistake in analysis of cluster randomized experiments is to ignore the effect of clustering and analyze the data as if each treatment group were a simple random sample. This typically leads to an overstatement of the precision of results and anticonservative conclusions about precision and statistical significance of treatment effects.…

Previous work on statistical power has discussed mainly single-level designs or 2-level balanced designs with random effects. Although balanced experiments are common, in practice balance cannot always be achieved. Work on class size is one example of unbalanced designs. This study provides methods for power analysis in 2-level unbalanced designs…

This study explored the influence of strengths-based development on leadership practices among undergraduate college students while controlling for gender, years of leadership experience, and number of completed leadership courses using a quasi-experimental approach with a randomized control-group pretest-posttest research design. The sample…

Written by a social worker for social work students, this is a nuts and bolts guide to statistics that presents complex calculations and concepts in clear, easy-to-understand language. It includes numerous examples, data sets, and issues that students will encounter in social work practice. The first section introduces basic concepts and terms to…

An alternative analysis of the two-group single response variable design is proposed. It involves the classification of experimental units to populations represented by the two groups. Three real data sets are provided to illustrate the utility of the classification analysis. A table of sample sizes required for the analysis is presented.…

Repeated measures experimental designs, often referred to as "within-subjects" designs, offer researchers opportunities to study research effects while "controlling" for subjects. These designs offer greater statistical power relative to sample size. However, threats to internal validity such as carryover or practice effects…

Investigates sample size requirements needed to achieve various levels of statistical power using posttest-only, gain-score, and analysis of covariance designs in evaluating training interventions. Results indicate the power to detect true effects differ according to type of design, correlation between pre- and posttest, and size of effect due to…

Researchers increasingly recognize that significance tests are limited in their ability to inform scientific practice. Common errors in interpreting significance tests and three strategies for augmenting the interpretation of significance test results are illustrated. The first strategy for augmenting the interpretation of significance tests…

This paper discusses the sample size problem and four factors affecting its solution: significance level, statistical power, analysis procedure, and effect size. The interrelationship between these factors is discussed and demonstrated by calculating minimal sample size requirements for a variety of research conditions. (Author)