The Benefit Offset National Demonstration (BOND) tested a $1 reduction in benefits per $2 earnings increase above the level at which Social Security Disability Insurance benefits drop from full to zero under current law. BOND included a rare and large "population-representative" experiment: It applied the rule to a nationwide, random…

To draw informed conclusions from research studies, research consumers need full and accurate descriptions of study methods and procedures. Preregistration has been proposed as a means to clarify reporting of research methods and procedures, with the goal of reducing bias in research. However, preregistration has been applied primarily to research…

This paper reviews literature on the rationale, challenges, and recommendations for choosing a nonequivalent comparison (NEC) group design when evaluating intervention effects. After reviewing frequently addressed threats to validity, the paper describes recommendations for strengthening the research design and how the recommendations were…

This poem, with stanzas in limerick form, refers humorously to the many threats to validity posed by problems in research design, including problems of sample selection, data collection, and data analysis. (SLD)

Criticizes an earlier essay that questioned the validity of contemporary educational research practices, especially those involving experimentation. Argues that research using experimentation is valid and can lead to new and generalizable knowledge. (FMW)

Methods of assessing effect size and the flows that limit their usefulness are discussed. The various statistical procedures for estimating variance accounted for are based on different statistical models producing sharply differing results. The methods reflect too greatly the particular study characteristics and hence have limited…

Six potential problems with the use of randomized experiments to evaluate programs in the field are addressed. Problems include treatment dilution, treatment contamination or confounding, inaccurate case flow and power estimates, violations of the random assignment processes, changes in the environmental context, and changes in the treatment…

The validity of true experiments is threatened by a class of self-report biases that affect all respondents at pretest, but which are diminished by treatment. Four of these inaccurate self-evaluation biases are discussed. Means of detection include external criteria, special conditions of measurement, and retrospective pretests. (TJH)

The decisive power of an experiment, for both social evaluation research and laboratory experimentation as well, is only as strong as the clarity of the basic assumptions which underlie it. A dialectical methodology is proposed for assessing the influence of key assumptions in both settings. (Author/CTM)

Some methodological issues in the study of levels of knowledge are reviewed, and needs for further research are explored, drawing on an analysis of 12 studies reported since the late 1970s. In the 12 studies, 16 quantitative experiments were conducted. These were assessed for internal and external validity. Analysis revealed some shortcomings in…

An integrating framework to explain the research process was presented in the first edition of this book. This edition, which brings the material up to date, is designed to serve the Master s level student better while still meeting the needs of doctoral students of educational research. The nature of research, methods for qualitative and…

This overview of the research process for the novice information retrieval (IR) researcher provides guidance on selecting a research question, choosing the type of test, defining variables, developing or using databases, finding and processing queries, assigning treatments to experimental units, collecting and analyzing the data, and presenting…