Rapid approaches are essential when resources are limited and when findings are required in real-time to inform decisions. Limitations exist in their design and implementation, which can lead to a reduced level of trust in findings. This review sought to map the methods used across rapid evaluations and research to facilitate timeliness and…

The recognition that researcher discretion coupled with unconscious biases and motivated reasoning sometimes leads to false findings ("p-hacking") led to the broad embrace of study preregistration and other open-science practices in experimental research. Paradoxically, the preregistration of quasi-experimental studies remains uncommon…

Research synthesis of evaluation findings is a multistep process. An investigator identifies a research question, acquires the relevant literature, codes findings from that literature, and analyzes the coded data to estimate the average treatment effect and its distribution in a population of interest. The process of estimating the average…

Evaluations that combine social science and law have tremendous potential to illuminate the effects of governmental policies and yield insights into how effectively policy makers' efforts achieve their aims. This potential is infrequently achieved, however, because such interdisciplinary research contains often overlooked substantive and…

Six potential problems with the use of randomized experiments to evaluate programs in the field are addressed. Problems include treatment dilution, treatment contamination or confounding, inaccurate case flow and power estimates, violations of the random assignment processes, changes in the environmental context, and changes in the treatment…

Correction for guessing does not fulfill its intended function when test takers who have nothing to gain from scoring will respond randomly when they could have answered correctly had they tried. Raw scores underestimate abilities. If random guessing is more prevalent in the control group, correction for guessing inflates treatment effects.…

The validity of true experiments is threatened by a class of self-report biases that affect all respondents at pretest, but which are diminished by treatment. Four of these inaccurate self-evaluation biases are discussed. Means of detection include external criteria, special conditions of measurement, and retrospective pretests. (TJH)

This article addresses the issue of falling response rates in telephone surveys. To better understand and maintain respondent goodwill, concepts of psychological contract and respondent expectations are introduced and explored. Results of the qualitative study show that respondent expectations are not only socially contingent but also…

Administered a survey to 606 National Science Foundation-funded principal investigators and their institutions' representatives (n=91) to study responses to unethical research behavior. Scientists appear to perceive that they uphold their responsibility to respond through disclosures in the research community; administrators propose to report such…

Cross-cultural adaptation of study instruments is a difficult, time-consuming, but arguably cost-effective process. If conducted properly, it has the advantage that the translated study instruments are accurate, easy to understand, accessible, and culturally appropriate to the target audience and produce reliable and valid data. This article…

A beverage-server intervention project at two West Coast Navy bases that attempted to reduce levels of alcoholic intoxication via policy changes and server training is described. Data obtained via interviews and structured observations of 1,511 club customers indicate methodological bias and self-selection effects. Bias adjustments were performed…

Risks of life on the street caused by inclement weather, harassment, and assault threaten the unsheltered homeless population. We address some challenges of enumerating the street homeless population by testing a novel capture-recapture (CR) estimation approach that models individuals' intermittent daytime visibility. We tested walking and…

This article reviews the agricultural research evaluation literature in an effort to assess the extent of coverage in this area. Coverage breakdowns with respect to evaluation approaches, types of research, and products and production processes are presented. Results suggest that there has been limited coverage of research undertaken in…

The purposes of this article are, first, to argue that analyses based on level of treatment implementation can lead to biased estimates of treatment effects, and second, to discuss alternatives to this approach. (PN)

The normative implications of applying significance tests in evaluation research are examined. The authors conclude that evaluators often make normative decisions, based on the traditional .05 significance level in studies with small samples. Additional reporting of the magnitude of impact, the significance level, and the power of the test is…