Objective: This article describes the evaluation of permanency roundtables, an intervention to help youth in foster care achieve legal permanency and the challenges evaluators faced in finding and using appropriate comparison data. Method: In 2009, permanency roundtables were conducted for 496 children in Georgia, most of whom had spent extended…

After months of waiting, the grant reviews came back: "excellent," "excellent," and "fair." What?! How can this be? Why is the third review so out of line with the first two? On more than one occasion a principal investigator (PI) has been frustrated not only by a negative funding decision but more so by the accompanying reviewer evaluation forms…

This paper presents the author's comment on "Evaluation Field Building in South Asia: Reflections, Anecdotes, and Questions" by Katherine Hay. Hay raises a number of extremely relevant issues relating to evaluation field building in South Asia. In this paper, the author aims to underscore the importance of three priorities for initiating…

Good evaluation requires that evaluation efforts themselves be evaluated. Many things can and often do go wrong in evaluation work. Accordingly, it is necessary to check evaluations for problems such as bias, technical error, administrative difficulties, and misuse. Such checks are needed both to improve ongoing evaluation activities and to assess…

Explores implications for evaluation practice of using five different perspectives on organizations, drawing on the four models of social program implementation developed by R. F. Elmore (1978). It is argued that evaluation will only really be effective when its focus, methods, and management reflect realistic assumptions about how organizations…

Demonstrates that the common procedure of using examination results as performance indicators based upon residuals from regression analysis when applied to aggregated data gives unstable results. Suggests that aggregate-level analyses are uninformative and that useful comparisons cannot be obtained without employing multilevel analyses using…

This digest illustrates the variety of basic and theoretical issues in evaluation with which aspiring evaluators should be conversant in order to claim that they know the knowledge base of their profession. Coverage of the issues includes: the four steps in the logic of evaluation; whether qualitative evaluations are valid; whether it matters if…

The Annual Performance Report submitted by each state to the Office of Vocational and Adult Education was used to monitor what states have developed and how and to what extent states have implemented their accountability system. The following findings of this analysis are synthesized in this report: (1) 32 states require evaluation of all programs…

For the past 20 years, the University of Hawaii Community Colleges have had a formal process for the review of educational programs, requiring that each program undergo a systematic review at least once every 5 years. The result tended to be a lengthy document that chronicled the history of the previous 5 years and was far more descriptive than…