Since 2007, two taxonomies have been proposed to identify the components of evaluation practice that may be specified in an evaluation policy. Little is known, however, about how these taxonomies align with evaluation policies developed by philanthropic foundations. Through thematic analysis, this article first compares 12 foundation evaluation…

This report describes the implementation of professional development (PD) reforms and efforts to use teacher effectiveness (TE) data to inform PD through the third year of the initiative for all seven sites: Hillsborough County Public Schools (HCPS), Shelby County Schools (SCS, formerly Memphis City Schools), Pittsburgh Public Schools (PPS), and…

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters…

The sense that replicability is an important aspect of empirical science led Killeen (2005a) to define "p[subscript rep]," the probability that a replication will result in an outcome in the same direction as that found in a current experiment. Since then, several authors have praised and criticized 'p[subscript rep]," culminating…

The term "intervention validity" refers to the extent to which assessment results can be used to guide the selection of interventions and evaluation of outcomes. In this article, the authors review the defining attributes of rating scales that distinguish them from other assessment tools, assumptions regarding the use of rating scales to…

Some questions about the relationship between program evaluation and forecasting are raised. Contrasting the two fields in terms of mind-set, purpose, problems, advantages, and use shows that although their modes of inquiry are indeed very different in many ways, there is interdependence between the two in several areas. (Author/JAZ)

This article reviews issues involved with using norm-referenced measures within an intervention validity model. Methodological examination of intervention validity has become more important as intervention methodologies have become part of federal legislation. However, current practice and federal guidelines often implicitly portray a dichotomy…

The new "Weighted Student Formula Yearbook 2009" from the Reason Foundation provides a simple framework for touting the successes of states and urban school districts that grant greater fiscal autonomy to schools. The report defines the Weighted Student Formula (WSF) reform extremely broadly, presenting a variety of reforms under the WSF umbrella.…

A report from the School Choice Demonstration Project examines issues concerning the funding formula used for the Milwaukee Parental Choice Program (MPCP). It finds that the program generates a net saving to taxpayers in Wisconsin but imposes a significant fiscal burden on taxpayers in Milwaukee. However, these findings depend significantly on how…

Treatment integrity (also referred to as "treatment fidelity," "intervention integrity," and "procedural reliability") is an important methodological concerning both research and practice because treatment integrity data are essential to making valid conclusions regarding treatment outcomes. Despite its relationship to validity, treatment…

The Center for Research on Education Outcomes (CREDO) at Stanford University conducted a large-scale analysis of the impact of charter schools on student performance. The center's data covered 65-70% of the nation's charter schools. Although results varied by state, 17% of the charter school students have significantly higher math results than …

"How New York City's Charter Schools Affect Achievement" estimates the effects on student achievement of attending a New York City charter school rather than a traditional public school and investigates the characteristics of charter schools associated with the most positive effects on achievement. Because the report relies on an…

Six potential problems with the use of randomized experiments to evaluate programs in the field are addressed. Problems include treatment dilution, treatment contamination or confounding, inaccurate case flow and power estimates, violations of the random assignment processes, changes in the environmental context, and changes in the treatment…

The validity of true experiments is threatened by a class of self-report biases that affect all respondents at pretest, but which are diminished by treatment. Four of these inaccurate self-evaluation biases are discussed. Means of detection include external criteria, special conditions of measurement, and retrospective pretests. (TJH)

Reviews current educational software evaluation methods, highlights problems, and describes the York Educational Software Evaluation Scales (YESES), an alternative criterion based model. Panel evaluation used by YESES is explained and YESES results are compared with evaluations from the Educational Products Information Exchange (EPIE) to indicate…