This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

Items on test score scales located at and below the Proficient cut score define the content area knowledge and skills required to achieve proficiency. Alternately, examinees who perform at the Proficient level on a test can be expected to be able to demonstrate that they have mastered most of the knowledge and skills represented by the items at…

Findings from an outcome evaluation of the California AIDS Education and Prevention Program, the largest of its kind in the country, are presented. Specific programs suffered from design defects that made it impossible to accurately determine their effectiveness. Attitude and behavior change was not properly addressed by the programs. (TJH)

Kentucky's systemic reform initiative has been heralded as one of the more comprehensive and well-sustained reforms in recent history. To evaluate the course of this reform, the challenges associated with researching and evaluating a statewide systemic reform effort are detailed here. By using a description of the politics of evaluating systemic…

This research investigated the effects of two contrasting pre-k curricula, relative to practice as usual, on subsequent academic achievement. One curriculum had a strong literacy focus, the other was a less didactic "developmentally appropriate" curriculum that allowed children to have more influence on classroom activities. The purpose…

A study compared the effects of different methods of analyzing trends to illustrate how failure to account for change in classification will lead to misinterpretation of data on the performance of students with disabilities. Data from five years of assessment in Texas is used to illustrate effects of classification changes. (Contains references.)…

This publication provides a framework that is designed for use by state agencies as a guide for planning and conducting evaluation of different aspects of the State Adult Education Program. The evaluation framework is presented using a set of design tables that offer different approaches states can use to address the Adult Education Act's…

The Council of Chief State School Officers has raised some issues pertinent to the implementation of a national project to collect statistical indicators of academic achievement and to compare them across states. The United States Department of Education shares this concern, with an emphasis on sampling schools for a school level analysis which…

Maryland, the first state to mandate performance assessments for its elementary school students, administers the Maryland School Performance Assessment Program as a week-long series of activities measuring reading, writing, language in use, science, and social studies for students in grades 3, 5, and 8. Open-ended questions are built around…

In 1991 Pennsylvania began implementation of a direct writing assessment at the sixth-grade and ninth-grade levels. A total of 18,758 sixth graders and 16,575 ninth graders wrote a response to 1 of 9 prompts reflecting 3 modes of writing. A six-point holistic scale was used to score the papers, with two readers scoring each paper. A third reader,…

At the heart of the No Child Left Behind Act (NCLB) is the call for all students to be "proficient" in reading and mathematics by 2014. Yet the law expects each state to define proficiency as it sees fit and design its own tests. This study investigated three research questions related to this policy: (1) How consistent are various…