ERIC Number: ED524891
Record Type: Non-Journal
Publication Date: 2011-Oct
Pages: 140
Abstractor: ERIC
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Whether and How to Use State Tests to Measure Student Achievement in a Multi-State Randomized Experiment: An Empirical Assessment Based on Four Recent Evaluations. NCEE 2012-4015
Somers, Marie-Andree; Zhu, Pei; Wong, Edmond
National Center for Education Evaluation and Regional Assistance
This study examines the practical implications of using state tests to measure student achievement in impact evaluations that span multiple states and grades. In particular, the study examines the sensitivity of impact findings to (1) the type of assessment used to measured achievement (state tests or an external assessment administered by the evaluators) and (2) to analytical decisions about how to pool state test data across states and grades. These questions are examined using data from four recent IES-funded randomized experiments where student achievement was measured using both state tests and a test administered by evaluators for the purposes of the study ("study-administered test"). These studies span multiple states--8 to 10 states depending on the experiment. Based on these four studies, the authors conclude that state tests provide a useful complement to a study-administered test, because they are policy-relevant measures of general achievement. However, in certain cases, state tests may not be a feasible substitute for a study-administered test, either because state tests are not administered in all relevant grades or because the primary outcome is a specific skill that is not measured by all states' tests. Inferences about program impacts are not sensitive to decisions about how test scores are scaled for the purposes of pooling the results across states or grades (for example, whether traditional or rank-based z-scores are used and whether z-scores are based on the sample or state distribution of scores). Finally, the most appropriate method for aggregating the impact findings across states or grades is to use fixed-effects (precision) weighting, because the conditions for using random-effects weighting are not met. Appended are: (1) Descriptive Information on Study and State Tests; (2) Assessing the Conditions for Equating State Test Scores; (3) Technical Notes; (4) Impact Tables; (5) Correlation between Student Achievement Measures; and (6) Statistical Tests of Differences in Impact Findings Across Achievement Measures. (Contains 27 tables, 13 figures, and 104 footnotes.)
Descriptors: Evaluators, Grades (Scholastic), Academic Achievement, Program Effectiveness, Inferences, Experiments, Scores, Scaling, Comparative Analysis, Student Evaluation, State Standards, Effect Size, Standardized Tests, Achievement Tests, Elementary Secondary Education, Outcomes of Education, Measures (Individuals), Test Validity, Test Reliability, Testing Programs, National Competency Tests
National Center for Education Evaluation and Regional Assistance. Available from: ED Pubs. P.O. Box 1398, Jessup, MD 20794-1398. Tel: 877-433-7827; Web site: http://ies.ed.gov/ncee/
Publication Type: Reports - Evaluative
Education Level: Elementary Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: National Center for Education Evaluation and Regional Assistance (ED)
Identifiers - Location: United States
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
IES Funded: Yes
Grant or Contract Numbers: N/A
IES Publication: http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20124015
Author Affiliations: N/A