The design and psychometric methodology of the National Assessment of Educational Progress (NAEP) is constantly evolving to meet the changing interests and demands stemming from a rapidly shifting educational landscape. NAEP has been built on strong research foundations that include conducting extensive evaluations and comparisons before new…

Standard setting is the process of identifying the point(s) on a scale that serve to differentiate between individuals of distinct proficiency levels. While standard setting is ultimately a policy decision, most of the process is carried out by subject matter experts who are tasked with reconciling item-level or examinee-level information (e.g.…

Angoff-based standard setting is widely used, especially for high-stakes licensure assessments. Nonetheless, some critics have claimed that the judgment task is too cognitively complex for panelists, whereas others have explicitly challenged the consistency in (replicability of) standard-setting outcomes. Evidence of consistency in item judgments…

New Meridian Corporation has developed the "Quality Testing Standards and Criteria for Comparability Claims" (QTS) to provide guidance to states that are interested in including New Meridian content and would like to either keep reporting scores on the New Meridian Scale or use the New Meridian performance levels; that is, the state…

The Angoff standard setting method relies on content experts to review exam items and make judgments about the performance of the minimally proficient examinee. Unfortunately, at times content experts may have gaps in their understanding of specific exam content. These gaps are particularly likely to occur when the content domain is broad and/or…

Over the past twenty years in performance testing a specific item type with distinguishing characteristics has arisen time and time again. It's been invented independently by dozens of test development teams. And yet this item type is not recognized in the research literature. This article is an invitation to investigate the item type, evaluate…

This study addressed what standard-setting panelists think about when they make item performance estimates for a barely proficient student. This study extended previous studies by considering the factors that influenced panelists' decisions in an Angoff (1971)-based standard-setting study as a function of their item performance estimates.…

In the specific methods of standard setting in testing, judgments about individuals being tested, contrasting groups being tested, and judgments about the test items are discussed. In judgments about individual test-takers, assumptions are presented based on the knowledge and skills the test is intended to measure, the test-takers' skills at the…

The results of classifying test items on the basis of their Mantel-Haenszel (MH) alpha estimates were compared to the results of classifying these items using an item response theory (IRT) based procedure involving the comparison of item difficulties in the interest of identifying the alpha value that maximized the decision concordance between the…

Studies of the Angoff method of standard setting suggest that judges agree in their estimates of the relative difficulties of test questions for minimally competent examinees and that each judge's estimates correlate well with the observed item difficulties for examinees whose total test scores are near the judge's personal standard (G. E.…

The research reported in this paper was conducted to gain information to guide the selection of standard setting procedures for use with polytomous items to set achievement levels on the National Assessment of Educational Progress (NAEP) assessments in U.S. History and geography. Standard-setting procedures were evaluated to determine the relative…

Studies of the mathematics assessments of the National Assessment of Educational Progress (NAEP) are summarized. One study found that omit rates for NAEP test items were higher for African Americans and Hispanics than for whites. The other found that descriptions and examples for the 1992 mathematics achievement levels were misleading. (SLD)

The licensure of public school personnel in Tennessee is a function of the State Department of Education (SDE). Additional requirements for this certification function were added by the Comprehensive Education Reform Act (Senate Bill Number 1) of the 1984 Tennessee General Assembly. The Act mandated that the SDE use tests to assess the competence…

This handbook contains information for high school educators--teachers, curriculum coordinators, counselors, and principals--as they prepare students to take the Prairie State Achievement Examination (PSAE). The first section explains the purpose of the PSAE, gives timelines and test-day schedules, and shows how the PSAE is constructed. The rest…