Evidence-centered design (ECD) is an orientation towards assessment development. It differs from conventional practice in several ways and consists of multiple activities. Each of these activities results in a set of useful documentation: domain analysis, domain modeling, construction of the assessment framework, and assessment…

Developing a cognitive model of task performance is an important and often overlooked phase in assessment design; failing to establish such a model can threaten the validity of the inferences made from the scores produced by an assessment (e.g., Leighton, 2004). Conducting think aloud interviews (TAIs), where students think aloud while completing…

Presented at the Annual Meeting of National Council on Measurement in Education (NCME) in San Diego, CA in April 2009. This presentation provides an overview of ECD. In addition, the presentation describes the benefits of, as well as the challenges that were faced, in implementing ECD in the Advanced Placement Program.

Presented at the Annual Meeting of National Council on Measurement in Education (NCME) in San Diego, CA in April 2009. This presentation describes the methodology that was used with subject-matter experts (SMEs) to articulate the content and skills important in the domain, and then the iterative processes that were used to articulate the claims…

This paper discusses issues related to the design and psychometric work for mixed-format tests --tests containing both multiple-choice (MC) and constructed-response (CR) items. The issues of validity, fairness, reliability and score consistency can be addressed but for mixed-format tests there are many decisions to be made and no examination or…

A new test of Advanced Placement Physics, explicitly designed to balance both content and cognitive-processing skills, was developed using Sternberg's theory of successful intelligence. The test was administered to 281 AP Physics students from 10 schools during the 2006-2007 school year. Six empirically distinguishable profiles of strengths and…

This study investigated the extent to which log-linear smoothing could improve the accuracy of common-item equating by the chained equipercentile method in small samples of examinees. Examinee response data from a 100-item test (the Advanced Placement Examination in United States History) were used to create two overlapping forms of 58 items each,…