ERIC Number: ED619021
Record Type: Non-Journal
Publication Date: 2019
Pages: 152
Abstractor: As Provided
ISBN: 978-1-0856-5178-3
ISSN: EISSN-
EISSN: N/A
Available Date: N/A
Evidence Model: Evaluate Complex Problem-Solving Competency from Process Data
Yang, Zhitong
ProQuest LLC, Ed.D. Dissertation, University of Pennsylvania
Computer-based assessments allow practitioners to collect rich process data by logging students' interactions with assessment tasks. In addition to providing final responses to test questions, computer-based assessments promise to furnish more evidence to support claims about what a student knows and can do through logging process data in log files. However, because these rich log file data often contain both useful information and noise, such as unpredictable actions or emerging behaviors, culling useful information is a challenging task. Some data-driven approaches (such as educational data mining) have recently been used to identify patterns in process data, but the identified patterns are often difficult to interpret. There is a crucial need to interpret process data based on cognitive theories to provide valid evidence for claims made in large-scale summative assessments. Evidence-centered design (ECD; Mislevy, Almond, & Lukas, 2004; Mislevy, Almond, Steinberg, & Lukas, 2006), a framework for educational assessment development, advocates that evidence identification and interpretation should be linked to the competencies/skills to be assessed and the design of a task used in an assessment. Guided by the ECD framework, this study employs a theory-driven approach to analyze and interpret rich process data generated from one interactive task used in the 2012 "Programme for International Student Assessment" (PISA) complex problem-solving assessment. To develop principles that can be used to identify evidence of complex problem-solving competencies/skills, this study examined the interactive task structure and reviewed cognitive theories of complex problem-solving, and based on which, a theory-based coding scheme was proposed. Based on the coding scheme, process data were coded to identify specific behaviors that are relevant to the complex problem-solving. An exploratory factor analysis revealed multi-dimensional cognitive abilities involved in the complex problem-solving. Then, the 2012 PISA complex problem-solving scoring rubric was evaluated in terms of whether there is evidence from process data to support it. Finally, the incremental predictability of student competency/skill evaluations based on behavioral coding is examined. Specifically, the study aimed to answer three research questions: (a) Can student process data generated from one interactive task in the 2012 PISA complex problem-solving assessment reflect distinct cognitive competencies/skills? (b) To what extent can evidence from process data support the ratings of student performance evaluated based on their final responses on the PISA complex problem-solving assessment? (c) To what extent do student groups sorted by a few behavioral indicators show different levels of academic achievement on the 2012 PISA assessments of complex problem-solving, science, mathematics, and reading? The results of the study showed that, based on a theory-based coding scheme, the process data generated from the 2012 PISA complex problem-solving task can be meaningfully interpreted and reflects distinct cognitive competencies/skills. In addition, this study found that process data provides supporting evidence to the 2012 PISA scoring rubric. Furthermore, the study identified behavioral indicators of targeted cognitive competencies/skills that differentiate student performance on complex problem-solving and other academic assessments. The study demonstrates how a theory-driven approach to evaluating student performance through process data could contribute to establishing an evidence model for a complex problem-solving task. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]
Descriptors: Problem Solving, Computer Assisted Testing, Data Analysis, Foreign Countries, International Assessment, Achievement Tests, Secondary School Students, Epistemology, Thinking Skills, Academic Achievement, Scoring Rubrics, Student Behavior
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Program for International Student Assessment
Grant or Contract Numbers: N/A
Author Affiliations: N/A