Within the current test-centered educational reform movement, considerable emphasis is placed on using assessment results to make instructional decisions for individual students. Test scores are used to estimate a student's current level of skill, monitor his or her progress during instruction, and identify whether the student has gained the expected skills and knowledge at the end of an instructional period. Informed and appropriate instructional decision-making, however, is predicated on several assumptions: (1) assessments can provide information about students' understanding, strategic competence, and reasoning skills within a domain; and (2) that this information can be obtained accurately and efficiently to help educators determine whether students will benefit from additional instructional supports to be successful (Ikeda, Neessen, & Witt, 2008). Meeting these assumptions, however, requires taking steps during the assessment development process to ensure valid inferences can be made about students' knowledge, skills, and abilities based on their performance on the assessment. The purpose of this study is to describe the development of a universal screening assessment of algebra readiness for Grades 2-4 with a particular focus on the collection of student think-aloud data to: (1) provide insights into students' cognitive and problem-solving processes; (2) investigate the relationships between students' predictive, planning, and procedural metacognition and their performance on multiple-choice mathematics items that were similar in structure and content to those that would be used in the universal screening assessment; and (3) inform the item writing and revision process. Specifically, this study focuses on the following research questions: (1) To what extent can multiple-choice mathematics items be written that require students' to demonstrate varying levels of cognitive engagement; and (2) What is the relation between students' predictive, planning, and procedural metacognition and their performance on multiple-choice mathematics items. Three tables and one figure (Directions for Concurrent and Retrospective Think-Alouds) are appended.