ERIC Number: ED659617
Record Type: Non-Journal
Publication Date: 2023-Sep-30
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Improving Measurement Practices through Equity-Forward Instrumentation Adaptation
Adam Smith; Megan Brunner; Michelle Tiu
Society for Research on Educational Effectiveness
Background and Context: Every student is a powerful learner capable of succeeding in mathematics. The EF+Math Program aims to dramatically improve math outcomes for Black and Latinx students and students of all races experiencing poverty (hereafter, our Priority Students). We are a community of researchers, developers, educators, and students who are developing research-informed mathematics learning approaches for grades 3-8 that combine executive function (EF) skills, conceptual understanding and multi-step problem solving, and equity. Our Inclusive Research and Development (R&D) model (Angevine et al., 2019; EF+Math Program, 2023) places an intentional focus on equity throughout the R&D process and centers educators and students as crucial partners from the start. In the Inclusive R&D process, we encourage cycles of design, iteration, and implementation in both research and development processes. Through these cycles of iteration, our teams have adjusted and refined their instruments to center and affirm our Priority Students. The portfolio of research studies take place in several elementary and middle school classrooms across the country. Objective and Analytic Approach: Our R&D teams are testing the math learning approaches that affirm the brilliance of our Priority Students, and that provide our students with rich and rigorous math learning opportunities. As such, they have designed innovative studies to examine the feasibility and efficacy of these approaches in districts, schools, and classrooms. As part of designing these studies, our teams critically examined the instruments traditionally employed to measure EF skills and psychosocial constructs such as belonging, math identity, and self-efficacy. The examination of these instruments catalyzed our teams to modify, adjust, and redesign the existing instruments to reflect our commitments to equity. As R&D teams have engaged in studies to examine the feasibility and efficacy of their math learning prototypes, we have examined the cycles of study design, implementation, and iteration across the portfolio. We analyzed each cycle based on its purpose and process for iteration, as well as the instrument that was revised. Three themes regarding embedding equity into measurement tools arose across the portfolio. We present these three themes below. Findings: Approach 1: Accessibility for All Students: Instruments are often designed with a specific population in mind, and this population can often exclude individuals for whom we wish to improve outcomes (Matsumoto, D. & Yoo, S. H., 2006). Such instruments often undergo validation studies with samples that follow this same exclusionary pattern. Many instruments that measure psychosocial constructs are designed for students of higher grades or have only been validated with White student populations. Through intentional design and thoughtful adjustments, our teams have modified these instruments to both accommodate and center our priority students. Specifically, educators have provided feedback regarding item language, length, and formats for student responses. Our teams are also engaged in validating adapted instruments for appropriateness of use with our priority students. Approach 2: Affirming the Brilliance of Students through Asset Framing: When examining psychosocial factors, items can sometimes signal that students are struggling, failing, or otherwise performing at a level that is not desirable (Russell, 2013; Stinson, D., 2013). These signals can be even stronger for students of marginalized identities (Schweinle & Mims, 2009; Stinson D., 2013)--like our priority students. Across the portfolio, several instruments included language that could signal to students that they are struggling math learners. Through co-design and iteration with educators, teams rewrote such items to reflect not only the brilliance of our students, but also to reflect that their psychosocial attributes do not make them "challenged" or "inferior". This included broadening response options as well as embedding additional framing for students before they engaged with the instruments. With these adaptations, students are able to be introspective about how they show up as math learners, rather than believing that they are underperforming. Approach 3: Leveraging multiple measures: By nature, latent constructs are difficult to observe, and instruments designed to measure such constructs are often unable to paint a holistic picture of the construct being measured (Russell, 2013). This might lead to inchoate understandings of these constructs, or interpretations of data can be taken out of context. Additionally, regarding psychosocial and/or affective states of students can often (and detrimentally) be situated as static, permanent, and immutable. For example, a student may be deemed "lazy" or "unfocused" if they exhibit low affect on a particular measure or seem easily distracted one day. Such interpretations can potentially reify deficit assumptions of students from marginalized backgrounds. To disrupt these biased and deficit applications of construct interpretation, R&D teams--in conjunction with educators and students--have developed approaches to not only measure constructs in multiple ways, but to also enact such measurement often and in real time. Some of our teams use a suite of instruments to measure executive function states, as well as other psychosocial constructs, like math identity, self-efficacy, belonging, and math identity. This allows researchers to honor the nuanced nature of constructs, and provides a more accurate and robust understanding of students. Such frequent and just-in-time information around EF skills and psychosocial states honor the fluid and dynamic nature of these constructs, thereby reinforcing an asset-framing about our Priority Students as dynamic, complex, and multifaceted human beings. Conclusion: Through critical examinations of instruments used in feasibility and efficacy studies of math learning prototypes, and subsequent cycles of design, implementation, and iteration, our teams of researchers and developers found opportunities to improve the instruments in ways that affirm our priority students and more accurately measure psychosocial constructs such as belonging, math identity, and self-efficacy. The practices that the R&D teams engaged in reflect our commitments to equity and our commitments to affirming and centering Black and Latinx students and students of all races experiencing poverty.
Descriptors: Mathematics Education, Outcomes of Education, Mathematics Achievement, Research and Development, Research Design, Identification, At Risk Students
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A
Author Affiliations: N/A