ERIC Number: ED659406
Record Type: Non-Journal
Publication Date: 2023-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Effects of the First Year of a 3-Year Intervention in Learning Trajectory-Oriented Formative Assessment on Grades PreK-3 Student Achievement
Caroline B. Ebby; Karina G. Diaz; Siling Guo; Zoë Ehrenberg
Society for Research on Educational Effectiveness
Background/Context: Developing strong number sense and additive reasoning in the early grades is a critical foundation for success in school and beyond. Longitudinal studies show that early competence with number and operations predicts later math achievement, reading achievement, high school retention rates, and employability and wages in adulthood (Duncan et al., 2007; Geary et al., 2013). Although there is a large body of research on how young students learn math and develop concepts of number and operation, translating research into classroom instructional practice has been an ongoing challenge for mathematics educators. The Ongoing Assessment Project (OGAP), a formative assessment system for core K-8 content areas, responded to this challenge by designing research-based learning trajectories, resources, and professional development for teachers. Prior studies of large-scale implementation of OGAP in two urban districts demonstrated a significant impact on teacher knowledge across grade levels, and a significant impact on student math performance in grades 3-5 (Supovitz et al., 2021). To date, there have been no studies of the impact of OGAP implementation on PreK-2 student learning. Purpose/Objective/Research Question: This study explores the impacts of one year of teacher professional development and support from OGAP on PreK-3 student learning in a large urban district. The study is guided by the following research questions: (1) To what extent does the learning trajectory-oriented formative assessment model foster change in PreK-3 student math performance after one year of implementation? and (2) what changes in knowledge and instructional practice were reported by teachers who received the OGAP model? Setting: School District of Philadelphia, Pennsylvania. Population/Participants/Subjects: The first phase of implementation (2021-22) involved four days of OGAP summer training for 38 PreK-3 teachers and leaders from six elementary schools and three in-school follow-up training and support sessions for all PreK-3 teachers and leaders in those schools during the academic year. The treatment group consisted of the 33 teachers who attended at least 3 days of the summer training; Because 2 of those teachers had two math classes, there were a total of 35 treatment classrooms in the study. The control group consisted of a total of 46 PreK-3 classrooms within five matched schools that did not receive OGAP training in 2021-22. All 11 schools in this study served between 92-100% students of color and 100% were from low-income families. Intervention/Program/Practice: OGAP is designed to build teachers' capacity to use research on student learning trajectories to elicit and respond to student thinking on a regular basis and tailor instruction to student needs. OGAP has a fully developed set of frameworks, formative assessment item banks, pre-assessments, and professional development materials for grades PreK-3 that have been through multiple rounds of piloting, field-testing, district and state-level implementation, and revision from 2016 to the present. Teachers are trained to implement OGAP through a formative assessment cycle that involves (1) selecting and administering a formative assessment item to elicit students' developing understanding, (2) analyzing students' written responses using the OGAP framework and learning progression, and (3) developing an instructional response that is based on the evidence. Research Design: Impacts of the intervention on student outcomes were assessed via a quasi-experimental design. The study compares the students in the schools where teacher receive OGAP training to students in a demographically similar matched sample of schools within the same network. Consequently, five out of nine schools that did not receive OGAP training in 2021-22, were matched on a set of students and school characteristics (e.g., enrollment, attendance rate, EL rate, race proportion) as well as academic outcomes (i.e., PSSA scores) at baseline to serve as comparison schools. Data Collection and Analysis: The Test of Early Mathematics Ability (TEMA-3) was administered to both the OGAP and comparison group students in the spring of 2022. Additionally, the research team collected survey data from Prek-3 teachers at OGAP schools in the spring of the 2021-22 school year. Star CBM Math, which assess foundational grade-level math skills aligned to the Common Core standards, data was also requested and received from the School District at the student level for OGAP and comparison schools. We estimated treatment effects using hierarchical linear regression models to reflect the multilevel structure of the data (Raudenbush & Bryk, 2002). The primary analyses examined evidence of any statistically significant differences between OGAP and Comparison group students, in TEMA-3 and STAR CBM scores. In addition, we use descriptive statistics (i.e., means, standard deviations) to examine teacher self-reported implementation of OGAP and change in practice, as measured by the survey. Findings/Results: OGAP trained teachers reported high levels of change in their mathematics teaching practice (see Figure 1), particularly in relation to the way they looked at student work, identified levels of student thinking, and gathered formative evidence. Over three-fourths of teachers agreed or strongly agreed that OGAP helped them develop both math content knowledge and new instructional strategies as shown in Figure 2. After controlling for Fall baseline STAR CBM scores, our analysis show students in classrooms where teachers participated in OGAP training outperformed students in comparison classrooms on the district administered STAR CBM scores (b=3.12, t(539)=1.96, p=0.049). However, we find no significant treatment effects on the PreK-3 TEMA-3 scores (Table 1). Conclusions: Results show promise of potential OGAP impact on PreK-3 students' outcomes, especially considering the unique challenges of the 2021-22 year and the return to in-person learning. The second phase of implementation (2022-23) involves summer and in-school follow-up OGAP training for a second cohort of teachers within the network. This training expansion allowed us to continue to follow the students who were studied in the first phase for an additional year to detect longer term impacts of the treatment.
Descriptors: Elementary School Students, Primary Education, Learning Trajectories, Formative Evaluation, Mathematics Tests, Mathematics Teachers, Faculty Development, Program Implementation
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Early Childhood Education; Primary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Pennsylvania (Philadelphia)
Grant or Contract Numbers: N/A
Author Affiliations: N/A