WestEd Assemblage

Assemblage · February 2, 2023

WestEd Assemblage — Assemblage, February 2, 2023.
Published

February 2, 2023

Event
Invited talk
Slides
23
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. WestEd Assemblage COVID-19 Impact & Recovery Damian Betebenner & Adam VanIwaarden Center for Assessment February 2nd, 2023
  2. About me • 2007 to Present: Center for Assessment • Developed Student Growth Percentiles (SGP) and worked with over two dozen states on leveraging their longitudinal data systems • With the pandemic, tried to help states that I work with utilize the data they have to better communicate what has happened/is happening to students. • 2004 to 2017: Assistant Professor at Boston College • 1995 to 2004: University of Colorado, Boulder — Ph.D and Post-Doc • 1989 to 2005: University of Wyoming — Ph.D Mathematics
  3. COVID-19 Background Introductory Concepts and Terminology
  4. COVID-19 Pandemic • The COVID-19 Pandemic led to, arguably, the largest educational disruption in the history of the United States. • Nearly 3 years into the pandemic and the impacts of the pandemic are still being felt. • The pandemic impacted students academically, emotionally, physically, and financially. • In this overview, we discuss three things: • Academic Impact: What is it and what happened to students. • Academic Recovery: What is it and what is happening. • Now What?: How can impact and recovery data be leveraged to help students going forward? Evidence-based investigations of what is working.
  5. What is Academic Impact? What is Academic Recovery? • The pandemic and all the ensuing disruptions functioned/functions as an academic headwind, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • In education odometer is to attainment as speedometer is to growth. • Academic impact is analogous to a headwind, decreasing the rate of student learning (deceleration) which, in turn leads to lower attainment
  6. What is Academic Impact? What is Academic Recovery? • Academic recovery is analogous to an academic tailwind, increasing the rate of student learning (acceleration) which, in turn leads to higher attainment. • During Fall 2021 to present, under the presumption that student education got back to normal, the hope was that rates of learning stabilized to pre-pandemic rates or, even better, increased to rates exceeding pre-pandemic rates allowing students to catch-up. • Examination of academic impact/recovery are opposite sides of the same coin and utilize the same analytics.
  7. Impact & Recovery • We investigate impact/recovery in two ways: • Change in same grade/same cohort attainment: We examine equi-percentile change in attainment. Impact: 2019 to 2021, Recovery 2021 to 2022 • Change in student growth: We examine change in baseline (pre-pandemic) student growth (SGPs). Impact: 2019 to 2021 (using 2017 to 2019 growth norms), Recovery: 2021 to 2022 (using 2018 to 2019 growth norms).
  8. Assessment Data • We investigated the impacts in achievement and growth of the pandemic across two time frames: • Impact: Spring 2020 through Summer 2021 • Recovery: Fall 2021 through Summer 2022 • To investigate impact we utilized two primary sources of data • State summative assessment data • Impact: State summative assessment Spring 2019 to Spring 2021 • Recovery: State summative Spring 2021 to Spring 2022 • State English Language Proficiency Data (ACCESS) • Impact: ACCESS Winter 2020 to Winter 2021 • Recovery: ACCESS Winter 2021 to Winter 2022
  9. Assessment Data • Other sources of data exists to investigate these phenomena • Interim assessment • NAEP • I have looked at state summative, interim and NAEP and the untidy reality is that: • Big picture: They broadly tell the same thing (e.g., bigger impacts in math than in ELA) • When you look more closely, they can tell different stories • Different assessment can tell different things, even if the same methodology is used on those data sets. • State assessment data is in my opinion the highest quality data being used right now.
  10. Impact From Spring 2020 to Summer 2021 Spring 2019 and Spring 2021 Assessments Winter 2020 and Winter 2021 Assessments
  11. Impact Analyses • Worked with 12 states to analyses their data to determine academic impact in 2021 • State summative tests were cancelled in 2020 so analyses of state summative tests were done from 2019 to 2021 • ELP assessments (WIDA-ACCESS) were not cancelled so analyzed data from 2020 to 2021
  12. ILEARN Mathematics Grade 8 Indiana 2019 to 2021 COVID⌧19 Academic Impact All Students Growth: Grade 6 2019 to Grade 8 2021 Status: Grade 8 2019 to Grade 8 2021 Summary 2019 Grade 6 Scaled Score Decile Cuts 2019 Grade 8 Scaled Score Academic Impact: Overall and by Quintile Grade 8 Scaled/Standardized Score and Achievement Level Cuts Overall Observed + Imputed 2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (Observed) 2 2021 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large Observed 2019 to 2021 Growth Impact 2019 to 2021 Status Impact 2019 and 2021 Mean Scaled Scores 2021 Grade 8 Scaled Score Decile Cuts Moderate Large Large Large Large 2021 Grade 8 Scaled Score Participation 60 0 0.00 Total Status Growth 6400 6400 2019: 37.4% Proficient 2021: 27.8% Proficient Growth/Status Change Legend 2019 Grade 6 Scaled Score with Achievement Level Cuts Grade 8 Overall Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 8 Median SGPs by Quintile 2019 & 2021 Grade 8 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 50.0 50.0 50.0 50.0 50.0 50.0 2019 Grade 8 Students 6395 (−1.44) 6492 (−0.54) 6555 ( 0.04) 6612 ( 0.57) 6698 ( 1.37) 6550 (0.00) 2021 Grade 8 Students 38.0 32.0 28.0 29.0 35.0 32.0 2021 Grade 8 Students 6378 (−1.59) 6458 (−0.85) 6518 (−0.30) 6583 ( 0.30) 6678 ( 1.18) 6523 (−0.25) Calculations suppressed Academic Impact (Observed) −12.0 −18.0 −22.0 −21.0 −15.0 −18.0 Change (Observed) −17 (−0.16) −33 (−0.31) −36 (−0.34) −30 (−0.27) −20 (−0.19) −27 (−0.25) Academic Impact (+ Imputed) −12.0 −18.0 −22.0 −21.0 −15.0 −18.0 Change (+ Imputed) −17 (2.21) −33 (1.48) −36 (1.01) −30 (0.61) −20 (0.01) −27 (−0.25) due to small group N (< 50) Count Observed 2019 15,360 15,419 15,592 15,390 15,385 77,146 Count Observed 2019 16,498 16,763 16,635 16,515 16,579 82,990 Count Observed 2021 15,556 15,470 15,591 15,783 15,828 78,228 Count Observed 2021 16,454 16,508 16,452 16,536 16,592 82,542 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 0.30 2021 to 2019 Grade 8 Status Change (Observed) 30 2021 to 2019 SGP Difference (+ Imputed) 30 2021 to 2019 Grade 8 Status Change (+ Imputed) 0.25 0.81: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 1.00: Correlation SGP and Conditional Status Difference 2019 to 2021 2019 Grade 6 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)
  13. Summary We summarize the academic impact both overall and by achievement quintile For Grade 8 mathematics, the overall academic impact for students was large (z decrease = -0.29 and median SGP decrease = Academic impact was not uniform across students. For this grade/content area higher achieving students were more severely impacted than lower achieving student.
  14. ILEARN ELA and Mathematics Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Deterioration Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Economically Disadvantaged Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Large Non−Economically Disadvantaged Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large African American Large Large Moderate Large Moderate Moderate Severe Severe Severe Severe Large Large Asian Moderate Moderate Moderate Moderate Moderate Modest to None Moderate Moderate Large Severe Severe Large White Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Hispanic Large Moderate Large Large Moderate Moderate Large Large Large Severe Large Large ELL Large Moderate Large Large Moderate Large Large Large Large Large Moderate Large Special Education Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Female Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Severe Large Large Male Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large
  15. WIDA ACCESS WIDA⌧ACCESS Composite WIDA IN 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement
  16. All Students Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 State A Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate State B Modest to None Modest to None Moderate Moderate Moderate Modest to None Modest to None Moderate Moderate Moderate Large Large Moderate Moderate State C Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State D Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State E Large Moderate Large Moderate Large Moderate Moderate Severe Large Large Severe Large Large State F Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State G Moderate Moderate Large Large Large Large Moderate Moderate Large Large Large Large State H Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State I Moderate Moderate Large Large Large Large Modest to None Severe Severe Severe Severe Large Severe Large State J Moderate Moderate Moderate Moderate Large Moderate State K Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe
  17. All Students Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
  18. Recovery From Fall 2021 to Summer 2022 Spring 2021 and Spring 2022 Assessments Winter 2021 and Winter 2022 Assessments
  19. Recovery Phase • During the 2021-2022 academic year, school returned to normal for most students. • The hope was that a return to normal would lead recovery of the academic losses incurred by students. • Because learning slowed during the initial phase of the pandemic, to make up for those losses, learning must accelerate to rates above normal in order to catch-up for those loses. • Data from two sources was utilized to examine whether recovery is occurring: • State summative data from 2021 and 2022 • WIDA-ACCESS data from 2021 and 2022
  20. Four Academic Recovery Scenarios: V-, U-, L-, and No- recovery
  21. ILEARN ELA and Mathematics Indiana 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Stabilization Stabilization Deterioration Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Stabilization Stabilization Stabilization Deterioration Stabilization Deterioration Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization African American Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Recovery Stabilization Stabilization Deterioration Stabilization Deterioration Asian Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization White Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Hispanic Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Stabilization Stabilization Stabilization Deterioration Stabilization Deterioration ELL Stabilization Stabilization Deterioration Deterioration Deterioration Deterioration Stabilization Stabilization Stabilization Stabilization Deterioration Deterioration Special Education Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Male Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
  22. Academic Impact/Recovery Across 11 States Mean Growth Impact Mean Growth Recovery by Grade by Grade Mean Status Recovery HI3 0.25 Mathematics Impact: 2019 to 2021 HI5 ELA Impact: 2019 to 2021 by Grade HI4 0.20 Mathematics Recovery: 2021 to 2022 AZ3 ELA Recovery: 2021 to 2022 AZ7 Recovery 0.15 HI3 HI6 NH4 AZ3 UT4 HI8 AZ5 NH5HI4 3 NH6 4 0.10 NH3 IN4 HI5AZ8 UT4 AZ6 IL4 IL3 UT3 UT3 ID3 IN3 ID6 ID4 IL4AZ8 AZ4 5 3 HI6UT6 NH7 UT6 HI7 UT5 6 ID4 NH6 0.05 ID3 IN3 IN6 ID5IN5 IN4 UT5 AZ6 ID7 UT7 ID5 IN7 8 7 ID6 IN8 IN5 ID7 IL5 7 IL3 NH3 ID8UT8 AZ4 UT7 0.00 HI7NH8 Status Change (effect size) IL8IN7 AZ7 NH5IL5 IL7 ID8 IN8 HI8 UT7 IL7 IL6 8 ID4 IN6 UT8 UT8 NH7 −0.05 HI7HI8 NH4 IL6 NH8 ID3 −0.10 HI4 AZ10 NH5 RI4 IL8 UT3 UT6 AZ7GA7 HI6 AZ8 UT5 AZ4 ID4 RI7 8 −0.15 NH4 GA4 ID3 UT4 IN4 GA8 AZ5 GA8 HI5AZ6 UT7 UT6 UT5 NH8 NH6 4 GA3 MA4 GA5 GA6 RI8 IN8 NH7 6 GA4 UT3 NH3 AZ8 AZ5RI5 RI7 UT8 RI6 IL8IN6 −0.20 IN3 MA3 HI3 HI7 IL7 IN5 IN7 Correlation: 0.91 3 GA3 IN3 RI3 IN7 IL6 7 AZ7 −0.25 NH4 IL4 NH3 IN4 IL3 GA5 GA7 IN5 IL6 IN8 Status = 0.012 * Growth + 0.005 8 UT4 IL3 IL8 Growth = 66.204 * Status + −1.523 4 IL7 IL4 NH8 NH5 3 −0.30 HI4 HI6 IN6 RI5HI8 AZ3 IL5 GA6 AZ6 HI5 NH6 −0.35 AZ4 RI6 NH7 IL5 Mean Status Impact RI8 RI4 AZ5 AZ3 RI3 −0.40 HI3 Impact by Grade MA3 MA4 −0.50 Note: Missing data in 2021 imputed using multiple imputation. Impact Growth Change (Median SGP change) Recovery
  23. Questions