Talks archive, 2020–2026 – COVID-19 Impact & Recovery

COVID-19 Impact & Recovery November 2022 Update Damian Betebenner & Adam VanIwaarden — Center for Assessment Hawaii DOE November 30th, 2022

Background Introductory Concepts and Terminology

COVID-19 Pandemic • The COVID-19 Pandemic led to, arguably, the largest educational disruption in the history of the United States. • Two-and-a-half years into the pandemic and the impacts of the pandemic are still being felt. • The pandemic impacted students academically, emotionally, physically, and financially. • In this overview, we address three things: • Academic Impact: What is it and what happened to students in Hawaii? • Academic Recovery: What is it and what is happening to students in Hawaii currently? • Now What?: How can impact and recovery data be leveraged to help students going forward? Evidence-based investigations of what is working.

What is Academic Impact? • The pandemic and all the ensuing disruptions functioned/ functions as an “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • In education these two impediments manifest as: ▪ Decrease in student growth (i.e. decrease in velocity = deceleration). ▪ Decrease in student attainment.

Assessment Data • We investigated the impacts in achievement and growth of the pandemic across two time frames: • Impact: Spring 2020 through Summer 2021 • Recovery: Fall 2021 through Summer 2022 • To investigate impact we utilized two primary sources of data • State summative assessment data (SBA) • Impact: SBA Spring 2019 to Spring 2021 (SBA testing cancelled in 2020) • Recovery: SBA Spring 2021 to Spring 2022 • State English Language Proficiency Data (ACCESS) • Impact: ACCESS Winter 2020 to Winter 2021 • Recovery: ACCESS Winter 2021 to Winter 2022

Impact From Spring 2020 to Summer 2021 SBA: Spring 2019 and Spring 2021 Assessments ACCESS: Winter 2020 and Winter 2021 Assessments

SBA Mathematics Grade 7 Hawaii 2019 to 2021 COVID⌧19 Academic Impact All Students Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary Observed + Imputed 2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score Academic Impact: Overall and by Quintile Grade 7 Scaled/Standardized Score and Achievement Level Cuts Observed Overall 2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed) 2021 Grade 7 Scaled Score Decile Cuts 2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large 2019 to 2021 Growth Impact 90 1 2019 to 2021 Status Impact 2019 and 2021 Mean Scaled Scores 0.5 Moderate Moderate Large Large Severe 2021 Grade 7 Scaled Score Participation 30 Total Status Growth 2300 2300 −2 2019: 38.2% Proficient −2.5 2021: 29.2% Proficient Growth/Status Change Legend 2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 7 Students 51.0 50.0 51.0 50.0 50.0 50.0 2019 Grade 7 Students 2363 (−1.44) 2465 (−0.54) 2531 ( 0.04) 2592 ( 0.58) 2680 ( 1.35) 2526 (0.00) 2021 Grade 7 Students 46.0 37.0 30.0 26.0 19.0 30.0 2021 Grade 7 Students 2348 (−1.57) 2450 (−0.67) 2508 (−0.16) 2565 ( 0.34) 2656 ( 1.14) 2505 (−0.18) Calculations suppressed Academic Impact (Observed) −5.0 −13.0 −21.0 −24.0 −31.0 −20.0 Change (Observed) −15 (−0.13) −14 (−0.13) −23 (−0.20) −27 (−0.24) −24 (−0.21) −21 (−0.18) Academic Impact (+ Imputed) −5.0 −10.0 −17.0 −21.8 −27.5 −16.0 Change (+ Imputed) −12 (0.23) −15 (0.24) −23 (0.18) −27 (0.16) −25 (0.21) −24 (−0.21) due to small group N (< 50) Count Observed 2019 2,210 2,308 2,375 2,368 2,408 11,669 Count Observed 2019 2,715 2,734 2,704 2,736 2,719 13,608 Count Observed 2021 1,785 1,907 2,037 2,059 2,187 9,975 Count Observed 2021 2,241 2,237 2,216 2,238 2,235 11,167 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 0.40 2021 to 2019 Grade 7 Status Change (Observed) 2021 to 2019 SGP Difference (+ Imputed) 2021 to 2019 Grade 7 Status Change (+ Imputed) 40 40 0.35 0.93: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 1.00: Correlation SGP and Conditional Status Difference 0.30 2019 to 2021 2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Ranks and Scaled Scores (2019 Norm)

Summary We summarize the academic impact both overall and by achievement quintile For Grade 7 mathematics, the overall academic impact for students was large (z decrease = -0.18 and median SGP decrease = Academic impact was not uniform across students. For this grade/content area higher achieving students were more severely impacted that lower achieving student.

SBA ELA and Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large Economically Disadvantaged Students Moderate Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large Non−Economically Disadvantaged Students Moderate Modest to None Large Moderate Moderate Moderate Large Large Severe Severe Large Large Special Education Students Moderate Modest to None Moderate Modest to None Modest to None Modest to None Large Moderate Moderate Moderate Modest to None Modest to None Asian Students Moderate Modest to None Moderate Large Moderate Moderate Large Large Severe Severe Severe Severe White Students Moderate Moderate Large Large Moderate Moderate Large Moderate Severe Large Large Large Hispanic Students Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe Native Hawaiian Students Moderate Moderate Moderate Moderate Moderate Modest to None Severe Large Severe Large Moderate Large ELL Students Moderate Improvement Moderate Moderate Moderate Moderate Large Moderate Large Large Moderate Large Female Students Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Severe Large Large Male Students Large Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large

WIDA ACCESS WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students

All Students Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Hawaii Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State A Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe State B Moderate Moderate Moderate Moderate Large Moderate State C Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State D Moderate Moderate Large Large Large Large Large Moderate Severe Severe Large Severe State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe State G Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State H Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State I Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate

All Students Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None

Recovery From Fall 2021 to Summer 2022 SBA: Spring 2021 and Spring 2022 Assessments ACCESS: Winter 2021 and Winter 2022 Assessments

Recovery Phase • During the 2021-2022 academic year, school returned to normal for most students. • The hope was that a return to normal would lead recovery of the academic losses incurred by students. • Because learning slowed during the initial phase of the pandemic, to make up for those losses, learning must accelerate to rates above normal in order to catch-up for those loses. • Data from two sources was utilized to examine whether recovery is occurring: • SBA data from 2021 and 2022 • WIDA-ACCESS data from 2021 and 2022

Four Academic Recovery Scenarios with SBA: V-, U-, L-, and No-recovery

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.55 −0.07 0.1 0.2 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Economically Disadvantaged Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Non−Economically Disadvantaged Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Asian Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Recovery Recovery Stabilization Hispanic Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Native Hawaiian Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Recovery Recovery Stabilization ELL Stabilization Deterioration Stabilization Stabilization Stabilization Deterioration Recovery Stabilization Recovery Stabilization Stabilization Deterioration Female Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Male Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Recovery Recovery Stabilization

Overall Recovery • Analysis of 2022 SBA and ACCESS data indicates a lot of trends in Hawaii to be happy about. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Hawaii comes out as the best in terms of pandemic recovery. • Results from 2023 data will be critical to assess whether these exemplary rates of learning (beyond what the state experience pre-COVID) are being maintained.

Academic Impact/Recovery Across 6 States Impact & Recovery Mathematics HI4 ELA 15 HI6 HI5 Hawaii showed large impacts 10 HI8 HI7 Recovery HI4 and recovery 5 HI3 HI5 HI6 HI8 HI7 Hawaii’s academic recovery 0 HI3 Academic Recovery results are the best I have seen −5 nationally −10 A critical “test” of this recovery −20 will be the 2023 results. If rates −25 Deterioration of learning in 2023 extend those −30 of 2022, then Hawaii will have −35 come out of the pandemic −40 better than before it started Large Impact Academic Impact Improvement

Now What Leveraging Assessment Data to Help Students Overcome COVID-19 Academic Impacts

Exemplar Schools: Impact & Recovery • Original discussions about examining ROI were redirected to examine Return on Activities: • Identify outlier schools in terms of impact/recovery. • Investigate whether activities (or lack thereof) could potentially be responsible for the school’s outlier status. • For schools with plausible activities leading to exemplary recovery, investigate links to relevant investment/funding. • Step 1 is to find these outlier schools. • We investigated schools along two dimensions: • Academic Impact (2019 to 2021 decrease in student learning) • Academic Recovery (2021 to 2022 increase in student learning relative to pre- pandemic 2018-2019 learning)

School Impact−Recovery: 2019 to 2021 and 2021 to 2022 Elementary Schools by Poverty: ELA & Math Combined Recovery + Outliers Academic Recovery 2021 to 2022 − Outliers Deterioration Free/Reduced Lunch: < 30% Free/Reduced Lunch: 30% to 45% Free/Reduced Lunch: 45% to 60% Free/Reduced Lunch: 60% to 75% Free/Reduced Lunch: > 75% Severe Impact Academic Impact 2019 to 2021 No Impact

Exemplar Schools: Impact & Recovery

Exemplar Schools: Impact & Recovery • As can be seen, Hawaii and its schools experienced a largely positive 2021-2022 • We’ve created Excel spreadsheets with the identified schools to help to sort through the many potential criteria to use to select exemplar schools • As we discussed in Portsmouth, there was some concern that of low motivation leading to poor results. To examine we filtered on students with very low to very high and very high to very low growth. • Filtering out these students led to non-appreciable changes to the impact- recovery examinations of schools.

Questions

Status Impact Black Line: 2019 (pre-covid) scaled-score distribution. Red line: 2021 scaled-score distribution. The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (top) and red line (bottom) X-axis: Grade 7 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the decrease in percent proficient. A dotted red line indicates the distribution of scores including missing students derived through multiple imputation.

Status Impact Red line: 2021-2019 scaled-score distribution difference. The impact of the pandemic (difference in the distributions) on student achievement (decrease) is indicated by the red line. X-axis: Grade 7 percentile rank Average impact z = -0.18 standard units. A dotted red line indicates the distribution of scores including missing students derived through multiple imputation. Larger impact for high achieving students than lower achieving students.

Growth Impact Black Line: (pre-covid) conditional scaled-score distribution — Grade 7/2019 conditioned on Grade Red line: conditional scaled-score distribution — Grade 7/2021 conditioned on Grade 5/2019 The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (top) and red line (bottom) X-axis: Grade 5 Scaled Score Fixing a level on the x-axis, the vertical distance between the curves is the average decrease in scale score associated with the pandemic for a student at that 5th grade level A dotted red line indicates the distribution of scores including missing students derived through multiple imputation.

Growth Impact Red line: SGP Difference Grade 7/2021 - Grade 5/2019 conditional scaled score distribution X-axis: Grade 5 Scaled Score Median SGP 2019 Grade 7: 50 Median SGP 2021 Grade 7: 30 Top quintile students median SGP dropped from 50 to 19 whereas bottom quintile students dropped from 51 to 46.

ELA And Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Students Non−Economically Disadvantaged Students Special Education Students Asian Students White Students Hispanic Students Native Hawaiian Students ELL Students Female Students Male Students

Status Recovery Black Line: 2021 (covid) scaled-score distribution. Red line: 2022 scaled-score distribution. The recovery from the pandemic on student achievement (increase) is illustrated as the area between the black line (bottom) and red line (top) X-axis: Grade 5 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the increase in percent proficient.

Status Recovery Red line: 2022 - 2021 scaled-score distribution difference. The recovery from the pandemic (difference in the distributions) on student achievement (increase) is indicated by the red line. X-axis: Grade 5 percentile rank Average impact z = +0.23 standard units. Larger recovery for high achieving students than lower achieving students.

Growth Recovery Black Line: (pre-covid) conditional scaled-score distribution — Grade 5/2019 conditioned on Grade 4/2018 Red line: conditional scaled-score distribution — Grade 5/2022 conditioned on Grade 4/2021 The recovery from the pandemic on student achievement (increase) is illustrated as the area between the black line (bottom) and red line (top) X-axis: Grade 4 Scaled Score Fixing a level on the x-axis, the vertical distance between the curves is the average increase in scale score associated with the recovery for a student at that 5th grade level

Growth Recovery Red line: SGP Difference Grade 5/2022 - Grade 4/2021 conditional scaled score distribution X-axis: Grade 4 Scaled Score Median SGP 2019 Grade 5: 50 Median SGP 2022 Grade 5: 65 Largest acceleration in growth seen for quintiles 1 to 4 with median SGPs ranging from 65 to 69.

Baseline SGP: What is Recovery • Given the negative pandemic impacts, any recovery implies learning acceleration. • However, not all acceleration can be considered recovery • Acceleration versus 2021 versus acceleration versus 2019 V-Shaped Recovery (> 60) 60 U-Shaped Recovery (50 to 60) Median 2021 Baseline SGP 50 Median 2022 Baseline SGP L-Shaped “Recovery” (40 to 50) Downward Spiral (< 40) Note: Any median SGP above 30 in 2022 would be considered acceleration. But only values 10 above 50 would be sufficient to begin catching students up to where they would have been absent the pandemic.

ELA And Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile Deterioration Recovery −30 −7 10 30 Growth Change NOTE: Quintiles for each student −0.55 −0.07 0.1 0.2 Status Change subgroup are based upon quintile Calculations suppressed cuts defined for All Students. Deterioration Stabilization Recovery due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

School Impact−Recovery: 2019 to 2021 and 2021 to 2022 Middle Schools by Poverty: ELA & Math Combined Recovery + Outliers Academic Recovery 2021 to 2022 − Outliers Deterioration Free/Reduced Lunch: < 30% Free/Reduced Lunch: 30% to 45% Free/Reduced Lunch: 45% to 60% Free/Reduced Lunch: 60% to 75% Free/Reduced Lunch: > 75% Severe Impact Academic Impact 2019 to 2021 No Impact

School Impact−Recovery: 2019 to 2021 and 2021 to 2022 Elementary−Middle Schools by Poverty: ELA & Math Combined Recovery + Outliers Academic Recovery 2021 to 2022 − Outliers Deterioration Free/Reduced Lunch: < 30% Free/Reduced Lunch: 30% to 45% Free/Reduced Lunch: 45% to 60% Free/Reduced Lunch: 60% to 75% Free/Reduced Lunch: > 75% Severe Impact Academic Impact 2019 to 2021 No Impact

About this copy

COVID-19 Impact & Recovery — Hawaii Department of Education, December 1, 2022.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.