Talks archive, 2020–2026 – COVID-19 Impact & Recovery

COVID-19 Impact & Recovery Background, Results, and Next Steps Damian Betebenner, Jeri Thompson, Chris Brandt & Adam VanIwaarden Center for Assessment March 23rd, 2023

Background Concepts and Terminology

COVID-19 Pandemic • The COVID-19 Pandemic led to the largest disruption of education in the history of the United States. • Three years into the pandemic the impacts are still being felt by students: academically, emotionally, physically, and financially and educators. • In this presentation, we address three things: 1. Academic Impact: What is it and what happened to students in Hawaii? 2. Academic Recovery: What is it and what is happening to students in Hawaii currently? 3. Now What?: How can impact and recovery data be leveraged to help students going forward? Evidence-based investigations of what is working.

What is Academic Impact? • Academic Impact: Also referred to as learning loss or unfinished learning. • The pandemic and all its ensuing disruptions functioned as an “academic headwind”, impeding the academic progress (i.e., learning) of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • Academic impact decreases the rate of student learning (deceleration) which, in turn, leads to lower attainment.

What is Academic Recovery? • Academic Recovery: Also referred to as learning recovery • Academic recovery refers to increases in student learning sufficient to ameliorate previous academic impacts. • Academic recovery is analogous to a tailwind, increasing the rate of student learning (acceleration) which, in turn leads to higher attainment. • To understand whether academic recovery is occurring, one must know the extent of academic impact.

Impact & Recovery: Educational Assessment • Hawaii’s Smarter Balanced and WIDA-ACCESS are ideally suited to help investigate academic impact and recovery. • With educational assessments, impact/recovery manifest as: • Decrease/increase in student growth • Decrease/increase in velocity = deceleration/acceleration • Decreases/increases in student growth lead to decreases/ increases in student attainment • Growth is to speedometer as attainment is to odometer.

Impact & Recovery: Educational Assessment • Two time frames: • Impact: Spring 2020 through Summer 2021 • Recovery: Fall 2021 through Summer 2022 • Two assessments: • Smarter Balanced Assessment Data (ELA & Mathematics) • Impact: Spring 2019 to Spring 2021 (Testing cancelled in 2020) • Recovery: Spring 2021 to Spring 2022 • WIDA-ACCESS English Language Proficiency Data • Impact: Winter 2020 to Winter 2021 • Recovery: Winter 2021 to Winter 2022 • Two methods: • Attainment: Change in student attainment (odometer) • Growth: Change in student growth (speedometer)

Impact From Spring 2020 to Summer 2021 Smarter Balanced: Spring 2019 and Spring 2021 Assessments ACCESS: Winter 2020 and Winter 2021 Assessments

Impact Phase • During spring 2019-2020 and the 2020-2021 academic year, school was extremely disrupted due to pandemic related mitigation efforts. • Due to all the disruptions the expectation was that student learning would be impacted (i.e., would slow down) leading to lower student growth and lower student attainment. • Data from two sources was utilized to examine the extent to which the pandemic impacted student learning: • Smarter Balanced data from 2019 and 2021 • WIDA-ACCESS data from 2020 and 2021

Academic Impact •Using student attainment and growth, academic impact is categorized along a continuum ranging from severe to modest to none. •Academic impact is examined both overall and by academic quintile. •Academic impact was examined by each grade and content area tested for all students as well as for several demographic subgroups. •Non-participation in 2021 assessments was larger than normal. We utilized multiple imputation techniques to understand the extent to which missing data impacted observed results.

SBA ELA and Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Deterioration Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large Economically Disadvantaged Moderate Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large Non−Economically Disadvantaged Moderate Modest to None Large Moderate Moderate Moderate Large Large Severe Severe Large Large Special Education Moderate Modest to None Moderate Modest to None Modest to None Modest to None Large Moderate Moderate Moderate Modest to None Modest to None Asian Moderate Modest to None Moderate Large Moderate Moderate Large Large Severe Severe Severe Severe White Moderate Moderate Large Large Moderate Moderate Large Moderate Severe Large Large Large Hispanic Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe Native Hawaiian Moderate Moderate Moderate Moderate Moderate Modest to None Severe Large Severe Large Moderate Large ELL Moderate Improvement Moderate Moderate Moderate Moderate Large Moderate Large Large Moderate Large Female Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Severe Large Large Male Large Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large

WIDA ACCESS WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students

All Students Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Hawaii Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State A Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe State B Moderate Moderate Moderate Moderate Large Moderate State C Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State D Moderate Moderate Large Large Large Large Large Moderate Severe Severe Large Severe State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe State G Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State H Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State I Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate

All Students Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None

Recovery From Fall 2021 to Summer 2022 Smarter Balanced: Spring 2021 and Spring 2022 Assessments ACCESS: Winter 2021 and Winter 2022 Assessments

Recovery Phase • During the 2021-2022 academic year, school returned to normal for most students. • The hope was that a return to normal would lead recovery of the academic losses incurred by students. • Because learning slowed during the initial phase of the pandemic, to make up for those losses, learning must accelerate to rates above normal in order to catch-up for those loses. • Data from two sources was utilized to examine whether recovery is occurring: • Smarter Balanced data from 2021 and 2022 • WIDA-ACCESS data from 2021 and 2022

Academic Recovery •Using student attainment and growth, academic recovery is categorized along a continuum ranging from deterioration to recovery. •Academic impact is examined both overall and by academic quintile. •Academic impact was examined by each grade and content area tested for all students as well as for several demographic subgroups.

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Non−Economically Disadvantaged Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Special Education Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Asian Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Native Hawaiian Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization ELL Stabilization Stabilization Deterioration Stabilization Stabilization Deterioration Recovery Recovery Stabilization Stabilization Stabilization Deterioration Female Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Male Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization

Overall Recovery • Analysis of 2022 SBA and ACCESS data indicates a lot of trends in Hawaii to be happy about. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Hawaii comes out as the best in terms of pandemic recovery. • Results from 2023 data will be critical to assess whether these exemplary rates of learning (beyond what the state experience pre-COVID) are being maintained.

Now What Leveraging Assessment Data to Help Students Overcome COVID-19 Academic Impacts

Exemplar Schools: Impact & Recovery • Original discussions about examining ROI were refined to examine Return on Activities: • Identify outlier schools in terms of impact/recovery. • Investigate whether activities (or lack thereof) could potentially be responsible for the school’s outlier status. • For schools with plausible activities leading to exemplary recovery, investigate links to relevant investment/funding. • Step 1 is to find these outlier schools. • We investigated schools along two dimensions: • Academic Impact (2019 to 2021 decrease in student learning) • Academic Recovery (2021 to 2022 increase in student learning relative to pre- pandemic 2018-2019 learning)

Exemplar Schools: Impact & Recovery • We searched for exemplar schools to place into three categories • Large Impact/Large Recovery (9 schools) • Minimal Impact/Large Recovery (4 schools) • Large Impact/Poor Recovery (3 schools) • 16 schools were sampled across a diverse range of socio-economic and demographic characteristics with the goal of finding replicable strategies that can be used by Hawaii schools to help students recover.

Exemplar Schools: Impact & Recovery • Some initial findings: • Distributive leadership works well in “normal” situations, authoritative leadership is necessary during a crisis. • Everyone was in crisis mode from March 2020-August 2020 • Strong leadership adapted existing / implemented new programs quickly and effectively • Funding was not a benefit or limitation for recovery • Strong leadership went into the community, school funds, foundations – found the money when needed or paid out of their own pocket for food distribution, providing masks, providing other health services • Strong leadership focused on the whole school community including parents and staff • Data use for identifying struggling students was flexible

Questions

Supplementary Slides

Status Impact Black Line: 2019 (pre-covid) scaled-score distribution. Red line: 2021 scaled-score distribution. The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (top) and red line (bottom) X-axis: Grade 7 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the decrease in percent proficient. A dotted red line indicates the distribution of scores including missing students derived through multiple imputation.

Status Impact Red line: 2021-2019 scaled-score distribution difference. The impact of the pandemic (difference in the distributions) on student achievement (decrease) is indicated by the red line. X-axis: Grade 7 percentile rank Average impact z = -0.18 standard units. A dotted red line indicates the distribution of scores including missing students derived through multiple imputation. Larger impact for high achieving students than lower achieving students.

Growth Impact Black Line: (pre-covid) conditional scaled-score distribution — Grade 7/2019 conditioned on Grade Red line: conditional scaled-score distribution — Grade 7/2021 conditioned on Grade 5/2019 The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (top) and red line (bottom) X-axis: Grade 5 Scaled Score Fixing a level on the x-axis, the vertical distance between the curves is the average decrease in scale score associated with the pandemic for a student at that 5th grade level A dotted red line indicates the distribution of scores including missing students derived through multiple imputation.

Growth Impact Red line: SGP Difference Grade 7/2021 - Grade 5/2019 conditional scaled score distribution X-axis: Grade 5 Scaled Score Median SGP 2019 Grade 7: 50 Median SGP 2021 Grade 7: 30 Top quintile students median SGP dropped from 50 to 19 whereas bottom quintile students dropped from 51 to 46.

ELA And Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Students Non−Economically Disadvantaged Students Special Education Students Asian Students White Students Hispanic Students Native Hawaiian Students ELL Students Female Students Male Students

Status Recovery Black Line: 2021 (covid) scaled-score distribution. Red line: 2022 scaled-score distribution. The recovery from the pandemic on student achievement (increase) is illustrated as the area between the black line (bottom) and red line (top) X-axis: Grade 5 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the increase in percent proficient.

Status Recovery Red line: 2022 - 2021 scaled-score distribution difference. The recovery from the pandemic (difference in the distributions) on student achievement (increase) is indicated by the red line. X-axis: Grade 5 percentile rank Average impact z = +0.23 standard units. Larger recovery for high achieving students than lower achieving students.

Growth Recovery Black Line: (pre-covid) conditional scaled-score distribution — Grade 5/2019 conditioned on Grade 4/2018 Red line: conditional scaled-score distribution — Grade 5/2022 conditioned on Grade 4/2021 The recovery from the pandemic on student achievement (increase) is illustrated as the area between the black line (bottom) and red line (top) X-axis: Grade 4 Scaled Score Fixing a level on the x-axis, the vertical distance between the curves is the average increase in scale score associated with the recovery for a student at that 5th grade level

Growth Recovery Red line: SGP Difference Grade 5/2022 - Grade 4/2021 conditional scaled score distribution X-axis: Grade 4 Scaled Score Median SGP 2019 Grade 5: 50 Median SGP 2022 Grade 5: 65 Largest acceleration in growth seen for quintiles 1 to 4 with median SGPs ranging from 65 to 69.

Baseline SGP: What is Recovery • Given the negative pandemic impacts, any recovery implies learning acceleration. • However, not all acceleration can be considered recovery • Acceleration versus 2021 versus acceleration versus 2019 V-Shaped Recovery (> 60) 60 U-Shaped Recovery (50 to 60) Median 2021 Baseline SGP 50 Median 2022 Baseline SGP L-Shaped “Recovery” (40 to 50) Downward Spiral (< 40) Note: Any median SGP above 30 in 2022 would be considered acceleration. But only values 10 above 50 would be sufficient to begin catching students up to where they would have been absent the pandemic.

ELA And Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile Deterioration Recovery −30 −7 10 30 Growth Change NOTE: Quintiles for each student −0.55 −0.07 0.1 0.2 Status Change subgroup are based upon quintile Calculations suppressed cuts defined for All Students. Deterioration Stabilization Recovery due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

About this copy

COVID-19 Impact & Recovery — Hawaii Department of Education, March 23, 2023.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.