Talks archive, 2020–2026 – COVID-19 Impact & Recovery in Hawaii

COVID-19 Impact & Recovery in Hawaii 2023 Update Damian Betebenner & Adam VanIwaarden Center for Assessment August 8th, 2023

Background Introductory Concepts and Terminology

Academic Recovery in 2023 • The availability of 2023 SBA data allows Hawai’i to update COVID-19 recovery analyses. • Spring 2023 marks the third year since the pandemic began and this will be the third report: • Summer 2021: Academic Impact • Summer 2022: Academic Recovery Year 1 • Summer 2023: Academic Recovery Year 2

Two Types of Recovery: System & Student • In 2023 we elaborate on the results from the recovery analyses in two ways: • System level recovery: Is the education system supporting rates of learning that are comparable to what it was prior to the pandemic? • Student level recovery: Are student attaining at levels that are comparable to what they were prior to the pandemic. • Previously, we focused mostly on system level recovery: examining rates of learning in 2021 and 2022 compared to those prior to the pandemic. • We continue system level analyses but also focus in on student level recovery.

Two Types of Recovery: System & Student • System level recovery • Comparison of 2019 and 2023 same grade student growth: We examine change in baseline student growth (SGPs) for 2019 students and 2023 students utilizing coefficient matrices derived from pre-pandemic cohorts of students. • Student level recovery • Comparison of 2019 to 2023 same grade attainment: We examine equi- percentile change in attainment for 2019 students and 2023 students.

System Recovery: Cohort-Based Comparison (SBA) Compare observed versus expected attainment for a cohort of students Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8

Student Recovery: Cross-Sectional Comparison (SBA) Compare same grade/content area attainment across di erent years Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8 ff

Recovery Year 1 From Fall 2021 to Summer 2022 SBA: Spring 2021 and Spring 2022 Assessments

Recovery Phase Year 1 • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Growth was characterized in three ways: • Deterioration: Growth less than pre-pandemic rates of learning • Stabilization: Growth the same as pre-pandemic rates of learning • Recovery: Growth exceeding pre-pandemic rates of learning • Because learning slowed during the initial phase of the pandemic, to make up for those losses, learning must accelerate to rates above normal in order to catch-up for those loses (recovery).

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Non−Economically Disadvantaged Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Special Education Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Asian Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Native Hawaiian Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization ELL Stabilization Stabilization Deterioration Stabilization Stabilization Deterioration Recovery Recovery Stabilization Stabilization Stabilization Deterioration Female Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Male Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization

Recovery Year 1 • Analysis of 2022 SBA data indicated several positive trends in Hawai’i to be happy about. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Hawai’i comes out as the best in terms of pandemic recovery. • Results from 2023 data will be critical to assess whether these exemplary rates of learning (beyond what the state experience pre-COVID) are being maintained.

Academic Impact/Recovery Across 8 States Impact & Recovery 20 HI5 Mathematics ELA Hawaii showed large impacts HI4 HI6 10 HI5 and recovery Recovery HI8 HI4 5 HI7 HI8 HI6 Hawaii’s academic recovery 0 HI3 HI3 HI7 results are the best I have seen Academic Recovery nationally −10 A critical “test” of this recovery −20 will be the 2023 results. If rates −25 Deterioration of learning in 2023 extend those −30 of 2022, then Hawaii will have −35 come out of the pandemic −40 better than before it started −40 −35 −30 −25 −20 −15 −10 −5 0 5 10 15 20 Large Impact Academic Impact Improvement

Recovery Year 2 From Fall 2022 to Summer 2023 SBA: Spring 2022 and Spring 2023 Assessments

Recovery Phase Year 2 • As mentioned previously, we distinguish between student level recovery and system level recovery this year. • Because student level recovery utilizes status comparisons between 2019 and 2023 and system level recovery utilizes growth comparisons between 2019 and 2023, status and growth was characterized in three ways: • Below: Attainment/Growth of students is less in 2023 than in 2019 • Comparable: Attainment/Growth of students is approximately the same in 2023 as in 2019. • Above: Attainment/Growth of students is greater in 2023 than in 2019

SBA ELA and Mathematics Hawaii 2022 to 2023 COVID⌧19 Academic System Recovery Grade & Content Area Behind Ahead System Recovery characterizes the extent Status Change to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Behind Comparable Ahead due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Economically Disadvantaged Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Non−Economically Disadvantaged Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Special Education Comparable Comparable Comparable Comparable Comparable Behind Comparable Comparable Comparable Comparable Comparable Comparable Asian Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable White Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Hispanic Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Native Hawaiian Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable ELL Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Female Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Male Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable

SBA ELA and Mathematics Hawaii 2022 to 2023 COVID⌧19 Academic System Recovery Grade, Content Area & Achievement Quintile ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

Student level recovery • It is critical to understand that recovery of the system does not imply student level recovery • Because students rates of learning slowed so dramatically during the pandemic, rates of learning necessary to catch students up to where they would have been would have to be equally dramatic (in the positive direction) • The following figures illustrate the extent to which student attainment is behind 2019 attainment levels. • Comparison to 2019 levels is a “low bar” as attainment levels in 2023 had the pandemic not occurred would almost certainly have been higher than 2019 levels.

SBA ELA and Mathematics Hawaii 2019 to 2023 COVID⌧19 Academic Student Recovery Grade & Content Area Behind Ahead Student Recovery characterizes the extent Status Change to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Behind Comparable Ahead due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Behind Behind Economically Disadvantaged Behind Comparable Comparable Comparable Comparable Comparable Behind Comparable Behind Behind Behind Behind Non−Economically Disadvantaged Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Behind Behind Special Education Behind Comparable Comparable Behind Behind Comparable Comparable Comparable Comparable Comparable Comparable Comparable Asian Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable White Comparable Comparable Behind Comparable Behind Comparable Comparable Comparable Comparable Comparable Comparable Behind Hispanic Behind Comparable Comparable Behind Behind Comparable Behind Comparable Comparable Behind Behind Behind Native Hawaiian Behind Comparable Comparable Comparable Comparable Behind Behind Comparable Behind Comparable Behind Behind ELL Comparable Comparable Comparable Ahead Ahead Ahead Comparable Comparable Comparable Comparable Comparable Comparable Female Comparable Comparable Comparable Comparable Behind Behind Comparable Comparable Comparable Behind Behind Behind Male Behind Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable Comparable

SBA ELA and Mathematics Hawaii 2019 to 2023 COVID⌧19 Academic Student Recovery Grade, Content Area & Achievement Quintile Behind Ahead Student Recovery characterizes the extent Status Change to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Behind Comparable Ahead due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

Recovery Year 2 • In some grades and content areas, student are well behind where their 2019 counterparts were. • This shows that even though Hawaii’s education system has bounced back well, large deficits remain to be overcome for thousands of students • Of particular concern is middle school math, especially for females.

Questions

About this copy

COVID-19 Impact & Recovery in Hawaii — Hawaii Department of Education, August 8, 2023.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.