Talks archive, 2020–2026 – COVID-19 Impact & Recovery in Hawaii

COVID-19 Impact & Recovery in Hawaii 2023 Update Damian Betebenner & Adam VanIwaarden Center for Assessment November 8th, 2023

Background Introductory Concepts and Terminology

Academic Recovery in 2023 • The availability of 2023 SBA data allows Hawaii to update COVID-19 recovery analyses. • Spring 2023 marks the third year since the pandemic began and this is the third report: • Summer 2021: Academic Impact • Summer 2022: Academic Recovery Year 1 • Summer 2023: Academic Recovery Year 2

Two Types of Recovery in 2023: System & Student • We elaborate on the results from the previous recovery analyses in two ways: • System level recovery: Is the education system supporting rates of learning that are comparable to what it was prior to the pandemic? • Student level recovery: Are students achieving at levels that are comparable to what they were prior to the pandemic? • Previously, we focused mostly on system-level recovery: examining rates of learning in 2021 and 2022 compared to those prior to the pandemic. • We continue examining system level analyses but also focus on student- level recovery.

Two Types of Recovery: System & Student • System-level recovery • Comparison of 2019 and 2023 same-grade student growth: We examined the change in baseline student growth (SGPs) for 2019 students and 2023 students utilizing coefficient matrices derived from pre-pandemic cohorts of students. • Student-level recovery • Comparison of 2019 to 2023 same-grade achievement: We examined the equipercentile change in achievement for 2019 students and 2023 students at same-grade

Student-level recovery • It is critical to understand that recovery of the system does not imply student level recovery • Because students’ rates of learning slowed so dramatically during the pandemic, rates of learning necessary to catch students up to where they would have been would have to be equally dramatic (in the positive direction) • The following figures illustrate the extent to which student attainment is behind 2019 attainment levels. • Comparison to 2019 attainment levels is a “low bar” for comparison: • Attainment levels in 2023, had the pandemic not occurred, would almost certainly have been higher than 2019 levels (based on “typical” rates of yearly improvement)

System Recovery: Cohort-Based Comparison (SBA) Compare observed versus expected attainment for a cohort of students Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8

Student Recovery: Cross-Sectional Comparison (SBA) Compare same grade/content area attainment across different years Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8

Why Baseline SGPs? • There was a question of why one needs to look at baseline referenced SGPs instead of, for example, scale score gains. There are several reasons. • SGPs are the growth metric that the state of Hawaii has been using for more than a decade. • SGPs are readily interpretable probabilistically whereas gain scores are not. • Quantile regression (on which SGPs are based) is invariant to mono-tone transformations of scale (equivariant to mono-tone transformations of scale). OLS regression, for example, is not. See Koenker (2005, p. 38) • Gain scores (which can be calculated since SBA is vertically scaled) are dependent upon: Content area, grade, prior level of achievement. • Gain scores were largely discredited as an effective means of communicating change. For example, not state uses them as a growth model.

Status based impact versus growth? • There was another question about the equivalence of status change and growth change. • Previously, I justified this change heuristically by noting that growth change (deceleration from 2021) had to lead to status change. • Also, based upon two years of data plotted for several states across grade and content areas show a correlation of 0.91. • This relationship isn’t a function of content area or year but a relationship between status change and conditional status change.

Status based impact versus growth? Academic Impact/Recovery Across 11 States Mean Growth Impact Mean Growth Recovery by Grade by Grade Mean Status Recovery Gr3 0.25 Mathematics Impact: 2019 to 2021 Gr5 ELA Impact: 2019 to 2021 by Grade Gr4 0.20 Mathematics Recovery: 2021 to 2022 Gr3 ELA Recovery: 2021 to 2022 Gr7 Recovery 0.15 Gr3 Gr6 Gr4 Gr3 Gr4 Gr8 Gr5 Gr5Gr4 3 Gr6 4 0.10 Gr3 Gr4 Gr5Gr8 Gr4 Gr6 Gr4 Gr3 Gr3 Gr3 Gr3 Gr6 Gr4 Gr4Gr8 Gr4 5 3 Gr6Gr6 Gr7 Gr6 Gr7 Gr5 6 Gr4 Gr6 0.05 Gr3 Gr3 Gr6 Gr5Gr5Gr4 Gr5 Gr6 Gr7 Gr7 Gr5 Gr7 8 7 Gr6 Gr5 Gr8 Gr7 Gr4 Gr5 7 6 Gr3 Gr8Gr8 Gr7 0.00 Gr7 Gr8 Status Change (effect size) Gr8Gr7 Gr7 Gr5Gr5 Gr7 Gr8 Gr8 Gr8 Gr7 Gr7 Gr6 8 Gr4 Gr6 Gr8Gr8 Gr7 −0.05 Gr7Gr8 Gr4 Gr6 Gr8 Gr3 −0.10 Gr4 Gr10 Gr5 Gr4 Gr8 Gr3 Gr6 Gr7Gr7 Gr6 Gr8 Gr5 Gr4 Gr7 8 −0.15 Gr4 Gr3 Gr4 Gr4 Gr8 Gr5 Gr8 Gr5Gr6 Gr7 Gr6 Gr5 Gr8 Gr6 4 Gr3 Gr4 Gr5 Gr6 Gr8 Gr8 Gr7 6 Gr4 Gr3 Gr3 Gr8Gr5Gr5 Gr7 Gr8 Gr6 Gr8Gr6 −0.20 Gr3 Gr3 Gr3 Gr7 Gr7 Gr5 Gr7 Correlation: 0.91 3 Gr3 Gr3 Gr7 Gr6 7 Gr7 −0.25 Gr4 Gr4 Gr3 Gr4 Gr3 Gr5 Gr7 Gr5 Gr6 Gr8 Status = 0.012 * Growth + 0.005 8 Gr4 Gr3 Gr8 Growth = 66.204 * Status + −1.523 4 Gr7 Gr4 Gr8 Gr6Gr5 Gr5 3 −0.30 Gr4 Gr6 Gr8 Gr3 Gr5 Gr6 Gr6 Gr5 Gr6 −0.35 Gr4 Gr6 Gr7 Gr5 Mean Status Impact Gr8 Gr4 Gr5 Gr3 −0.40 Gr3 Impact by Grade Gr3 Gr4 −0.50 Note: Missing data in 2021 imputed using multiple imputation. Impact Growth Change (Median SGP change) Recovery

SBA ELA and Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Deterioration Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Impact ratings calculated using baseline growth Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. change when available and transformed growth otherwise. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

SBA ELA and Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Deterioration Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Impact ratings calculated using status change Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. for all grades and content areas. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

Recovery Year 1 From Fall 2021 to Summer 2022 SBA: Spring 2021 and Spring 2022 Assessments

Recovery Phase Year 1 • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Achievement (Student Recovery) and Growth (System Recovery) were characterized in three ways: • Lagging: Behind pre-pandemic levels (attainment or growth) • On Par: About the same as pre-pandemic levels (attainment or growth) • Leading: Ahead of pre-pandemic levels (attainment or growth) • Because learning slowed during the initial phase of the pandemic (2020 to 2021), to make up for those losses, learning must accelerate to rates above normal (i.e., leading)

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic System Recovery Grade & Content Area Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Leading Leading Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Economically Disadvantaged On Par On Par Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Non−Economically Disadvantaged Leading On Par Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Special Education On Par On Par On Par On Par On Par Lagging Leading On Par Leading On Par On Par On Par Asian Leading Leading Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading White On Par On Par Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Hispanic On Par On Par Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Native Hawaiian On Par Leading Leading Leading On Par On Par Leading Leading Leading Leading Leading Leading ELL On Par On Par On Par On Par On Par On Par Leading On Par Leading Leading Leading On Par Female Leading Leading Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading Male Leading Leading Leading Leading Leading On Par Leading Leading Leading Leading Leading Leading

SBA ELA and Mathematics Hawaii 2019 to 2022 COVID⌧19 Academic Student Recovery Grade & Content Area Lagging Leading Student Recovery characterizes the extent Status Change −0.38 −0.09 0.12 0.38 to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students On Par On Par On Par On Par On Par On Par Lagging On Par On Par Lagging Lagging Lagging Economically Disadvantaged Lagging On Par On Par On Par On Par On Par Lagging Lagging Lagging Lagging Lagging Lagging Non−Economically Disadvantaged On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging Special Education On Par On Par On Par Lagging On Par On Par Lagging Lagging On Par Lagging On Par On Par Asian On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging On Par On Par White On Par On Par On Par On Par On Par On Par Lagging On Par On Par On Par On Par Lagging Hispanic Lagging On Par On Par Lagging On Par Lagging Lagging Lagging On Par Lagging Lagging Lagging Native Hawaiian Lagging On Par On Par On Par On Par Lagging Lagging Lagging Lagging Lagging Lagging Lagging ELL On Par On Par On Par Leading Leading Leading On Par Lagging On Par On Par On Par Lagging Female On Par On Par On Par On Par On Par On Par Lagging Lagging On Par Lagging Lagging Lagging Male On Par On Par On Par On Par On Par On Par Lagging On Par On Par On Par On Par On Par

Recovery Year 1 • Analysis of 2022 SBA data indicated several positive trends in Hawai’i to be happy about. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Hawai’i comes out as the best in terms of pandemic recovery. • Despite a robust recovery, students remain slightly behind 2019 levels in ELA and more in mathematics.

Recovery Year 2 From Fall 2022 to Summer 2023 SBA: Spring 2022 and Spring 2023 Assessments

Recovery Phase Year 2 • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Achievement (Student Recovery) and Growth (System Recovery) were characterized in three ways: • Lagging: Behind pre-pandemic levels (attainment or growth) • On Par: About the same as pre-pandemic levels (attainment or growth) • Leading: Ahead of pre-pandemic levels (attainment or growth) • Because learning slowed during the initial phase of the pandemic (2020 to 2021), to make up for those losses, learning must accelerate to rates above normal (i.e., leading)

2023 Status Results • Percent Proficient/Standardized Score (pooled standard deviation)/Count for SBA: ELA Mathematics Grade

2023 SGP Results • Mean/Median/Count for baseline referenced SGPs in ELA and mathematics are as follows: ELA Mathematics Grade

Status Recovery Black Line: 2019 (pre-covid) scaled-score distribution. Red line: 2023 scaled-score distribution. The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (bottom) and the red line (top) X-axis: Grade 8 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the increase in percent proficient. Decrease of 7.2% proficient and decrease of 0.16 standard deviation

Status Recovery Red line: 2023 - 2019 scaled-score distribution difference. The recovery from the pandemic (difference in the distributions) on student achievement (increase) is indicated by the red line. X-axis: Grade 8 percentile rank Average impact z = -0.16 standard units. Larger impact for Q3 and Q4 students.

Growth Recovery Black Line: (pre-covid) conditional scaled-score distribution — Grade 8/2019 conditioned on Grade 7/2018 Red line: conditional scaled-score distribution — Grade 8/2023 conditioned on Grade 7/2022 The lag from the pandemic on student growth (increase) is illustrated as the area between the black line (top) and the red line (bottom) X-axis: Grade 7 Scaled Score and achievement level cuts. Fixing a level on the x-axis, the vertical distance between the curves is the average decrease in scale score associated with the impact for a student at that 8th-grade level.

Growth Recovery Red line: SGP Difference Grade 8/2023 - Grade 8/2019 conditional scaled score distribution X-axis: Grade 7 Scaled Score and associated deciles and achievement level cuts Median SGP 2019 Grade 8: 50 Median SGP 2023 Grade 8: 46 Higher achieving groups (above the 40th percentile) demonstrating lower rates of learning than pre- COVID.

Summary We summarize the academic recovery in terms of Growth (System Level Recovery) and Status (Student Level Recovery) both overall and by achievement quintile Growth is, overall, on par with the pre-covid growth. For the higher quintiles, growth is on-par bordering on lagging. Status is, overall, lagging. That is, student attainment has not returned to pre- pandemic levels. This reaffirms the point made two years ago: For students to recover, learning rates will have to

SBA ELA and Mathematics Hawaii 2019 to 2023 COVID⌧19 Academic Student Recovery Grade, Content Area & Achievement Quintile Lagging Leading Student Recovery characterizes the extent Status Change −0.38 −0.09 0.12 0.38 to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

SBA ELA and Mathematics Hawaii 2019 to 2023 COVID⌧19 Academic System Recovery Grade, Content Area & Achievement Quintile Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

SBA Mathematics Grade 8 Hawaii 2019 to 2023 COVID⌧19 Academic Recovery Female Students Growth: Grade 7/2018 to 8/2019 versus Grade 7/2022 to 8/2023 Status: Grade 8/2019 versus Grade 8/2023 Summary 2018 Grade 7 Scaled Score Decile Cuts 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Observed All Students Grade 8 Scaled/Standardized Score and Achievement Level Cuts Female Students 2200 2400 2500 2700 Growth/Status Recovery Legend 2019 Grade 8 Scaled Score Decile Cuts 2018 to 2019 Mean Conditional Status 90 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 1 2023 Percentiles (Observed) 2019 to 2023 Growth Impact 2019 to 2023 Status Impact −0.38 −0.09 0.12 0.38 2019 and 2023 Mean Scaled Scores 70 Lagging On Par Leading 2019 Grade 8 Scaled Score 50 Calculations suppressed 40 −0.5 due to small group N (< 50) 10 Participation Total Status Growth 2300 2300 −2 2019: 41.1% Proficient 2019 6,227 6,227 (100.0%) 5,763 (92.5%) 2023: 29.3% Proficient 2023 4,963 4,708 (94.9%) 4,169 (84.0%) 2018 Grade 7 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 8 Median SGPs by Quintile 2019 & 2023 Grade 8 Mean Scaled (Standardized) Scores by Quintile On Par Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 53.0 54.0 53.0 56.0 49.0 53.0 2019 Grade 8 Students 2373 (0.09) 2475 (0.09) 2545 (0.09) 2614 (0.09) 2711 (0.09) 2555 (0.09) Q1 Q2 Q3 Q4 Q5 2023 Grade 8 Students 57.0 56.0 48.0 43.0 44.5 50.0 2023 Grade 8 Students 2355 (−0.17) 2452 (−0.17) 2519 (−0.17) 2589 (−0.17) 2701 (−0.17) 2522 (−0.17) On Par On Par On Par Lagging On Par Academic Impact (Observed) 4.0 2.0 −5.0 −13.0 −4.5 −3.0 Change (Observed) −18 (−0.15) −22 (−0.18) −27 (−0.22) −25 (−0.21) −10 (−0.08) −33 (−0.27) Count Observed 2019 995 1,150 1,218 1,176 1,224 5,763 Count Observed 2019 1,017 1,212 1,288 1,348 1,362 6,227 Count Observed 2023 831 833 887 849 768 4,168 Count Observed 2023 898 996 982 951 881 4,708 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 2023 to 2019 Grade 8 Status Change (Observed) Overall 2023 to 2019 Conditional Status Difference (Observed) 0.48: Correlation SGP and Status Percentile Change Lagging Conditional Status/Growth Change 25 0.97: Correlation SGP and Conditional Status Difference 25 0.20 15 15 Lagging Lagging Lagging Lagging Lagging 2019 to 2023 2018 Grade 7 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA Mathematics Grade 8 Hawaii 2019 to 2023 COVID⌧19 Academic Recovery Male Students Growth: Grade 7/2018 to 8/2019 versus Grade 7/2022 to 8/2023 Status: Grade 8/2019 versus Grade 8/2023 Summary 2018 Grade 7 Scaled Score Decile Cuts 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Observed All Students Grade 8 Scaled/Standardized Score and Achievement Level Cuts Male Students 2200 2400 2500 2700 Growth/Status Recovery Legend 2019 Grade 8 Scaled Score Decile Cuts 2018 to 2019 Mean Conditional Status 90 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 1 2023 Percentiles (Observed) 2019 to 2023 Growth Impact 2019 to 2023 Status Impact −0.38 −0.09 0.12 0.38 2019 and 2023 Mean Scaled Scores 70 Lagging On Par Leading 2019 Grade 8 Scaled Score 50 Calculations suppressed 40 −0.5 due to small group N (< 50) 10 Participation Total Status Growth 2300 2300 −2 2019: 34.7% Proficient 2019 6,694 6,694 (100.0%) 6,164 (92.1%) 2023: 31.7% Proficient 2023 5,425 5,169 (95.3%) 4,598 (84.8%) 2018 Grade 7 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 8 Median SGPs by Quintile 2019 & 2023 Grade 8 Mean Scaled (Standardized) Scores by Quintile On Par Q1 Q2 Q3 Q4 Q5Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 48.0 47.0 45.0 46.0 50.0 47.0 2019 Grade 8 Students 2365 (−0.09) 2474 (−0.09) 2544 (−0.09) 2613 (−0.09) 2717 (−0.09) 2532 (−0.09) Q1 Q2 Q3 Q4 Q5 2023 Grade 8 Students 53.0 51.0 45.0 44.0 44.0 48.0 2023 Grade 8 Students 2353 (−0.14) 2453 (−0.14) 2519 (−0.14) 2589 (−0.14) 2705 (−0.14) 2525 (−0.14) On Par On Par On Par On Par On Par Academic Impact (Observed) 5.0 4.0 0.0 −2.0 −6.0 1.0 Change (Observed) −13 (−0.10) −21 (−0.17) −25 (−0.20) −24 (−0.20) −12 (−0.10) −7 (−0.06) Count Observed 2019 1,363 1,220 1,171 1,245 1,1656,164 Count Observed 2019 1,568 1,368 1,308 1,228 1,222 6,694 Count Observed 2023 898 929 873 907 991 4,598 Count Observed 2023 1,077 981 988 1,029 1,094 5,169 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 2023 to 2019 Grade 8 Status Change (Observed) Overall 2023 to 2019 Conditional Status Difference (Observed) 0.13: Correlation SGP and Status Percentile Change On Par Conditional Status/Growth Change 25 0.96: Correlation SGP and Conditional Status Difference 25 0.20 15 15 On Par On Par On Par On Par On Par 2019 to 2023 2018 Grade 7 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA Mathematics Grade 8 Hawaii 2019 to 2021 COVID⌧19 Academic Impact Female Students Growth: Grade 6 2019 to Grade 8 2021 Status: Grade 8 2019 to Grade 8 2021 Summary 2019 Grade 6 Scaled Score Decile Cuts Observed + Imputed All Students 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Academic Impact: Overall and by Quintile Grade 8 Scaled/Standardized Score and Achievement Level Cuts Female Students Observed 2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed) Overall 2021 Grade 8 Scaled Score Decile Cuts 2019 to 2021 Mean Conditional Status (Observed) 1 2021 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large 2019 to 2021 Growth Impact 2019 to 2021 Status Impact 2019 and 2021 Mean Scaled Scores Moderate Large Severe Severe Severe 2021 Grade 8 Scaled Score Participation 30 Total Status Growth 2300 2300 −2 2019: 41.1% Proficient 2021: 25.5% Proficient Growth/Status Change Legend 2019 Grade 6 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 8 Median SGPs by Quintile 2019 & 2021 Grade 8 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 53.0 54.0 55.0 55.0 51.0 53.0 2019 Grade 8 Students 2373 (0.09) 2475 (0.09) 2545 (0.09) 2614 (0.09) 2711 (0.09) 2555 (0.09) 2021 Grade 8 Students 50.0 34.0 26.0 20.0 19.0 27.0 2021 Grade 8 Students 2351 (−0.22) 2448 (−0.22) 2503 (−0.22) 2560 (−0.22) 2664 (−0.22) 2516 (−0.22) Calculations suppressed Academic Impact (Observed) −3.0 −20.0 −29.0 −35.0 −32.0 −26.0 Change (Observed) −22 (−0.18) −27 (−0.22) −42 (−0.34) −54 (−0.44) −48 (−0.39) −39 (−0.32) Academic Impact (+ Imputed) −11.0 −17.9 −25.0 −31.0 −31.1 −23.3 Change (+ Imputed) −19 (0.19) −27 (0.23) −42 (0.16) −54 (0.13) −50 (0.26) −45 (−0.37) due to small group N (< 50) Count Observed 2019 877 1,076 1,125 1,169 1,213 5,460 Count Observed 2019 1,017 1,212 1,288 1,348 1,362 6,227 Count Observed 2021 684 877 951 1,017 1,034 4,563 Count Observed 2021 860 971 1,025 1,049 1,168 5,073 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 2021 to 2019 Grade 8 Status Change (Observed) 30 30 0.25 2021 to 2019 Grade 8 Status Change (+ Imputed) 2021 to 2019 SGP Difference (+ Imputed) 0.94: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 1.00: Correlation SGP and Conditional Status Difference 2019 to 2021 2019 Grade 6 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA Mathematics Grade 8 Hawaii 2019 to 2021 COVID⌧19 Academic Impact Male Students Growth: Grade 6 2019 to Grade 8 2021 Status: Grade 8 2019 to Grade 8 2021 Summary 2019 Grade 6 Scaled Score Decile Cuts Observed + Imputed All Students 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Academic Impact: Overall and by Quintile Grade 8 Scaled/Standardized Score and Achievement Level Cuts Male Students Observed 2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed) Overall 2021 Grade 8 Scaled Score Decile Cuts 2019 to 2021 Mean Conditional Status (Observed) 1 2021 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large 2019 to 2021 Growth Impact 2019 to 2021 Status Impact 2019 and 2021 Mean Scaled Scores Moderate Moderate Moderate Large Severe 2021 Grade 8 Scaled Score Participation 30 Total Status Growth 2300 2300 −2 2019: 34.7% Proficient 2021: 24.1% Proficient Growth/Status Change Legend 2019 Grade 6 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 8 Median SGPs by Quintile 2019 & 2021 Grade 8 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 48.0 47.0 45.0 46.5 49.0 47.0 2019 Grade 8 Students 2365 (−0.09) 2474 (−0.09) 2544 (−0.09) 2613 (−0.09) 2717 (−0.09) 2532 (−0.09) 2021 Grade 8 Students 46.0 37.0 28.0 19.0 16.0 28.0 2021 Grade 8 Students 2344 (−0.29) 2448 (−0.29) 2503 (−0.29) 2561 (−0.29) 2669 (−0.29) 2507 (−0.29) Calculations suppressed Academic Impact (Observed) −2.0 −10.0 −17.0 −27.5 −33.0 −19.0 Change (Observed) −20 (−0.17) −25 (−0.21) −41 (−0.33) −52 (−0.42) −48 (−0.39) −25 (−0.20) Academic Impact (+ Imputed) −6.4 −10.3 −14.0 −24.4 −31.0 −17.0 Change (+ Imputed) −17 (0.21) −26 (0.24) −41 (0.17) −53 (0.13) −50 (0.27) −31 (−0.25) due to small group N (< 50) Count Observed 2019 1,319 1,168 1,140 1,100 1,096 5,823 Count Observed 2019 1,556 1,380 1,308 1,228 1,222 6,694 Count Observed 2021 1,049 991 991 953 1,078 5,062 Count Observed 2021 1,165 1,089 1,046 1,129 1,215 5,644 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 2021 to 2019 Grade 8 Status Change (Observed) 30 30 0.25 2021 to 2019 Grade 8 Status Change (+ Imputed) 2021 to 2019 SGP Difference (+ Imputed) 0.89: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 1.00: Correlation SGP and Conditional Status Difference 2019 to 2021 2019 Grade 6 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA Mathematics Grade 8 Hawaii 2019 to 2022 COVID⌧19 Academic Recovery Female Students Growth: Grade 7/2018 to 8/2019 versus Grade 7/2021 to 8/2022 Status: Grade 8/2019 versus Grade 8/2022 Summary 2018 Grade 7 Scaled Score Decile Cuts All Students 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Observed Grade 8 Scaled/Standardized Score and Achievement Level Cuts Female Students 2200 2400 2500 2700 Growth/Status Recovery Legend 2019 Grade 8 Scaled Score Decile Cuts 2018 to 2019 Mean Conditional Status 90 2019 Percentiles (Observed) −30 −7 10 30 2021 to 2022 Mean Conditional Status (Observed) 1 2022 Percentiles (Observed) 2019 to 2022 Growth Impact 2019 to 2022 Status Impact −0.38 −0.09 0.12 0.38 2019 and 2022 Mean Scaled Scores 70 Lagging On Par Leading 2019 Grade 8 Scaled Score Calculations suppressed −0.5 due to small group N (< 50) 10 Participation Total Status Growth 2300 2300 −2 2019: 41.1% Proficient 2019 6,227 6,227 (100.0%) 5,763 (92.5%) 2022: 30.9% Proficient 2022 6,329 6,066 (95.8%) 4,831 (76.3%) 2018 Grade 7 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2022 Grade 8 Median SGPs by Quintile 2019 & 2022 Grade 8 Mean Scaled (Standardized) Scores by Quintile On Par Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 53.5 54.0 53.0 56.0 49.0 53.0 2019 Grade 8 Students 2374 (0.09) 2475 (0.09) 2545 (0.09) 2614 (0.09) 2711 (0.09) 2555 (0.09) Q1 Q2 Q3 Q4 Q5 2022 Grade 8 Students 61.0 56.0 46.0 48.0 54.0 53.0 2022 Grade 8 Students 2356 (−0.13) 2460 (−0.13) 2523 (−0.13) 2588 (−0.13) 2694 (−0.13) 2527 (−0.13) On Par On Par Lagging Lagging On Par Academic Impact (Observed) 7.5 2.0 −7.0 −8.0 5.0 0.0 Change (Observed) −18 (−0.15) −15 (−0.12) −22 (−0.18) −26 (−0.21) −18 (−0.14) −28 (−0.22) Count Observed 2019 994 1,150 1,218 1,176 1,224 5,762 Count Observed 2019 1,031 1,198 1,288 1,348 1,362 6,227 Count Observed 2022 897 1,002 1,007 984 941 4,831 Count Observed 2022 1,118 1,248 1,216 1,282 1,202 6,066 Status Recovery: Overall and by Quintile 2022 to 2019 Scaled/Standardized Score Difference 2022 to 2019 SGP Difference (Observed) 2022 to 2019 Grade 8 Status Change (Observed) Overall 2022 to 2019 Conditional Status Difference (Observed) 0.92: Correlation SGP and Status Percentile Change Lagging Conditional Status/Growth Change 25 0.82: Correlation SGP and Conditional Status Difference 25 0.20 15 15 Lagging Lagging Lagging Lagging Lagging 2019 to 2022 2018 Grade 7 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA Mathematics Grade 8 Hawaii 2019 to 2022 COVID⌧19 Academic Recovery Male Students Growth: Grade 7/2018 to 8/2019 versus Grade 7/2021 to 8/2022 Status: Grade 8/2019 versus Grade 8/2022 Summary 2018 Grade 7 Scaled Score Decile Cuts All Students 10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score Observed Grade 8 Scaled/Standardized Score and Achievement Level Cuts Male Students 2200 2400 2500 2700 Growth/Status Recovery Legend 2019 Grade 8 Scaled Score Decile Cuts 2018 to 2019 Mean Conditional Status 90 2019 Percentiles (Observed) −30 −7 10 30 2021 to 2022 Mean Conditional Status (Observed) 1 2022 Percentiles (Observed) 2019 to 2022 Growth Impact 2019 to 2022 Status Impact −0.38 −0.09 0.12 0.38 2019 and 2022 Mean Scaled Scores 70 Lagging On Par Leading 2019 Grade 8 Scaled Score Calculations suppressed −0.5 due to small group N (< 50) 10 Participation Total Status Growth 2300 2300 −2 2019: 34.7% Proficient 2019 6,694 6,694 (100.0%) 6,164 (92.1%) 2022: 30.7% Proficient 2022 6,644 6,376 (96.0%) 5,057 (76.1%) 2018 Grade 7 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2022 Grade 8 Median SGPs by Quintile 2019 & 2022 Grade 8 Mean Scaled (Standardized) Scores by Quintile On Par Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 48.0 47.0 45.0 46.0 50.0 47.0 2019 Grade 8 Students 2365 (−0.09) 2474 (−0.09) 2544 (−0.09) 2613 (−0.09) 2717 (−0.09) 2532 (−0.09) Q1 Q2 Q3 Q4 Q5 2022 Grade 8 Students 54.0 50.5 45.0 45.0 47.5 48.0 2022 Grade 8 Students 2351 (−0.17) 2460 (−0.17) 2524 (−0.17) 2590 (−0.17) 2700 (−0.17) 2522 (−0.17) On Par On Par On Par On Par On Par Academic Impact (Observed) 6.0 3.5 0.0 −1.0 −2.5 1.0 Change (Observed) −15 (−0.12) −14 (−0.12) −20 (−0.17) −23 (−0.19) −17 (−0.13) −10 (−0.08) Count Observed 2019 1,363 1,220 1,171 1,245 1,164 6,163 Count Observed 2019 1,568 1,368 1,308 1,228 1,222 6,694 Count Observed 2022 1,013 964 986 1,016 1,078 5,057 Count Observed 2022 1,378 1,241 1,268 1,196 1,293 6,376 Status Recovery: Overall and by Quintile 2022 to 2019 Scaled/Standardized Score Difference 2022 to 2019 SGP Difference (Observed) 2022 to 2019 Grade 8 Status Change (Observed) Overall 2022 to 2019 Conditional Status Difference (Observed) 0.70: Correlation SGP and Status Percentile Change On Par Conditional Status/Growth Change 25 0.91: Correlation SGP and Conditional Status Difference 25 0.20 15 15 On Par On Par Lagging Lagging On Par 2019 to 2022 2018 Grade 7 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)

SBA ELA and Mathematics Hawaii 2019 to 2022 COVID⌧19 Academic System Recovery Grade & Content Area Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students On Par On Par On Par On Par On Par On Par Lagging Leading Leading On Par On Par On Par Economically Disadvantaged Lagging On Par On Par On Par On Par On Par Lagging Leading On Par On Par On Par On Par Non−Economically Disadvantaged On Par Leading On Par On Par On Par On Par On Par Leading Leading On Par On Par On Par Special Education On Par On Par Lagging Lagging On Par Lagging Lagging On Par On Par On Par On Par On Par Asian On Par Leading On Par On Par On Par On Par On Par Leading Leading On Par Leading On Par White On Par On Par On Par On Par On Par On Par Lagging Leading On Par On Par On Par On Par Hispanic Lagging On Par On Par On Par On Par On Par Lagging Leading Leading On Par On Par On Par Native Hawaiian Lagging Leading On Par On Par On Par On Par Lagging Leading On Par On Par On Par On Par ELL On Par On Par Lagging On Par On Par Lagging On Par Leading On Par On Par On Par Lagging Female On Par Leading On Par On Par On Par On Par Lagging Leading Leading On Par On Par On Par Male On Par On Par On Par On Par On Par On Par Lagging Leading Leading On Par On Par On Par

Recovery Year 2 • Analysis of 2023 SBA data indicates two things: • Rates of learning, on average, are slightly exceeding or at pre-pandemic rates of growth. That is, the system has recovered, and then some. • Student attainment, on average, is still lagging behind because rates of learning have not accelerated enough to catch students up. That is, students have not recovered. • In both ELA and mathematics, learning acceleration from the lows of the pandemic would be considered, on average, to be system stabilization. • System stabilization (a return to normal rates of learning) will be insufficient to catch students up. • Averages can obscure important heterogeneity:

Questions

About this copy

COVID-19 Impact & Recovery in Hawaii — Hawaii Department of Education, November 8, 2023.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.