COVID-19 Impact & Recovery in Hawaii
2024 Update
Damian Betebenner & Adam VanIwaarden
Center for Assessment
December 5th, 2024
Background
Introductory Concepts and Terminology
Academic Recovery in 2024
• The availability of 2024 SBA data allows Hawaii to update
COVID-19 recovery analyses.
• Spring 2024 marks the third year since the pandemic began
and this is the third report:
• Summer 2021: Academic Impact
• Summer 2022: Academic Recovery Year 1
• Summer 2023: Academic Recovery Year 2
• Summer 2024: Academic Recovery Year 3
Two Types of Recovery in 2024: System & Student
• We elaborate on the results from the previous recovery analyses in two
ways:
• System level recovery: Is the education system supporting rates of
learning that are comparable to what it was prior to the pandemic?
• Student level recovery: Are students achieving at levels that are
comparable to what they were prior to the pandemic?
• Previously, we focused mostly on system-level recovery: examining rates
of learning in 2021 and 2022 compared to those prior to the pandemic.
• We continue examining system level analyses but also focus on student-
level recovery.
Two Types of Recovery: System & Student
• System-level recovery
• Comparison of 2019 and 2024 same-grade student growth: We
examine the change in baseline student growth (SGPs) for 2019
students and 2024 students utilizing coefficient matrices derived
from pre-pandemic cohorts of students.
• Student-level recovery
• Comparison of 2019 to 2024 same-grade achievement: We
examine the equi-percentile change in achievement for 2019
students and 2024 students at same-grade
Student-level recovery
• It is critical to understand that recovery of the system does not imply student level
recovery
• Because students’ rates of learning slowed so dramatically during the pandemic, rates of
learning necessary to catch students up to where they would have been would have to be
equally dramatic (in the positive direction)
• As of 2024, 4 cohorts of students have exited the system (via graduation) achieving at
levels, on average, below where they would have been had the pandemic not occurred.
• The following figures illustrate the extent to which student attainment is behind 2019
attainment levels.
• Comparison to 2019 attainment levels is a “low bar” for comparison:
• Attainment levels in 2024, had the pandemic not occurred, would almost certainly
have been higher than 2019 levels (based on “typical” rates of yearly improvement)
System Recovery: Cohort-Based Comparison (SBA)
Compare observed versus expected attainment for a cohort of students
Grade 3
Grade 4
No Testing due to
COVID-19
Grade 5
Pandemic
Grade 6
Grade 7
Grade 8
Student Recovery: Cross-sectional Comparison (SBA)
Compare observed versus expected attainment for a cohort of students
Grade 3
Grade 4
No Testing due to
COVID-19
Grade 5
Pandemic
Grade 6
Grade 7
Grade 8
Why Baseline SGPs?
• There was a question of why one needs to look at baseline referenced SGPs
instead of, for example, scale score gains. There are several reasons.
• SGPs are the growth metric that the state of Hawaii has been using for more
than a decade.
• SGPs are readily interpretable probabilistically whereas gain scores are not.
• Quantile regression (on which SGPs are based) is invariant to mono-tone
transformations of scale (equivariant to mono-tone transformations of scale).
OLS regression, for example, is not. See Koenker (2005, p. 38)
• Gain scores (which can be calculated since SBA is vertically scaled) are
dependent upon: Content area, grade, prior level of achievement.
• Gain scores were largely discredited as an effective means of communicating
change. For example, no state uses them as a growth model.
Status based impact versus growth?
• There was another question about the equivalence of status change and
growth change.
• Previously, I justified this change heuristically by noting that growth
change (deceleration from 2021) had to lead to status change.
• That is, decreases in learning (as quantified by growth) necessarily lead
to decreases in achievement.
• Also, this is empirically verified based upon two years of data plotted for
several states across grade and content areas show a correlation of 0.91.
• This relationship isn’t a function of content area or year but a
relationship between status change and conditional status (i.e., growth)
change.
Status based impact versus growth?
Recovery Year 1
From Fall 2021 to Summer 2022
SBA: Spring 2021 and Spring 2022 Assessments
Recovery Phase Year 1
• Student growth (baseline referenced to pre-pandemic rates of learning) was
examined by content area and grade as well as for the major student
demographic subgroups.
• Achievement (Student Recovery) and Growth (System Recovery) were
characterized in three ways:
• Lagging: Behind pre-pandemic levels (attainment or growth)
• On Par: About the same as pre-pandemic levels (attainment or growth)
• Leading: Ahead of pre-pandemic levels (attainment or growth)
• Because learning slowed during the initial phase of the pandemic (2020 to
2021), to make up for those losses, learning must accelerate to rates above
normal (i.e., leading)
Recovery Year 1
• Analysis of 2022 SBA data indicated several positive trends in Hawai’i to be happy
about.
• In both ELA and mathematics, learning acceleration was at least as much to be
considered stabilization and in many cases could be considered as genuine
recovery.
• Compared to a dozen other states that the Center conducts these analyses,
Hawai’i comes out as the best in terms of pandemic recovery.
• Despite a robust recovery, students remain slightly behind 2019 levels in ELA and
further behind in mathematics.
Recovery Year 1: Qualitative Investigation
• HIDOE commissioned a study, based upon an initial ROI investigation based upon money spent during the pandemic
to help students.
• We identified 15 schools to investigate (via 90 minute zoom interviews) more thoroughly based upon three patterns:
• High impact/high recovery
• High impact/little to no recovery
• Low impact/high recovery
• My colleague Jeri Thompson and I investigated this recovery, focusing on schools demonstrating large rates of
recovery that continued through 2023.
• As was reported previously, Hawaii, overall experienced a robust recovery in 2022 in rates of student learning —
much more so than other states.
• Our interviews failed to identify a program or intervention responsible for what we observed at the 15 schools we
investigated.
• More on year 2.
Recovery Year 2
From Fall 2022 to Summer 2023
SBA: Spring 2022 and Spring 2023 Assessments
Recovery Phase Year 2
• Student growth (baseline referenced to pre-pandemic rates of learning) was
examined by content area and grade as well as for the major student
demographic subgroups.
• Achievement (Student Recovery) and Growth (System Recovery) were
characterized in three ways:
• Lagging: Behind pre-pandemic levels (attainment or growth)
• On Par: About the same as pre-pandemic levels (attainment or growth)
• Leading: Ahead of pre-pandemic levels (attainment or growth)
• Because learning slowed during the initial phase of the pandemic (2020 to
2021), to make up for those losses, learning must accelerate to rates above
normal (i.e., leading)
Recovery Year 2
• Analysis of 2023 SBA data indicates two things:
• Rates of learning, on average, are slightly exceeding or at pre-pandemic rates of
growth. That is, the system has recovered, and then some.
• Student attainment, on average, is still lagging behind because rates of learning have
not accelerated enough to catch students up. That is, students have not recovered.
• In both ELA and mathematics, learning acceleration from the lows of the pandemic would
be considered, on average, to be system stabilization.
• System stabilization (a return to normal rates of learning) will be insufficient to catch
students up.
• Averages can obscure important heterogeneity:
Recovery Year 2: Qualitative Investigation
• Upon reflection of our initial findings, we hypothesize that the robust recovery was driven by cultural factors more so than programmatic
factors.
• To investigate further, we identified 5 schools that continued to experience superior rates of growth to those demonstrated prior to the onset
of the pandemic. 4 of those schools agreed to a full day meeting to discuss impact and recovery due to the pandemic.
• Kalihi-waena Elementary
• Mililani ‘Ike Elementary
• Waipahu Elementary
• Mililani Middle
• Our hypothesis is that these schools thrived due to efforts to maintain strong relationships during the COVID-19 crisis. Specifically, we believe
Hawaii’s culture of ‘Ohana led to:
• strong relational networks among students, teachers, parents, and the broader community that supported emotional and academic
resilience.enhanced collaboration between schools and families, fostering a sense of mutual responsibility for student success.
• a focus on belonging and cultural identity, which helped students navigate challenges and stay connected to their educational journey.
• a shared community ethos where collective well-being was prioritized, enabling targeted and empathetic support for struggling students
and families.
• enduring trust and communication frameworks that were established during the pandemic and maintained in the recovery period.
Recovery Year 3
From Fall 2023 to Summer 2024
SBA: Spring 2023 and Spring 2024 Assessments
Recovery Phase Year 3
• Student growth (baseline referenced to pre-pandemic rates of learning) was
examined by content area and grade as well as for the major student
demographic subgroups.
• Achievement (Student Recovery) and Growth (System Recovery) were
characterized in three ways:
• Lagging: Behind pre-pandemic levels (attainment or growth)
• On Par: About the same as pre-pandemic levels (attainment or growth)
• Leading: Ahead of pre-pandemic levels (attainment or growth)
• Because learning slowed during the initial phase of the pandemic (2020 to
2021), to make up for those losses, learning must accelerate to rates above
normal (i.e., leading)
2023 Status Results
• Percent Proficient/Standardized Score (pooled standard deviation)/Count
for SBA:
ELA Mathematics
Grade
2023 SGP Results
• Mean/Median/Count for baseline referenced SGPs in ELA and
mathematics are as follows:
ELA Mathematics
Grade
Status Recovery
Black Line: 2019 (pre-covid) scaled-score
distribution.
Red line: 2023 scaled-score distribution.
The impact of the pandemic on student
achievement (decrease) is illustrated as the area
between the black line (bottom) and the red line
(top)
X-axis: Grade 8 percentile rank
The horizontal distance between the curves at
the proficiency cut level (horizontal dashed blue
line) illustrates the increase in percent proficient.
Decrease of 7.2% proficient and decrease of 0.16
standard deviation
Status Recovery
Red line: 2023 - 2019 scaled-score
distribution difference.
The recovery from the pandemic
(difference in the distributions) on
student achievement (increase) is
indicated by the red line.
X-axis: Grade 8 percentile rank
Average impact z = -0.16 standard units.
Larger impact for Q3 and Q4 students.
Growth Recovery
Black Line: (pre-covid) conditional scaled-score
distribution — Grade 8/2019 conditioned on
Grade 7/2018
Red line: conditional scaled-score distribution —
Grade 8/2023 conditioned on Grade 7/2022
The lag from the pandemic on student growth
(increase) is illustrated as the area between the
black line (top) and the red line (bottom)
X-axis: Grade 7 Scaled Score and achievement
level cuts.
Fixing a level on the x-axis, the vertical distance
between the curves is the average decrease in
scale score associated with the impact for a
student at that 8th-grade level.
Growth Recovery
Red line: SGP Difference Grade
8/2023 - Grade 8/2019 conditional
scaled score distribution
X-axis: Grade 7 Scaled Score and
associated deciles and achievement
level cuts
Median SGP 2019 Grade 8: 50
Median SGP 2023 Grade 8: 46
Higher achieving groups (above the
40th percentile) demonstrating lower
rates of learning than pre-COVID.
Summary
We summarize the academic recovery in
terms of Growth (System Level Recovery)
and Status (Student Level Recovery) both
overall and by achievement quintile
Growth is, overall, on par with the pre-covid
growth. For the higher quintiles, growth is
on-par bordering on lagging.
Status is, overall, lagging. That is, student
attainment has not returned to pre-
pandemic levels. This reaffirms the point
made two years ago: For students to recover,
learning rates will have to
Recovery Year 3
• Analysis of 2023 SBA data indicates two things:
• Rates of learning, on average, are returning to pre-pandemic levels. Elevated rates of
learning observed in 2022 and 2023 are moderating.
• Student attainment, on average, is still slightly behind pre-pandemic levels.
• In both ELA and mathematics, learning acceleration from the lows of the pandemic would
be considered, on average, to be system stabilization.
• System stabilization (a return to normal rates of learning) will be insufficient to catch
students up.
• Averages can obscure important heterogeneity: For example, between males and females
in mathematics
About this copy
COVID-19 Impact & Recovery in Hawaii — Hawaii Technical Advisory Committee, December 5, 2024.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.