COVID-19 Impact & Recovery
Background, Results, and Next Steps
Damian Betebenner & Adam VanIwaarden
Center for Assessment
January 17th, 2023
Background
Introductory Concepts and Terminology
COVID-19 Pandemic
• The COVID-19 Pandemic led to, arguably, the largest educational disruption in
the history of the United States.
• Nearly 3 years into the pandemic and the impacts of the pandemic are still being
felt.
• The pandemic impacted students academically, emotionally, physically, and
financially.
• In this overview, we address three things:
• Academic Impact: What is it and what happened to students in New
Hampshire?
• Academic Recovery: What is it and what is happening to students in New
Hampshire currently?
• Now What?: How can impact and recovery data be leveraged to help
students going forward? Evidence-based investigations of what is working.
What is Academic Impact?
What is Academic Recovery?
• The pandemic and all the ensuing disruptions functioned/functions
as an academic headwind, impeding (in general) the academic
progress of students.
• Headwinds impede progress in two ways:
▪ They slow one’s rate of progress (speedometer)
▪ And by slowing one’s rate of progress they lead to less distance being travelled
(odometer).
• In education odometer is to attainment as speedometer is to growth.
• Academic impact is analogous to a headwind, decreasing the rate of
student learning (deceleration) which, in turn leads to lower
attainment
What is Academic Impact?
What is Academic Recovery?
• Academic recovery is analogous to an academic tailwind,
increasing the rate of student learning (acceleration) which, in
turn leads to higher attainment.
• During Fall 2021 to present, under the presumption that student
education got back to normal, the hope was that rates of learning
stabilized to pre-pandemic rates or, even better, increased to rates
exceeding pre-pandemic rates allowing students to catch-up.
• Examination of academic impact/recovery are opposite sides of
the same coin and utilize the same analytics.
Impact & Recovery
• We investigate impact/recovery in two ways:
• Change in same grade/same cohort attainment: We examine equi-percentile
change in attainment. Impact: 2019 to 2021, Recovery 2021 to 2022
• Change in student growth: We examine change in baseline (pre-pandemic)
student growth (SGPs). Impact: 2019 to 2021 (using 2017 to 2019 growth
norms), Recovery: 2021 to 2022 (using 2018 to 2019 growth norms).
Assessment Data
• We investigated the impacts in achievement and growth of the pandemic across
two time frames:
• Impact: Spring 2020 through Summer 2021
• Recovery: Fall 2021 through Summer 2022
• To investigate impact we utilized two primary sources of data
• State summative assessment data (NH-SAS)
• Impact: NH-SAS Spring 2019 to Spring 2021 (NH-SAS testing cancelled in
• Recovery: NH-SAS Spring 2021 to Spring 2022
• State English Language Proficiency Data (ACCESS)
• Impact: ACCESS Winter 2020 to Winter 2021
• Recovery: ACCESS Winter 2021 to Winter 2022
Impact
From Spring 2020 to Summer 2021
NH-SAS: Spring 2019 and Spring 2021 Assessments
ACCESS: Winter 2020 and Winter 2021 Assessments
SAS Mathematics Grade 7
New Hampshire 2019 to 2021 COVID⌧19 Academic Impact All Students
Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary
2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score
Academic Impact: Overall and by Quintile
Grade 7 Scaled/Standardized Score and Achievement Level Cuts
2.5 Overall
Observed + Imputed
2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (+ Imputed)
2021 Percentiles (+ Imputed) Large
2019 to 2021 Growth Impact 2019 to 2021 Status Impact
2021 Grade 7 Scaled Score Decile Cuts
Observed
2019 and 2021 Mean Scaled Scores
Large Large Large Severe Severe
2021 Grade 7 Scaled Score
70 70 550 Participation
50 −0.29 Total Status Growth
2019: 47.1% Proficient
2021: 34.1% Proficient
−3 Growth/Status Change Legend
2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Percentile Ranks
Modest
Severe Large Moderate to Improvement
2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled (Standardized) Scores by Quintile None
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 7 Students 50.0 50.0 49.0 50.0 51.0 50.0 2019 Grade 7 Students 457 (−1.46) 517 (−0.45) 547 ( 0.05) 575 ( 0.52) 622 ( 1.32) 544 (0.00)
2021 Grade 7 Students 33.0 30.0 27.0 24.0 25.0 28.0 2021 Grade 7 Students 440 (−1.76) 501 (−0.73) 530 (−0.23) 558 ( 0.24) 605 ( 1.03) 527 (−0.29) Calculations suppressed
Academic Impact (Observed) −17.0 −20.0 −22.0 −26.0 −26.0 −22.0 Change (Observed) −18 (−0.30) −16 (−0.28) −17 (−0.29) −17 (−0.28) −17 (−0.29) −17 (−0.29)
Academic Impact (+ Imputed) −18.0 −19.0 −22.0 −25.0 −26.0 −21.8 Change (+ Imputed) −16 ( 0.21) −16 (−0.14) −17 (−0.32) −17 (−0.48) −17 (−0.76) −20 (−0.35)
due to small group N (< 50)
Count Observed 2019 2,155 2,185 2,215 2,208 2,178 10,941 Count Observed 2019 2,422 2,435 2,494 2,469 2,447 12,267
Count Observed 2021 1,518 1,677 1,713 1,772 1,818 8,498 Count Observed 2021 2,061 1,998 2,078 2,035 2,052 10,224
2021 to 2019 Scaled/Standardized Score Difference
2021 to 2019 SGP Difference (Observed) 0.55 2021 to 2019 Grade 7 Status Change (Observed)
30 2021 to 2019 SGP Difference (+ Imputed) 30 0.50 2021 to 2019 Grade 7 Status Change (+ Imputed)
0.31: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
2021 to 2019 Conditional Status Difference (Observed) 0.45
0.68: Correlation SGP and Conditional Status Difference 0.40
2019 to 2021
2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Ranks and Scaled Scores (2019 Norm)
Summary
We summarize the academic impact both
overall and by achievement quintile
For Grade 7 mathematics, the overall
academic impact for students was large (z
decrease = -0.29 and median SGP decrease =
Academic impact was not uniform across
students. For this grade/content area higher
achieving students were more severely
impacted than lower achieving student.
SAS ELA and Mathematics
New Hampshire 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Economically Disadvantaged Students Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Moderate
Non−Economically Disadvantaged Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Special Education Students Large Moderate Modest to None Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate
African American Students Large Moderate Modest to None Modest to None Large Moderate Large Moderate Severe Large Moderate Moderate
Asian Students Modest to None Moderate Modest to None Moderate Moderate Modest to None Large Large Large Large Large Large
White Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Hispanic Students Large Large Moderate Moderate Large Moderate Large Severe Severe Large Large Large
ELL Students Moderate Moderate Moderate Moderate Large Moderate Large Large Large Large Large Large
Female Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Male Students Large Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
WIDA ACCESS WIDA⌧ACCESS Composite
WIDA NH 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
All Students Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
All Students
Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
New Hampshire Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
State A Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate
State B Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large
State C Modest to None Modest to None Moderate Moderate Moderate Modest to None Moderate Moderate Moderate Large Large Moderate
State D Moderate Moderate Large Large Large Large Moderate Moderate Large Large Large Large
State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large
State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe
State G Moderate Moderate Moderate Moderate Large Moderate
State H Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate
State I Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large
State J Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe
All Students
Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None
State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None
State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None
State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None
State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None
State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement
State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
Recovery
From Fall 2021 to Summer 2022
NH-SAS: Spring 2021 and Spring 2022 Assessments
ACCESS: Winter 2021 and Winter 2022 Assessments
Recovery Phase
• During the 2021-2022 academic year, school returned to normal for most
students.
• The hope was that a return to normal would lead recovery of the academic losses
incurred by students.
• Because learning slowed during the initial phase of the pandemic, to make up for
those losses, learning must accelerate to rates above normal in order to catch-up
for those loses.
• Data from two sources was utilized to examine whether recovery is occurring:
• NH-SAS data from 2021 and 2022
• WIDA-ACCESS data from 2021 and 2022
Four Academic
Recovery
Scenarios with NH-
SAS: V-, U-, L-, and
No-recovery
SAS Reading and Mathematics
New Hampshire 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.38 −0.09 0.12 0.38 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
Reading Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization
Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
African American Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Recovery
Asian Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Stabilization Stabilization
White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization
Limited English Proficient Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization
Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Male Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Overall “Recovery”
• In both ELA and mathematics, rates of learning appear acceleration was at least as
much to be considered stabilization and in many cases could be considered as
genuine recovery.
• As such, one could say that the “system” has recovered in that it has returned to
rates of learning that it produced pre-pandemic.
• It is critical to note that system level recovery is not the same as student recovery.
• Because student learning was depressed between 2019 and 2021. Student
recovery will require rates of learning well in excess of what was typical pre-
pandemic.
• Results from 2023 data will be critical to assess whether these exemplary rates of
learning (beyond what the state experience pre-COVID) are being maintained.
Academic Impact/Recovery Across 8 States
Impact & Recovery Recovery
Mathematics
ELA
New Hampshire showed large 10
NH5
impacts and recovery 5
NH6
NH4
NH8
NH7 NH3
NH6
NH5
Stabilization
NH4 NH8
New Hampshire’s acadtemic 0
NH3
NH7
results in 2021-2022 represent
Academic Recovery
stabilization bordering on −10
Deterioration
recovery −15
A critical “test” of this recovery
will be the 2023 results. If rates of
learning in 2023 extend those of
2022, then New Hampshire will
have come out of the pandemic
better than before it started −40 −35 −30 −25 −20 −15 −10 −5 0 5 10 15 20
Large Impact Modest Impact Improvement
Academic Impact
About this copy
COVID-19 Impact & Recovery — New Hampshire Department of Education, January 12, 2023.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.