COVID-19 2023 Impact & Recovery
Background, Results, and Next Steps
Damian Betebenner & Adam VanIwaarden
Center for Assessment
July 21st, 2023
Background
Introductory Concepts and Terminology
COVID-19 Pandemic
• The COVID-19 Pandemic led to, arguably, the largest educational disruption in
the history of the United States.
• Nearly 3 years into the pandemic and the impacts of the pandemic are still being
felt.
• The pandemic impacted students academically, emotionally, physically, and
financially.
• In this overview, we address three things:
• Academic Impact: What is it and what happened to students in New
Hampshire?
• Academic Recovery: What is it and what is happening to students in New
Hampshire currently?
• Now What?: How can impact and recovery data be leveraged to help
students going forward? Evidence-based investigations of what is working.
What is Academic Impact?
What is Academic Recovery?
• The pandemic and all the ensuing disruptions functioned/functions
as an academic headwind, impeding (in general) the academic
progress of students.
• Headwinds impede progress in two ways:
▪ They slow one’s rate of progress (speedometer)
▪ And by slowing one’s rate of progress they lead to less distance being travelled
(odometer).
• In education odometer is to attainment as speedometer is to growth.
• Academic impact is analogous to a headwind, decreasing the rate of
student learning (deceleration) which, in turn leads to lower
attainment
What is Academic Impact?
What is Academic Recovery?
• Academic recovery is analogous to an academic tailwind,
increasing the rate of student learning (acceleration) which, in
turn leads to higher attainment.
• During Fall 2021 to present, under the presumption that student
education got back to normal, the hope was that rates of learning
stabilized to pre-pandemic rates or, even better, increased to rates
exceeding pre-pandemic rates allowing students to catch-up.
• Examination of academic impact/recovery are opposite sides of
the same coin and utilize the same analytics.
Impact & Recovery
• We investigate impact/recovery in two ways:
• Change in same grade/same cohort attainment: We examine equi-percentile
change in attainment. Impact: 2019 to 2021, Recovery 2021 to 2022
• Change in student growth: We examine change in baseline (pre-pandemic)
student growth (SGPs). Impact: 2019 to 2021 (using 2017 to 2019 growth
norms), Recovery: 2021 to 2022 (using 2018 to 2019 growth norms).
Assessment Data
• We investigated the impacts in achievement and growth of the pandemic across
two time frames:
• Impact: Spring 2020 through Summer 2021
• Recovery: Fall 2021 through Summer 2022
• To investigate impact we utilized two primary sources of data
• State summative assessment data (NH-SAS)
• Impact: NH-SAS Spring 2019 to Spring 2021 (NH-SAS testing cancelled in
• Recovery: NH-SAS Spring 2021 to Spring 2022
• State English Language Proficiency Data (ACCESS)
• Impact: ACCESS Winter 2020 to Winter 2021
• Recovery: ACCESS Winter 2021 to Winter 2022
Impact
From Spring 2020 to Summer 2021
NH-SAS: Spring 2019 and Spring 2021 Assessments
ACCESS: Winter 2020 and Winter 2021 Assessments
SAS Mathematics Grade 7
New Hampshire 2019 to 2021 COVID⌧19 Academic Impact All Students
Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary
2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score
Academic Impact: Overall and by Quintile
Grade 7 Scaled/Standardized Score and Achievement Level Cuts
2.5 Overall
Observed + Imputed
2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (+ Imputed)
2021 Percentiles (+ Imputed) Large
2019 to 2021 Growth Impact 2019 to 2021 Status Impact
2021 Grade 7 Scaled Score Decile Cuts
Observed
2019 and 2021 Mean Scaled Scores
Large Large Large Severe Severe
2021 Grade 7 Scaled Score
70 70 550 Participation
50 −0.29 Total Status Growth
2019: 47.1% Proficient
2021: 34.1% Proficient
−3 Growth/Status Change Legend
2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Percentile Ranks
Modest
Severe Large Moderate to Improvement
2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled (Standardized) Scores by Quintile None
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 7 Students 50.0 50.0 49.0 50.0 51.0 50.0 2019 Grade 7 Students 457 (−1.46) 517 (−0.45) 547 ( 0.05) 575 ( 0.52) 622 ( 1.32) 544 (0.00)
2021 Grade 7 Students 33.0 30.0 27.0 24.0 25.0 28.0 2021 Grade 7 Students 440 (−1.76) 501 (−0.73) 530 (−0.23) 558 ( 0.24) 605 ( 1.03) 527 (−0.29) Calculations suppressed
Academic Impact (Observed) −17.0 −20.0 −22.0 −26.0 −26.0 −22.0 Change (Observed) −18 (−0.30) −16 (−0.28) −17 (−0.29) −17 (−0.28) −17 (−0.29) −17 (−0.29)
Academic Impact (+ Imputed) −18.0 −19.0 −22.0 −25.0 −26.0 −21.8 Change (+ Imputed) −16 ( 0.21) −16 (−0.14) −17 (−0.32) −17 (−0.48) −17 (−0.76) −20 (−0.35)
due to small group N (< 50)
Count Observed 2019 2,155 2,185 2,215 2,208 2,178 10,941 Count Observed 2019 2,422 2,435 2,494 2,469 2,447 12,267
Count Observed 2021 1,518 1,677 1,713 1,772 1,818 8,498 Count Observed 2021 2,061 1,998 2,078 2,035 2,052 10,224
2021 to 2019 Scaled/Standardized Score Difference
2021 to 2019 SGP Difference (Observed) 0.55 2021 to 2019 Grade 7 Status Change (Observed)
30 2021 to 2019 SGP Difference (+ Imputed) 30 0.50 2021 to 2019 Grade 7 Status Change (+ Imputed)
0.31: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
2021 to 2019 Conditional Status Difference (Observed) 0.45
0.68: Correlation SGP and Conditional Status Difference 0.40
2019 to 2021
2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Ranks and Scaled Scores (2019 Norm)
Summary
We summarize the academic impact both
overall and by achievement quintile
For Grade 7 mathematics, the overall
academic impact for students was large (z
decrease = -0.29 and median SGP decrease =
Academic impact was not uniform across
students. For this grade/content area higher
achieving students were more severely
impacted than lower achieving student.
SAS ELA and Mathematics
New Hampshire 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Economically Disadvantaged Students Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Moderate
Non−Economically Disadvantaged Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Special Education Students Large Moderate Modest to None Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate
African American Students Large Moderate Modest to None Modest to None Large Moderate Large Moderate Severe Large Moderate Moderate
Asian Students Modest to None Moderate Modest to None Moderate Moderate Modest to None Large Large Large Large Large Large
White Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Hispanic Students Large Large Moderate Moderate Large Moderate Large Severe Severe Large Large Large
ELL Students Moderate Moderate Moderate Moderate Large Moderate Large Large Large Large Large Large
Female Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
Male Students Large Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
WIDA ACCESS WIDA⌧ACCESS Composite
WIDA NH 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
All Students Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
All Students
Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
New Hampshire Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
State A Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate
State B Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large
State C Modest to None Modest to None Moderate Moderate Moderate Modest to None Moderate Moderate Moderate Large Large Moderate
State D Moderate Moderate Large Large Large Large Moderate Moderate Large Large Large Large
State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large
State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe
State G Moderate Moderate Moderate Moderate Large Moderate
State H Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate
State I Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large
State J Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe
All Students
Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None
State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None
State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None
State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None
State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None
State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement
State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
Recovery Years 1 and 2
Year 1: From Fall 2021 to Summer 2022, Year 2: From Fall 2022 to Summer 2023
NH-SAS: Spring 2021, Spring 2022 and Spring 2023 Assessments
ACCESS: Winter 2021, Winter 2022 and Winter 2023 Assessments
Recovery Phase
• During the 2021-2022 academic year, school returned to normal for most students.
• The 2022-2023 academic year saw continued that trend.
• The hope was that a return to normal would lead recovery of the academic losses
incurred by students.
• Because learning slowed during the initial phase of the pandemic, to make up for
those losses, learning must accelerate to rates above normal in order to catch-up
for those loses.
• Data from two sources was utilized to examine whether recovery is occurring:
• NH-SAS data from 2021, 2022 and 2023
• WIDA-ACCESS data from 2021, 2022 and 2023
Recovery Phase: Cautions in Interpretation
• Interpretation of “recovery” based upon outcomes from tests can be misleading.
• Since NCLB, interpretation of education system quality is often based upon attainment of students
(e.g., Percentage of proficient students)
• Because student attainment dropped precipitously in New Hampshire (and nationwide) and has
not yet recovered to pre-pandemic percentages, there is a common misunderstanding that the
education system continues to suffer from the pandemic.
• To provide the appropriate nuance to understand academic recovery we distinguish between:
• Student Recovery: Are students back to levels of attainment they would have been had the
pandemic not occurred?
• Education System Recovery: Is the system back to supporting rates of learning observed
before the pandemic?
• NOTE: student learning (i.e., student growth) precedes student attainment. Higher/lower rates of
learning lead to high/low attainment.
Student Level Recovery: ELA
• Though it is impossible to know exactly what level of attainment students today
would have reached had the pandemic not occurred, a conservative basis of
comparison would be the 2019 student scores (mean scaled score & % proficient)
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
2020 No Testing
Student Level Recovery: Mathematics
• Though it is impossible to know exactly what level of attainment students today
would have reached had the pandemic not occurred, a conservative basis of
comparison would be the 2019 student scores (mean scaled score & % proficient)
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
2020 No Testing
System Level Recovery: ELA
• Based upon pre-pandemic rates of student learning, we can compare rates of
learning following the pandemic to determine the extent to which the system has
returned to “normal”.
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
2019 NA 50 50 50 50 50
2020 No Testing
2021 NA NA 49 45 49 46
2022 NA 51 52 54 48 51
2023 NA 52 58 55 55 59
System Level Recovery: Mathematics
• Based upon pre-pandemic rates of student learning, we can compare rates of
learning following the pandemic to determine the extent to which the system has
returned to “normal”.
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
2019 NA 50 50 50 50 50
2020 No Testing
2021 NA NA 37 35 35 36
2022 NA 55 56 56 53 54
2023 NA 52 51 54 52 54
Four Academic
Recovery
Scenarios with NH-
SAS: V-, U-, L-, and
No-recovery
2021-2022 & 2022-23: Recovery Phase
• Overall results suggest that New Hampshire students are experiencing something resembling a U-
shaped recovery.
• Depending upon how gradual the U-shaped recovery, the time-span needed to fully recovery may
extend beyond the amount of time
• The education system has recovered and is actually performing slightly better than it had
historically. in that it has returned to rates of learning that it produced pre-pandemic.
• It is critical to note that system level recovery is not the same as student recovery.
• Because student learning was significantly depressed between 2019 and 2021. Student recovery
will require rates of learning well in excess of what was typical pre-pandemic.
• Compared to results the Center has modeled in a dozen other states, New Hampshire is at or near
the top in terms of how it’s system and students are faring.
• Averages can obscure important variability. Whereas some students are faring well, others
struggle.
SAS Reading and Mathematics
New Hampshire 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.38 −0.09 0.12 0.38 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
Reading Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization
Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
African American Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Recovery
Asian Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Stabilization Stabilization
White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization
Limited English Proficient Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization
Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Male Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
SAS ELA and Mathematics
New Hampshire 2022 to 2023 COVID⌧19 Academic Recovery Grade & Content Area
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.38 −0.09 0.12 0.38 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Non−Economically Disadvantaged Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
African American Stabilization Deterioration Stabilization Stabilization Stabilization Recovery Recovery Stabilization Deterioration Recovery Recovery Recovery
Asian Stabilization Stabilization Recovery Stabilization Stabilization Recovery Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization
White Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization
Limited English Proficient Stabilization Stabilization Recovery Stabilization Recovery Recovery Stabilization Stabilization Stabilization Recovery Stabilization Recovery
Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Female Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Male Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
2021-2022 & 2022-23: Recovery Phase
• Results from 2023 following those in 2022 suggest New Hampshire is on the right path.
• In both ELA and mathematics, rates of learning returned to historical (pre-pandemic
norms) and in many grades exceeded pre-pandemic rates of learning. This increase in
rates of learning beyond pre-pandemic is helping students recover and to gradually
catch-back-up.
• The education system has recovered and is actually performing slightly better than it had
historically. in that it has returned to rates of learning that it produced pre-pandemic.
• It is critical to note that system level recovery is not the same as student recovery.
• Because student learning was significantly depressed between 2019 and 2021. Student
recovery will require rates of learning well in excess of what was typical pre-pandemic.
• Students not making sufficient progress eventually run out of time needed to catch-up.
About this copy
New Hampshire 2023 Academic Impact Recovery Results — New Hampshire Department of Education, July 22, 2023.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.