COVID-19 Academic Impact
Conceptual Basis and Results
Damian Betebenner
Center for Assessment
Center for Assessment Staff Meeting
November 10th, 2021
Academic Impact
• Suggesting that the pandemic has impacted students academically
implies that had the pandemic not happened there would have
been a different (better?) outcome for the students.
• This alternate outcome suggests a counter-factual:
• Outcome Pandemic: Academic status of student(s) where the
pandemic occurs.
• Outcome No-pandemic: Academic status of student(s) where
the pandemic does not occur.
• We use educational assessments to detect this impact.
Academic Impact
• Suggesting that the pandemic has impacted students academically
implies that had the pandemic not happened there would have
been a different (better?) outcome for the students.
• This alternate outcome suggests a counter-factual:
• Outcome Pandemic: Academic status of student(s) where the
pandemic occurs.
• Outcome No-pandemic: Academic status of student(s) where
the pandemic does not occur.
• We use educational assessments to detect this impact.
Academic Impact
• The pandemic and all the various, ensuing disruptions functions as a kind of “academic
headwind”, impeding (in general) the academic progress of students.
• Headwinds impede progress in two ways:
• They slow one’s rate of progress
• The slower rate of progress leads to less distance being travelled.
• In education these two impediments manifest themselves (roughly) with assessments as:
• Decrease in student growth (decrease in learning velocity).
• Decrease in student attainment (decrease in amount learned).
• In some states, understanding this phenomenon requires special consideration of low
participation/changing demographics.
Focusing State
Educational
Accountability
Systems: Four
Methods for
Judging School
Quality and
Progress
Dale Carlson
Academic Impact vis-à-vis Status
• Decrease in attainment (as a decrease in percent proficient) between 2019 and 2021 by grade and
content area is the most often reported statistic related to state testing in 2021.
• Other metrics are possible:
• Decrease in mean scale score (with a vertical scale)
• Decrease in percentile rank (NWEA reports this)
• Decrease/increase in percent of students below some threshold (iReady reports this)
• Using 2019 cohort status as a counterfactual for where students would have been in 2021 had the
pandemic not occurred is problematic for several reasons including:
• Changes in student cohort composition are not accounted for.
• Improvement between 2019 and 2021 that would have occurred is not accounted for.
• Missing data in 2021 is not accounted for.
Wisconsin (WSAS)
Status Change:
MAP Status Change:
Ethnicity by Grade
iReady Status Change:
By Grade (Reading)
Discussion
• Across the dozen states I’ve looked at, larger status drops, in general,
in mathematics than ELA.
• At grade by content area level, decreases in math usually between 10
and 20 percentage points, ELA between 5 and 15 percentage points.
• Effect sizes decreases between -0.2 and -0.4 in math, -0.05 to -0.3 in
ELA.
• Drops are unambiguously large.
• There are differences in identifying which groups are impacted more
than others.
Pandemic Related Academic Impact in Indiana 10/29/21 11
Academic Impact vis-à-vis Growth
• Conditional status is a sharper way to investigate impact.
• SGPs are conditional status.
• Ho’s Fair Trend is conditional status.
• Gain scores
• Consistent with the SGP results, ILEARN gain scores were, in general,
less than what is traditionally observed across a two-year time span.
• The following figures show gain/loss by prior achievement (i.e., scale
score) with B-spline curves of best fit for both 2019 to 2021 and 2017
to 2021.
Pandemic Related Academic Impact in Indiana 10/29/21 12
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 6
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 7
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 8
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 5
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 6
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 7
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 8
Gain/Loss Score
2017 to 2019 Average Gains (2−Year Pre−COVID Gains)
2019 to 2021 Baseline P50 Growth
2019 to 2021 OLS Growth
2018 to 2019 Average Gains (1−Year Pre−COVID Gains)
2019 to 2021 Average Gains (2−Year COVID Gains)
Pre−COVID to COVID Decrease
2019 Scale Score (with prior score deciles indicated)
Questions
Pandemic Related Academic Impact in Indiana 10/29/21 27
About this copy
COVID-19 Academic Impact — Center for Assessment staff meeting, November 9, 2021.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.