COVID-19 Academic Impact

Center for Assessment staff meeting · November 9, 2021

COVID-19 Academic Impact — Center for Assessment staff meeting, November 9, 2021.
Published

November 9, 2021

Event
Staff session
Slides
27
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. COVID-19 Academic Impact Conceptual Basis and Results Damian Betebenner Center for Assessment Center for Assessment Staff Meeting November 10th, 2021
  2. Academic Impact • Suggesting that the pandemic has impacted students academically implies that had the pandemic not happened there would have been a different (better?) outcome for the students. • This alternate outcome suggests a counter-factual: • Outcome Pandemic: Academic status of student(s) where the pandemic occurs. • Outcome No-pandemic: Academic status of student(s) where the pandemic does not occur. • We use educational assessments to detect this impact.
  3. Academic Impact • Suggesting that the pandemic has impacted students academically implies that had the pandemic not happened there would have been a different (better?) outcome for the students. • This alternate outcome suggests a counter-factual: • Outcome Pandemic: Academic status of student(s) where the pandemic occurs. • Outcome No-pandemic: Academic status of student(s) where the pandemic does not occur. • We use educational assessments to detect this impact.
  4. Academic Impact • The pandemic and all the various, ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two impediments manifest themselves (roughly) with assessments as: • Decrease in student growth (decrease in learning velocity). • Decrease in student attainment (decrease in amount learned). • In some states, understanding this phenomenon requires special consideration of low participation/changing demographics.
  6. Focusing State Educational Accountability Systems: Four Methods for Judging School Quality and Progress Dale Carlson
  7. Academic Impact vis-à-vis Status • Decrease in attainment (as a decrease in percent proficient) between 2019 and 2021 by grade and content area is the most often reported statistic related to state testing in 2021. • Other metrics are possible: • Decrease in mean scale score (with a vertical scale) • Decrease in percentile rank (NWEA reports this) • Decrease/increase in percent of students below some threshold (iReady reports this) • Using 2019 cohort status as a counterfactual for where students would have been in 2021 had the pandemic not occurred is problematic for several reasons including: • Changes in student cohort composition are not accounted for. • Improvement between 2019 and 2021 that would have occurred is not accounted for. • Missing data in 2021 is not accounted for.
  8. Wisconsin (WSAS) Status Change:
  9. MAP Status Change: Ethnicity by Grade
  10. iReady Status Change: By Grade (Reading)
  11. Discussion • Across the dozen states I’ve looked at, larger status drops, in general, in mathematics than ELA. • At grade by content area level, decreases in math usually between 10 and 20 percentage points, ELA between 5 and 15 percentage points. • Effect sizes decreases between -0.2 and -0.4 in math, -0.05 to -0.3 in ELA. • Drops are unambiguously large. • There are differences in identifying which groups are impacted more than others. Pandemic Related Academic Impact in Indiana 10/29/21 11
  12. Academic Impact vis-à-vis Growth • Conditional status is a sharper way to investigate impact. • SGPs are conditional status. • Ho’s Fair Trend is conditional status. • Gain scores • Consistent with the SGP results, ILEARN gain scores were, in general, less than what is traditionally observed across a two-year time span. • The following figures show gain/loss by prior achievement (i.e., scale score) with B-spline curves of best fit for both 2019 to 2021 and 2017 to 2021. Pandemic Related Academic Impact in Indiana 10/29/21 12
  13. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2019 Scale Score (with prior score deciles indicated)
  14. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2019 Scale Score (with prior score deciles indicated)
  15. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2019 Scale Score (with prior score deciles indicated)
  16. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  17. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  18. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2019 Scale Score (with prior score deciles indicated)
  19. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 5 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  20. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 6 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  21. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 7 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  22. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: ELA Grade 8 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  23. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 5 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  24. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 6 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  25. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 7 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  26. All Students: 2019 to 2021 Gain/Loss by 2019 Prior Achievement: MATHEMATICS Grade 8 Gain/Loss Score 2017 to 2019 Average Gains (2−Year Pre−COVID Gains) 2019 to 2021 Baseline P50 Growth 2019 to 2021 OLS Growth 2018 to 2019 Average Gains (1−Year Pre−COVID Gains) 2019 to 2021 Average Gains (2−Year COVID Gains) Pre−COVID to COVID Decrease 2019 Scale Score (with prior score deciles indicated)
  27. Questions Pandemic Related Academic Impact in Indiana 10/29/21 27