Report on COVID-19 Academic Impact in Indiana: An Overview

Indiana Department of Education · May 13, 2021

Report on COVID-19 Academic Impact in Indiana: An Overview — Indiana Department of Education, May 13, 2021.
Published

May 13, 2021

Event
State agency
Slides
17
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Report on COVID-19 Academic Impact in Indiana An Overview Damian Betebenner, Adam Van Iwaarden & Nathan Dadey Center for Assessment Indiana Assessment Technical Advisory Committee May 13, 2021
  2. Indiana NWEA MAP Assessment Administration • NWEA provided Indiana DOE with NWEA MAP assessment data from their Fall 2020-2021 and Winter 2020-2021 administration windows. (??? Dates for the administration windows, who tested remotely versus in-person) • NWEA MAP assessment data consisted primarily of Language Arts and Mathematics data. There were minor case counts in Science. • NWEA MAP data existed for Grades K to 10 • Statewide participation in NWEA MAP assessments approached and, in most cases, exceeded 40% in all grades except 9 and 10 Pandemic Related Academic Impact in Indiana 2
  3. Indiana NWEA MAP Assessment Administration Pandemic Related Academic Impact in Indiana 3
  4. Pandemic related academic impact • Due to school disruption, it is commonly believed that students have been negatively impacted academically. • If we think of a student progressing forward academically, the pandemic has, by and large, acted like a headwind, slowing the academic progress of students. • Academic impact (sometimes referred to learning loss or unfinished learning) is the difference between where the student’s academic attainment would have been had the pandemic not happened and where the student’s academic attainment actually was. • Two ways of examining academic impact: • Status comparisons (how does current status for a group (e.g., 4th graders) compare to that group (4th graders) in the past. Assumes comparable groups. • Growth comparisons (how does current growth for a group (e.g., 4th graders) compare to that group (4th graders) in the past. Assumes comparable groups. Pandemic Related Academic Impact in Indiana 4
  5. Pandemic related academic impact • Because we only have Fall 2020-2021 and Winter 2020-2021 data we cannot perform status comparisons with historical data. • We are similarly impeded with growth data but because growth percentiles (nationally) have a mean/median of 50, we can look at deviations from 50 as a rough estimate of the academic impact. • Ideally, instead of 50 we would use historical growth mean/median growth percentiles for Indiana to determine how current growth deviates from historical growth. • We will proceed with this line of analysis/reasoning with the hope of tightening the analyses going forward with additional historical data. Pandemic Related Academic Impact in Indiana 5
  6. Two views of academic impact (Phase 1 to Phase 2) • NWEA MAP assessment data provides several student growth percentiles (they refer to it as a conditional growth percentile) • NWEA MAP assessments are given 3 times a year leading to several growth scenarios: • Fall to Fall • Fall to Winter • Fall to Spring • Winter to Winter • Winter to Fall (not calculated) Pandemic Related Academic Impact in Indiana 6
  7. Pandemic related academic impact/learning loss Pandemic Related Academic Impact in Indiana 7
  8. Two views of academic impact (Phase 1 to Phase 2) • Fall to Fall Growth (i.e., Fall 2019 to Fall 2020) will provide us with some insight into the decreased learning rates of students. • Winter to Winter Growth (i.e., Winter 2019-2020 to Winter 2020-2021) will similarly provide us with some insight into the decreased learning rates of students. Pandemic Related Academic Impact in Indiana 8
  9. Two views of academic impact (Phase 1 to Phase 2) Fall to Fall Growth Winter to Winter Growth Pandemic Related Academic Impact in Indiana 9
  10. One view of academic maintenance/recovery (Phase 2) Fall to Winter Growth Pandemic Related Academic Impact in Indiana 10
  11. Fall-to-Fall Growth (Growth Before-During Pandemic) Grade Language Arts Mathematics Mean SGP Count Mean SGP Count Pandemic Related Academic Impact in Indiana 11
  12. Winter-to-Winter Growth (Growth Before-During Pandemic) Grade Language Arts Mathematics Mean SGP Count Mean SGP Count Pandemic Related Academic Impact in Indiana 12
  13. Fall-to-Winter Growth (Growth during pandemic) Grade Language Arts Mathematics Mean SGP Count Mean SGP Count Pandemic Related Academic Impact in Indiana 13
  14. Fall-to-Fall Growth (by ethnicity, across all grades) Grade Language Arts Mathematics Mean SGP Count Mean SGP Count American Indian 48.9 446 40.0 339 Asian 54.6 6,776 51.7 5,612 Black 44.7 20,176 39.2 17,469 Hispanic 46.1 32,237 39.1 25,308 Multiracial 46.6 14,544 39.3 11,653 Native Hawaiian/PI 43.7 348 40.2 266 White 47.4 228,253 38.5 174,291 Pandemic Related Academic Impact in Indiana 14
  15. Fall-to-Fall Growth (larger districts) Grade Language Arts Mathematics Mean SGP Count Mean SGP Count Pandemic Related Academic Impact in Indiana 15
  16. Interpretation of growth results • NOTE: We assume normal growth in Indian pre-COVID was approximately 50. • Deviation from 50 is indicative of academic impact/learning loss. • A change in mean SGP of 5 corresponds to an effect size of 0.18 • Roughly (trying to translate into time): • Effect size >= 0.6 (mean SGP < 35): 12+ weeks behind • 0.45 < Effect size < 0.6 (35 < mean SGP < 40): 8-11 weeks behind • 0.2 < Effect size < 0.45 (40 < mean SGP < 45): 3 to 8 weeks behind • Effect size < 0.2 (mean SGP >=45): Less than 3 weeks behind Pandemic Related Academic Impact in Indiana 16
  17. Summary of growth results • Academic impact was more substantial in mathematics than in language arts. • One hypothesis for the lesser impact in language arts than in mathematics is that language arts is more easily “taught” at home. • Another hypothesis for the lesser impact in language arts is that the NWEA MAP assessment don’t address the content standards to the same depth as would a state summative assessment. • Academic impact was slightly larger in lower grades than in upper grades. • Results are consistent with what has been observed in other states. Pandemic Related Academic Impact in Indiana 17