COVID-19 Academic Impact in Indiana

Indiana Department of Education · July 26, 2021

COVID-19 Academic Impact in Indiana — Indiana Department of Education, July 26, 2021.
Published

July 26, 2021

Event
State agency
Slides
16
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. COVID-19 Academic Impact in Indiana Damian Betebenner, Adam Van Iwaarden & Nathan Dadey Center for Assessment Indiana TAC Meeting July 26, 2021
  2. How do we quantify academic impact using test data • The pandemic was like a “headwind” that (presumably) slowed down the learning of students due to a variety of factors. • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth • NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical growth to compare to. • Next best way: Look at how much 2021 status changed: • ILEARN: Use 2019 status (or some adjustment thereof) as a proxy. • ACCESS: Use 2020 status (or some adjustment thereof) as a proxy. • Adjustments include (potentially) imputation for missing data as well as propensity score matching for • NOTE: When providing 2021 data, users will (correctly) want to compare their data to 2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an apples-to-apples comparison. Pandemic Related Academic Impact in Indiana 2
  3. How do we quantify academic impact using test data • Historical consecutive and skip-year growth norms were created for ILEARN and ACCESS in May, 2021 for use with 2021 ILEARN and ACCESS data. • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth • NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical growth to compare to. • Next best way: Look at how much 2021 status changed: • ILEARN: Use 2019 status (or some adjustment thereof) as a proxy. • ACCESS: Use 2020 status (or some adjustment thereof) as a proxy. • Adjustments include (potentially) imputation for missing data as well as propensity score matching for • NOTE: When providing 2021 data, users will (correctly) want to compare their data to 2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an apples-to-apples comparison. Pandemic Related Academic Impact in Indiana 3
  4. Results • As the Board of Education presentation indicated, overall academic impact was moderate/large in ELA and large in mathematics. • Both status-based analyses as well as growth-based analyses support these findings. • The following tables illustrate numerically what these impacts were. • For rough comparison, the effect size impacts we are seeing for achievement are roughly 1 to 5 times as large as what was reported for Hurricane Katrina in Louisiana depending upon grade and content area. (Effect sizes of -0.06 and -0.17 are associated with Katrina). • Groups experiencing minimal impact should be scrutinized closely with regard to pre-pandemic level of educational efficacy. Pandemic Related Academic Impact in Indiana 4
  5. ILEARN Attainment: 2019 to 2021 Percent Proficient ELA Mathematics Grade Percent Percent Difference Percent Percent Difference Proficient Proficient 2019 to 2021 Proficient Proficient 2019 to 2021 Pandemic Related Academic Impact in Indiana 5
  6. ILEARN Attainment: 2019 to 2021 Mean Scale Score Difference Gamma Effect Size: ELA Decile SS 2019 gamma.ss.ela.g3 gamma.ss.ela.g4 gamma.ss.ela.g5 gamma.ss.ela.g6 gamma.ss.ela.g7 gamma.ss.ela.g8 Pandemic Related Academic Impact in Indiana 6
  7. ILEARN Attainment: 2019 to 2021 Mean Scale Score Difference Gamma Effect Size: Mathematics 2019 SS Decile gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math. Pandemic Related Academic Impact in Indiana 7
  8. ILEARN Growth: 2019 to 2021 Student Growth ELA Mathematics Grade Median SGP Median SGP Difference 2019 Median SGP Median SGP Difference 2019 2019 2021 to 2021 2019 2021 to 2021 Pandemic Related Academic Impact in Indiana 8
  9. ILEARN Growth: 2019 to 2021 SGP Difference Gamma Effect Size: ELA Decile SS 2019 gamma.sgp.ela.g5 gamma.sgp.ela.g6 gamma.sgp.ela.g7 gamma.sgp.ela.g8 Pandemic Related Academic Impact in Indiana 9
  10. ILEARN Growth: 2019 to 2021 SGP Difference Gamma Effect Size: Mathematics Decile SS 2019 gamma.sgp.math.g5 gamma.sgp.math.g6 gamma.sgp.math.g7 gamma.sgp.math.g8 Pandemic Related Academic Impact in Indiana 10
  11. ILEARN Attainment: 2019 to 2021 Percent Proficient by Ethnicity ELA Mathematics Ethnicity Percent Percent Difference 2019 Percent Percent Difference 2019 Proficient 2019 Proficient to 2021 Proficient 2019 Proficient 2021 to 2021 American Indian 44% 37.3% -6.7% 42.9% 31.3% -11.6% Asian 63.8% 56.7% -7.1% 67.9% 56.5% -11.4% Black 24.7% 17.4% -7.3% 22.5% 11.6% -10.9% Hispanic 35.5% 27.3% -8.2% 34.4% 22.0% -12.4% Multiracial 43.3% 36.4% -6.9% 41.7% 30.5% -11.2% Native Hawaiian 42.4% 32.7% -9.7% 46.1% 29.1% -17.0% or PI White 54.2% 47.1% -7.1% 54.6% 44.4% -12.2% Pandemic Related Academic Impact in Indiana 11
  12. ILEARN Growth: 2019 to 2021 Student Growth by Ethnicity ELA Mathematics Ethnicity Median SGP Median Difference 2019 Median Median Difference 2019 2019 SGP to 2021 SGP SGP to 2021 American Indian 49.5 40.0 -9.5 49.0 30.0 -19.0 Asian 66.0 56.0 -13.0 68.0 44.0 -24.0 Black 42.0 30.0 -12.0 43.0 21.0 -22.0 Hispanic 51.0 37.0 -14.0 49.0 28.0 -21.0 Multiracial 47.0 37.0 -10.0 47.0 27.0 -20.0 Native Hawaiian 50.0 42.0 -8.0 57.0 35.0 -22.0 or PI White 51.0 41.0 -10.0 51.0 33.0 -18.0 Pandemic Related Academic Impact in Indiana 12
  13. ILEARN Attainment: 2019 to 2021 Student Attainment by Special Education Status ELA Mathematics Special Education Median Median Difference Median Median Difference Status SGP SGP 2019 to SGP SGP 2019 to General 52.0 41.0 -11.0 52.0 31.0 -21.0 Education Special 41.0 31.0 -10.0 36.0 29.0 -7.0 Education Pandemic Related Academic Impact in Indiana 13
  14. ILEARN Growth: 2019 to 2021 Student Growth by Special Education Status ELA Mathematics Special Education Median Median Difference Median Median Difference Status SGP SGP 2019 to SGP SGP 2019 to General 52.0 41.0 -11.0 52.0 31.0 -21.0 Education Special 41.0 31.0 -10.0 36.0 29.0 -7.0 Education Pandemic Related Academic Impact in Indiana 14
  15. Performance by initial level of achievement • A consistent finding put forward from interim assessment reports and picked up by educational pundits is that it is the most ”vulnerable” who are experiencing the biggest impacts. • Based solely upon demographic subgroup analysis, that finding was not supported. • To dig deeper we regularly look at SGP distributions by prior score decile. For cohort referenced SGPs we expect a uniform distribution. For baseline referenced SGPs, deviations from uniform indicate impact. Pandemic Related Academic Impact in Indiana 15
  16. School level impact • We are currently in the process of finalizing school level impact analyses and want to share some of the initial looks at these results. • To calculate impact at the school level we calculate for each school: Unadjusted Impact = Median SGP2019- Median SGP2021 • We adjust for regression to the mean for this difference score by using pre-pandemic (2017 to 2019) school level median SGP scores. • The following bubble plots illustrate academic impact for Indiana Schools. Pandemic Related Academic Impact in Indiana 16