COVID-19 Academic Impact in Indiana
Damian Betebenner, Adam Van Iwaarden & Nathan Dadey
Center for Assessment
Indiana TAC Meeting
July 26, 2021
How do we quantify academic impact using test data
• The pandemic was like a “headwind” that (presumably) slowed down the learning of
students due to a variety of factors.
• We can use test data in two ways to quantify the impact of this headwind:
• Best way: Quantify how much the rate of learning (i.e., growth) slowed:
• ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth.
• ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth
• NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical
growth to compare to.
• Next best way: Look at how much 2021 status changed:
• ILEARN: Use 2019 status (or some adjustment thereof) as a proxy.
• ACCESS: Use 2020 status (or some adjustment thereof) as a proxy.
• Adjustments include (potentially) imputation for missing data as well as propensity score matching for
• NOTE: When providing 2021 data, users will (correctly) want to compare their data to
2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an
apples-to-apples comparison.
Pandemic Related Academic Impact in Indiana 2
How do we quantify academic impact using test data
• Historical consecutive and skip-year growth norms were created for ILEARN and
ACCESS in May, 2021 for use with 2021 ILEARN and ACCESS data.
• We can use test data in two ways to quantify the impact of this headwind:
• Best way: Quantify how much the rate of learning (i.e., growth) slowed:
• ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth.
• ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth
• NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical
growth to compare to.
• Next best way: Look at how much 2021 status changed:
• ILEARN: Use 2019 status (or some adjustment thereof) as a proxy.
• ACCESS: Use 2020 status (or some adjustment thereof) as a proxy.
• Adjustments include (potentially) imputation for missing data as well as propensity score matching for
• NOTE: When providing 2021 data, users will (correctly) want to compare their data to
2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an
apples-to-apples comparison.
Pandemic Related Academic Impact in Indiana 3
Results
• As the Board of Education presentation indicated, overall academic impact was
moderate/large in ELA and large in mathematics.
• Both status-based analyses as well as growth-based analyses support these
findings.
• The following tables illustrate numerically what these impacts were.
• For rough comparison, the effect size impacts we are seeing for achievement
are roughly 1 to 5 times as large as what was reported for Hurricane Katrina in
Louisiana depending upon grade and content area. (Effect sizes of -0.06 and
-0.17 are associated with Katrina).
• Groups experiencing minimal impact should be scrutinized closely with regard
to pre-pandemic level of educational efficacy.
Pandemic Related Academic Impact in Indiana 4
ILEARN Attainment: 2019 to 2021 Percent Proficient
ELA Mathematics
Grade
Percent Percent Difference Percent Percent Difference
Proficient Proficient 2019 to 2021 Proficient Proficient 2019 to 2021
Pandemic Related Academic Impact in Indiana 5
ILEARN Attainment: 2019 to 2021 Mean Scale Score
Difference Gamma Effect Size: ELA
Decile SS 2019 gamma.ss.ela.g3 gamma.ss.ela.g4 gamma.ss.ela.g5 gamma.ss.ela.g6 gamma.ss.ela.g7 gamma.ss.ela.g8
Pandemic Related Academic Impact in Indiana 6
ILEARN Attainment: 2019 to 2021 Mean Scale Score
Difference Gamma Effect Size: Mathematics
2019 SS Decile gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math. gamma.ss.math.
Pandemic Related Academic Impact in Indiana 7
ILEARN Growth: 2019 to 2021 Student Growth
ELA Mathematics
Grade
Median SGP Median SGP Difference 2019 Median SGP Median SGP Difference 2019
2019 2021 to 2021 2019 2021 to 2021
Pandemic Related Academic Impact in Indiana 8
ILEARN Growth: 2019 to 2021 SGP Difference Gamma Effect
Size: ELA
Decile SS 2019 gamma.sgp.ela.g5 gamma.sgp.ela.g6 gamma.sgp.ela.g7 gamma.sgp.ela.g8
Pandemic Related Academic Impact in Indiana 9
ILEARN Growth: 2019 to 2021 SGP Difference Gamma Effect
Size: Mathematics
Decile SS 2019 gamma.sgp.math.g5 gamma.sgp.math.g6 gamma.sgp.math.g7 gamma.sgp.math.g8
Pandemic Related Academic Impact in Indiana 10
ILEARN Attainment: 2019 to 2021 Percent Proficient by
Ethnicity
ELA Mathematics
Ethnicity
Percent Percent Difference 2019 Percent Percent Difference 2019
Proficient 2019 Proficient to 2021 Proficient 2019 Proficient 2021 to 2021
American Indian 44% 37.3% -6.7% 42.9% 31.3% -11.6%
Asian 63.8% 56.7% -7.1% 67.9% 56.5% -11.4%
Black 24.7% 17.4% -7.3% 22.5% 11.6% -10.9%
Hispanic 35.5% 27.3% -8.2% 34.4% 22.0% -12.4%
Multiracial 43.3% 36.4% -6.9% 41.7% 30.5% -11.2%
Native Hawaiian 42.4% 32.7% -9.7% 46.1% 29.1% -17.0%
or PI
White 54.2% 47.1% -7.1% 54.6% 44.4% -12.2%
Pandemic Related Academic Impact in Indiana 11
ILEARN Growth: 2019 to 2021 Student Growth by Ethnicity
ELA Mathematics
Ethnicity Median SGP Median Difference 2019 Median Median Difference 2019
2019 SGP to 2021 SGP SGP to 2021
American Indian 49.5 40.0 -9.5 49.0 30.0 -19.0
Asian 66.0 56.0 -13.0 68.0 44.0 -24.0
Black 42.0 30.0 -12.0 43.0 21.0 -22.0
Hispanic 51.0 37.0 -14.0 49.0 28.0 -21.0
Multiracial 47.0 37.0 -10.0 47.0 27.0 -20.0
Native Hawaiian 50.0 42.0 -8.0 57.0 35.0 -22.0
or PI
White 51.0 41.0 -10.0 51.0 33.0 -18.0
Pandemic Related Academic Impact in Indiana 12
ILEARN Attainment: 2019 to 2021 Student Attainment by
Special Education Status
ELA Mathematics
Special
Education Median Median Difference Median Median Difference
Status SGP SGP 2019 to SGP SGP 2019 to
General 52.0 41.0 -11.0 52.0 31.0 -21.0
Education
Special 41.0 31.0 -10.0 36.0 29.0 -7.0
Education
Pandemic Related Academic Impact in Indiana 13
ILEARN Growth: 2019 to 2021 Student Growth by Special
Education Status
ELA Mathematics
Special
Education Median Median Difference Median Median Difference
Status SGP SGP 2019 to SGP SGP 2019 to
General 52.0 41.0 -11.0 52.0 31.0 -21.0
Education
Special 41.0 31.0 -10.0 36.0 29.0 -7.0
Education
Pandemic Related Academic Impact in Indiana 14
Performance by initial level of achievement
• A consistent finding put forward from interim assessment reports and
picked up by educational pundits is that it is the most ”vulnerable”
who are experiencing the biggest impacts.
• Based solely upon demographic subgroup analysis, that finding was
not supported.
• To dig deeper we regularly look at SGP distributions by prior score
decile. For cohort referenced SGPs we expect a uniform distribution.
For baseline referenced SGPs, deviations from uniform indicate
impact.
Pandemic Related Academic Impact in Indiana 15
School level impact
• We are currently in the process of finalizing school level impact
analyses and want to share some of the initial looks at these results.
• To calculate impact at the school level we calculate for each school:
Unadjusted Impact = Median SGP2019- Median SGP2021
• We adjust for regression to the mean for this difference score by using
pre-pandemic (2017 to 2019) school level median SGP scores.
• The following bubble plots illustrate academic impact for Indiana
Schools.
Pandemic Related Academic Impact in Indiana 16
About this copy
COVID-19 Academic Impact in Indiana — Indiana Department of Education, July 26, 2021.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.