Report on COVID-19 Academic Impact in Indiana
An Overview
Damian Betebenner, Adam Van Iwaarden & Nathan Dadey
Center for Assessment
Indiana Assessment Technical Advisory Committee
May 13, 2021
Indiana NWEA MAP Assessment Administration
• NWEA provided Indiana DOE with NWEA MAP assessment data
from their Fall 2020-2021 and Winter 2020-2021 administration
windows. (??? Dates for the administration windows, who tested
remotely versus in-person)
• NWEA MAP assessment data consisted primarily of Language
Arts and Mathematics data. There were minor case counts in
Science.
• NWEA MAP data existed for Grades K to 10
• Statewide participation in NWEA MAP assessments approached
and, in most cases, exceeded 40% in all grades except 9 and 10
Pandemic Related Academic Impact in Indiana 2
Indiana NWEA MAP Assessment Administration
Pandemic Related Academic Impact in Indiana 3
Pandemic related academic impact
• Due to school disruption, it is commonly believed that students have been negatively
impacted academically.
• If we think of a student progressing forward academically, the pandemic has, by and
large, acted like a headwind, slowing the academic progress of students.
• Academic impact (sometimes referred to learning loss or unfinished learning) is the
difference between where the student’s academic attainment would have been had the
pandemic not happened and where the student’s academic attainment actually was.
• Two ways of examining academic impact:
• Status comparisons (how does current status for a group (e.g., 4th graders) compare to that group (4th
graders) in the past. Assumes comparable groups.
• Growth comparisons (how does current growth for a group (e.g., 4th graders) compare to that group
(4th graders) in the past. Assumes comparable groups.
Pandemic Related Academic Impact in Indiana 4
Pandemic related academic impact
• Because we only have Fall 2020-2021 and Winter 2020-2021 data we cannot
perform status comparisons with historical data.
• We are similarly impeded with growth data but because growth percentiles
(nationally) have a mean/median of 50, we can look at deviations from 50 as a
rough estimate of the academic impact.
• Ideally, instead of 50 we would use historical growth mean/median growth
percentiles for Indiana to determine how current growth deviates from historical
growth.
• We will proceed with this line of analysis/reasoning with the hope of tightening
the analyses going forward with additional historical data.
Pandemic Related Academic Impact in Indiana 5
Two views of academic impact (Phase 1 to Phase 2)
• NWEA MAP assessment data provides several student growth
percentiles (they refer to it as a conditional growth percentile)
• NWEA MAP assessments are given 3 times a year leading to
several growth scenarios:
• Fall to Fall
• Fall to Winter
• Fall to Spring
• Winter to Winter
• Winter to Fall (not calculated)
Pandemic Related Academic Impact in Indiana 6
Pandemic related academic impact/learning loss
Pandemic Related Academic Impact in Indiana 7
Two views of academic impact (Phase 1 to Phase 2)
• Fall to Fall Growth (i.e., Fall 2019 to Fall 2020) will provide us
with some insight into the decreased learning rates of students.
• Winter to Winter Growth (i.e., Winter 2019-2020 to Winter
2020-2021) will similarly provide us with some insight into the
decreased learning rates of students.
Pandemic Related Academic Impact in Indiana 8
Two views of academic impact (Phase 1 to Phase 2)
Fall to Fall Growth Winter to Winter Growth
Pandemic Related Academic Impact in Indiana 9
One view of academic maintenance/recovery (Phase 2)
Fall to Winter Growth
Pandemic Related Academic Impact in Indiana 10
Fall-to-Fall Growth (Growth Before-During Pandemic)
Grade Language Arts Mathematics
Mean SGP Count Mean SGP Count
Pandemic Related Academic Impact in Indiana 11
Winter-to-Winter Growth (Growth Before-During
Pandemic)
Grade Language Arts Mathematics
Mean SGP Count Mean SGP Count
Pandemic Related Academic Impact in Indiana 12
Fall-to-Winter Growth (Growth during pandemic)
Grade Language Arts Mathematics
Mean SGP Count Mean SGP Count
Pandemic Related Academic Impact in Indiana 13
Fall-to-Fall Growth (by ethnicity, across all grades)
Grade Language Arts Mathematics
Mean SGP Count Mean SGP Count
American Indian 48.9 446 40.0 339
Asian 54.6 6,776 51.7 5,612
Black 44.7 20,176 39.2 17,469
Hispanic 46.1 32,237 39.1 25,308
Multiracial 46.6 14,544 39.3 11,653
Native Hawaiian/PI 43.7 348 40.2 266
White 47.4 228,253 38.5 174,291
Pandemic Related Academic Impact in Indiana 14
Fall-to-Fall Growth (larger districts)
Grade Language Arts Mathematics
Mean SGP Count Mean SGP Count
Pandemic Related Academic Impact in Indiana 15
Interpretation of growth results
• NOTE: We assume normal growth in Indian pre-COVID was
approximately 50.
• Deviation from 50 is indicative of academic impact/learning loss.
• A change in mean SGP of 5 corresponds to an effect size of 0.18
• Roughly (trying to translate into time):
• Effect size >= 0.6 (mean SGP < 35): 12+ weeks behind
• 0.45 < Effect size < 0.6 (35 < mean SGP < 40): 8-11 weeks behind
• 0.2 < Effect size < 0.45 (40 < mean SGP < 45): 3 to 8 weeks behind
• Effect size < 0.2 (mean SGP >=45): Less than 3 weeks behind
Pandemic Related Academic Impact in Indiana 16
Summary of growth results
• Academic impact was more substantial in mathematics than in
language arts.
• One hypothesis for the lesser impact in language arts than in
mathematics is that language arts is more easily “taught” at home.
• Another hypothesis for the lesser impact in language arts is that the
NWEA MAP assessment don’t address the content standards to the
same depth as would a state summative assessment.
• Academic impact was slightly larger in lower grades than in upper
grades.
• Results are consistent with what has been observed in other
states.
Pandemic Related Academic Impact in Indiana 17
About this copy
Report on COVID-19 Academic Impact in Indiana: An Overview — Indiana Department of Education, May 13, 2021.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.