COVID-19 Academic Impact in Illinois
Damian Betebenner & Adam Van Iwaarden
Center for Assessment
July 26, 2021
How do we quantify academic impact using test data
• The pandemic was like a “headwind” that (presumably) slowed down the
learning of students due to a variety of factors.
• Headwinds impact progress in two ways.
• They decrease the rate of progress (i.e., decrease student growth).
• They decrease the distance travelled (i.e., decrease attainment).
• We can use test data in two ways to quantify the impact of this headwind:
• Best way: Quantify how much the rate of learning (i.e., growth) slowed:
• IAR: Use historical skip-year growth norms to quantify growth and compare to pre-COVID
growth.
• Next best way: Look at how much 2021 status changed:
• ILEARN: Use 2019 status (or some adjustment thereof) as a proxy.
• NOTE: When providing 2021 data, users will (correctly) want to compare
their data to 2019 (or 2020 with ACCESS). The task is to make sure that
such a comparison is an apples-to-apples comparison.
Pandemic Related Academic Impact in Illinois 2
Results
• Overall academic impact was large in both ELA and mathematics.
• Recovery associated with large impacts will likely take multiple years and
require additional supports beyond just normal schooling (right-size the
intervention)
• Both status-based analyses as well as growth-based analyses support these
findings.
• The following tables illustrate numerically what these impacts were.
• For rough comparison, the effect size impacts we are seeing for
achievement are roughly 1 to 5 times as large as what were reported for
Hurricane Katrina in Louisiana depending upon grade and content area.
(Effect sizes of -0.06 and -0.17 are associated with Katrina).
• Groups experiencing minimal impact should be scrutinized closely with
regard to pre-pandemic level of educational efficacy.
Pandemic Related Academic Impact in Illinois 3
IAR Attainment: 2019 to 2021 Percent Proficient
ELA Mathematics
Grade
Percent Percent Difference Percent Percent Difference
Proficient Proficient 2019 to 2021 Proficient Proficient 2019 to 2021
Pandemic Related Academic Impact in Illinois 4
IAR Growth: 2019 to 2021 Student Growth
ELA Mathematics
Grade
Median SGP Median SGP Difference 2019 Median SGP Median SGP Difference 2019
2019 2021 to 2021 2019 2021 to 2021
Pandemic Related Academic Impact in Illinois 5
Performance by initial level of achievement
• We are beginning to look at impact by demographic subgroup
(coming shortly) as well as impact by academic subgroup.
• We examined academic impact by the prior academic achievement
(low versus high achieving using prior scale score decile).
• To dig deeper we regularly look at SGP distributions by prior score
decile. For cohort referenced SGPs we expect a uniform distribution.
For baseline referenced SGPs, deviations from uniform indicate
impact.
Pandemic Related Academic Impact in Illinois 6
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 ELA SGP, Grade 5 (N = 90,180)
SGP Deciles by Prior Achievement Level (ELA)
Level 5 (3.6%)
Prior Achievement Level
Level 4 (35.4%)
Level 3 (23.2%)
Level 2 (18%)
Level 1 (19.8%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 46
Extremes (Range)
Highest OSS: [3.338,3.476) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17.4 82.6 N = 23
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.442) 14.89 16.03 11.99 9.43 11.15 7.91 7.02 8.30 6.61 6.66 (9.8%) 100
2nd/[−1.442,−0.893) 17.41 15.54 12.40 8.84 10.34 8.04 7.77 6.88 6.48 6.29 (10%)
Prior Scale Score Decile/Range *
3rd/[−0.893,−0.5659) 19.48 14.76 11.44 10.84 8.47 8.02 7.64 6.79 6.69 5.88 (9.1%)
Empirical SGP Distribution
4th/[−0.5659,−0.2032) 21.34 14.39 11.01 9.58 9.07 7.89 7.51 6.94 6.73 5.54 (10.8%)
5th/[−0.2032,0.0923) 21.32 14.89 10.89 9.28 8.30 9.86 6.16 7.41 6.22 5.66 (9.6%)
6th/[0.0923,0.3751) 20.53 13.14 12.72 7.91 8.19 9.67 7.99 7.36 6.79 5.71 (10.5%)
7th/[0.3751,0.6523) 22.91 14.15 11.50 9.01 8.75 8.36 6.88 7.28 5.92 5.26 (9.6%)
8th/[0.6523,1.026) 23.30 13.66 11.80 8.82 10.42 7.46 6.43 7.43 5.24 5.44 (10.2%)
9th/[1.026,1.443) 23.94 13.65 12.25 9.47 10.13 7.32 6.33 6.25 5.30 5.36 (10.2%)
10th/[1.443,15] 29.20 15.34 12.26 8.70 8.11 6.47 5.45 4.82 5.19 4.45 (10.3%)
*Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 ELA SGP, Grade 6 (N = 89,764)
SGP Deciles by Prior Achievement Level (ELA)
Level 5 (8.7%)
Prior Achievement Level
Level 4 (31.9%)
Level 3 (26%)
Level 2 (18.5%)
Level 1 (14.9%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 65
Extremes (Range)
Highest OSS: [3.349,3.536) 0.0 0.0 4.3 0.0 0.0 0.0 0.0 4.3 8.7 4.3 21.7 56.5 N = 23
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.652) 12.80 10.85 11.42 10.42 10.22 9.62 8.95 8.65 8.54 8.55 (9.9%) 100
2nd/[−1.652,−1.034) 14.53 11.94 11.46 11.10 9.29 9.59 9.04 8.42 7.88 6.76 (9.7%)
Prior Scale Score Decile/Range *
3rd/[−1.034,−0.6413) 15.36 12.85 11.03 10.40 9.52 9.80 8.71 8.47 7.32 6.54 (9.8%)
Empirical SGP Distribution
4th/[−0.6413,−0.2949) 15.58 13.81 11.96 11.11 9.64 9.44 8.44 7.18 6.70 6.15 (10.6%)
5th/[−0.2949,0.0302) 15.76 14.29 12.67 10.78 10.14 8.92 8.14 7.28 6.66 5.36 (9.7%)
6th/[0.0302,0.2857) 17.74 13.79 12.05 10.51 9.88 9.01 8.08 6.98 6.14 5.81 (9.8%)
7th/[0.2857,0.588) 17.84 14.14 11.66 10.48 9.88 8.72 8.25 6.83 6.37 5.82 (9.2%)
8th/[0.588,0.9404) 19.88 13.89 12.34 10.25 9.03 8.36 8.09 7.19 6.11 4.86 (10.8%)
9th/[0.9404,1.461) 22.00 15.13 12.55 9.91 8.91 7.86 7.79 6.55 5.05 4.25 (10.5%)
10th/[1.461,15] 24.18 15.32 12.04 9.61 9.38 7.88 6.79 5.77 5.55 3.47 (10%)
*Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 Mathematics SGP, Grade 5 (N = 89,200)
SGP Deciles by Prior Achievement Level (Mathematics)
Level 5 (8.7%)
Prior Achievement Level
Level 4 (34.6%)
Level 3 (25%)
Level 2 (19.7%)
Level 1 (12%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [−15,−4.44] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 48
Extremes (Range)
Highest OSS: [3.765,4.606) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.5 7.5 90.0 N = 40
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.536) 19.60 14.12 13.36 9.39 10.16 7.64 7.87 7.09 6.10 4.67 (9.7%) 100
2nd/[−1.536,−1.065) 20.13 13.97 12.86 11.41 8.99 8.80 8.03 5.97 5.11 4.71 (9.3%)
Prior Scale Score Decile/Range *
3rd/[−1.065,−0.6694) 22.57 16.70 11.73 10.83 9.09 7.35 7.13 5.20 4.79 4.62 (10.5%)
Empirical SGP Distribution
4th/[−0.6694,−0.3653) 24.21 15.13 12.06 10.18 8.74 7.65 6.61 5.44 4.87 5.11 (10.1%)
5th/[−0.3653,−0.0344) 24.34 15.12 11.83 9.49 9.19 6.98 5.77 7.01 4.63 5.65 (10.2%)
6th/[−0.0344,0.2259) 24.38 15.32 11.81 8.89 8.28 7.28 6.60 6.28 5.52 5.64 (9.8%)
7th/[0.2259,0.5214) 25.20 14.52 11.19 8.84 8.15 7.00 6.65 6.61 5.86 5.98 (9.8%)
8th/[0.5214,0.8636) 24.61 13.63 11.29 8.24 7.71 8.14 7.49 6.17 6.27 6.44 (10.5%)
9th/[0.8636,1.291) 22.94 13.52 10.62 9.09 8.27 8.49 6.91 6.79 6.60 6.78 (9.6%)
10th/[1.291,15] 19.06 12.69 9.57 9.05 8.90 9.41 6.51 7.60 7.60 9.61 (10.5%)
*Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 Mathematics SGP, Grade 6 (N = 88,633)
SGP Deciles by Prior Achievement Level (Mathematics)
Level 5 (3.6%)
Prior Achievement Level
Level 4 (34%)
Level 3 (29%)
Level 2 (19.6%)
Level 1 (13.8%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 200
Extremes (Range)
Highest OSS: [3.384,4.065) 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.0 0.0 14.8 18.5 63.0 N = 27
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.643) 10.92 12.91 12.90 12.44 11.67 10.19 9.89 8.58 5.99 4.51 (8.8%) 100
2nd/[−1.643,−1.056) 15.59 14.99 14.19 12.24 10.60 9.10 8.04 6.30 5.16 3.79 (10.6%)
Prior Scale Score Decile/Range *
3rd/[−1.056,−0.7248) 20.49 16.96 13.04 11.62 9.05 7.72 6.85 5.54 4.64 4.08 (9.7%)
Empirical SGP Distribution
4th/[−0.7248,−0.3645) 23.90 16.74 12.94 10.31 8.86 7.47 6.35 4.98 4.32 4.13 (9.8%)
5th/[−0.3645,−0.0413) 24.09 16.21 13.12 9.81 8.81 7.47 6.55 5.32 4.57 4.05 (10.8%)
6th/[−0.0413,0.1864) 23.40 15.60 12.95 10.12 8.51 7.81 6.91 5.66 4.92 4.13 (9%)
7th/[0.1864,0.5255) 23.43 15.34 11.62 10.33 9.09 7.21 6.97 5.93 5.33 4.76 (10.8%)
8th/[0.5255,0.8381) 24.13 14.89 11.92 9.85 8.79 6.87 6.96 5.91 5.31 5.38 (9.6%)
9th/[0.8381,1.321) 23.33 14.64 11.27 10.49 8.87 7.32 6.77 6.27 5.81 5.22 (10.7%)
10th/[1.321,15] 20.16 13.51 11.22 9.86 8.91 9.25 7.58 7.27 6.19 6.06 (10.1%)
*Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 Mathematics SGP, Grade 7 (N = 87,285)
SGP Deciles by Prior Achievement Level (Mathematics)
Level 5 (5.7%)
Prior Achievement Level
Level 4 (26.6%)
Level 3 (28.8%)
Level 2 (27.6%)
Level 1 (11.3%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [−15,−3.83] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 48
Extremes (Range)
Highest OSS: [4.134,4.825) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.3 8.3 29.2 12.5 41.7 N = 24
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.542) 9.38 11.29 11.09 11.81 11.40 11.10 10.10 8.99 7.69 7.15 (9.7%) 100
2nd/[−1.542,−1.139) 11.60 13.33 12.65 11.45 11.50 10.49 9.47 7.92 6.46 5.13 (9.3%)
Prior Scale Score Decile/Range *
3rd/[−1.139,−0.8564) 13.75 14.50 13.06 11.79 10.66 9.02 8.56 7.53 6.16 4.97 (9.3%)
Empirical SGP Distribution
4th/[−0.8564,−0.5544) 15.16 14.51 13.06 11.18 10.52 8.97 8.45 6.86 6.42 4.86 (10.8%)
5th/[−0.5544,−0.3049) 16.92 15.20 12.23 10.20 9.78 8.39 8.58 7.56 6.17 4.97 (9.6%)
6th/[−0.3049,0.017) 16.92 14.55 12.41 10.63 10.06 9.12 7.92 7.56 6.35 4.49 (10.1%)
7th/[0.017,0.3388) 18.54 14.81 11.71 11.44 9.89 8.49 8.36 7.10 5.67 3.98 (10.1%)
8th/[0.3388,0.749) 20.80 15.85 12.44 11.18 9.57 8.28 7.42 6.30 4.74 3.41 (10.5%)
9th/[0.749,1.286) 22.75 17.28 13.02 10.40 8.91 7.67 6.57 5.05 4.69 3.68 (10.6%)
10th/[1.286,15] 25.61 15.12 12.04 9.78 9.52 8.11 5.92 5.77 4.67 3.46 (10%)
*Prior score deciles can be uneven depending upon the prior score distribution
Student Growth Percentile Goodness−of−Fit Descriptives
2020−2021.2 Mathematics SGP, Grade 8 (N = 87,180)
SGP Deciles by Prior Achievement Level (Mathematics)
Level 5 (3.4%)
Level 4 (23.6%)
Prior Achievement Level
Level 3 (30.1%)
Level 2 (28.8%)
Level 1 (14%)
Median Student Growth Percentile
Ceiling / Floor Effects Test
Student Growth Percentile Range
Current Observed Score
5 9 to to to to to to to to to to
to to 10 20 30 40 50 60 70 80 90 95
Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 70
Extremes (Range)
Highest OSS: [4.086,4.104) 0.0 0.0 0.0 0.0 0.0 4.5 0.0 4.5 0.0 36.4 18.2 36.4 N = 22
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[−15,−1.622) 11.57 13.53 11.79 11.50 10.78 10.70 8.75 8.09 6.85 6.44 (9.7%) 100
2nd/[−1.622,−1.171) 13.35 13.98 13.05 11.04 10.22 9.79 9.02 7.47 6.66 5.42 (9.2%)
Prior Scale Score Decile/Range *
3rd/[−1.171,−0.8296) 14.24 14.83 12.13 11.82 10.30 9.73 8.27 7.27 6.39 5.03 (9.9%)
Empirical SGP Distribution
4th/[−0.8296,−0.5464) 15.95 15.21 12.25 11.22 10.47 9.53 7.96 7.11 5.84 4.46 (9.7%)
5th/[−0.5464,−0.297) 17.85 14.45 13.26 10.78 9.92 8.51 8.09 6.48 5.89 4.75 (10.7%)
6th/[−0.297,0.0185) 17.91 15.31 13.11 11.17 9.72 8.50 7.32 6.82 5.84 4.32 (10.8%)
7th/[0.0185,0.2955) 19.02 15.38 12.67 10.76 9.74 8.81 7.15 6.15 5.33 5.00 (9.5%)
8th/[0.2955,0.6893) 21.81 14.32 12.70 10.12 9.25 8.30 7.20 6.30 5.21 4.81 (10.1%)
9th/[0.6893,1.154) 20.13 15.61 12.88 10.90 8.67 8.20 7.26 6.13 5.67 4.55 (10%)
10th/[1.154,15] 19.25 14.31 11.78 10.33 8.55 8.38 8.20 7.20 6.99 5.01 (10.4%)
*Prior score deciles can be uneven depending upon the prior score distribution
School level impact
• We are currently in the process of finalizing school level impact
analyses and want to share some of the initial looks at these results.
• To calculate impact at the school level we calculate for each school:
Unadjusted Impact = Median SGP2019- Median SGP2021
• We adjust for regression to the mean for this difference score by
using pre-pandemic (2017 to 2019) school level median SGP scores.
• Results forthcoming shortly
Pandemic Related Academic Impact in Illinois 7
About this copy
COVID-19 Academic Impact in Illinois — Illinois Department of Education, July 28, 2021.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.