COVID-19 Academic Impact in Illinois

Illinois Department of Education · July 28, 2021

COVID-19 Academic Impact in Illinois — Illinois Department of Education, July 28, 2021.
Published

July 28, 2021

Event
State agency
Slides
15
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. COVID-19 Academic Impact in Illinois Damian Betebenner & Adam Van Iwaarden Center for Assessment July 26, 2021
  2. How do we quantify academic impact using test data • The pandemic was like a “headwind” that (presumably) slowed down the learning of students due to a variety of factors. • Headwinds impact progress in two ways. • They decrease the rate of progress (i.e., decrease student growth). • They decrease the distance travelled (i.e., decrease attainment). • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • IAR: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • Next best way: Look at how much 2021 status changed: • ILEARN: Use 2019 status (or some adjustment thereof) as a proxy. • NOTE: When providing 2021 data, users will (correctly) want to compare their data to 2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an apples-to-apples comparison. Pandemic Related Academic Impact in Illinois 2
  3. Results • Overall academic impact was large in both ELA and mathematics. • Recovery associated with large impacts will likely take multiple years and require additional supports beyond just normal schooling (right-size the intervention) • Both status-based analyses as well as growth-based analyses support these findings. • The following tables illustrate numerically what these impacts were. • For rough comparison, the effect size impacts we are seeing for achievement are roughly 1 to 5 times as large as what were reported for Hurricane Katrina in Louisiana depending upon grade and content area. (Effect sizes of -0.06 and -0.17 are associated with Katrina). • Groups experiencing minimal impact should be scrutinized closely with regard to pre-pandemic level of educational efficacy. Pandemic Related Academic Impact in Illinois 3
  4. IAR Attainment: 2019 to 2021 Percent Proficient ELA Mathematics Grade Percent Percent Difference Percent Percent Difference Proficient Proficient 2019 to 2021 Proficient Proficient 2019 to 2021 Pandemic Related Academic Impact in Illinois 4
  5. IAR Growth: 2019 to 2021 Student Growth ELA Mathematics Grade Median SGP Median SGP Difference 2019 Median SGP Median SGP Difference 2019 2019 2021 to 2021 2019 2021 to 2021 Pandemic Related Academic Impact in Illinois 5
  6. Performance by initial level of achievement • We are beginning to look at impact by demographic subgroup (coming shortly) as well as impact by academic subgroup. • We examined academic impact by the prior academic achievement (low versus high achieving using prior scale score decile). • To dig deeper we regularly look at SGP distributions by prior score decile. For cohort referenced SGPs we expect a uniform distribution. For baseline referenced SGPs, deviations from uniform indicate impact. Pandemic Related Academic Impact in Illinois 6
  7. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 ELA SGP, Grade 5 (N = 90,180) SGP Deciles by Prior Achievement Level (ELA) Level 5 (3.6%) Prior Achievement Level Level 4 (35.4%) Level 3 (23.2%) Level 2 (18%) Level 1 (19.8%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 46 Extremes (Range) Highest OSS: [3.338,3.476) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 17.4 82.6 N = 23 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.442) 14.89 16.03 11.99 9.43 11.15 7.91 7.02 8.30 6.61 6.66 (9.8%) 100 2nd/[−1.442,−0.893) 17.41 15.54 12.40 8.84 10.34 8.04 7.77 6.88 6.48 6.29 (10%) Prior Scale Score Decile/Range * 3rd/[−0.893,−0.5659) 19.48 14.76 11.44 10.84 8.47 8.02 7.64 6.79 6.69 5.88 (9.1%) Empirical SGP Distribution 4th/[−0.5659,−0.2032) 21.34 14.39 11.01 9.58 9.07 7.89 7.51 6.94 6.73 5.54 (10.8%) 5th/[−0.2032,0.0923) 21.32 14.89 10.89 9.28 8.30 9.86 6.16 7.41 6.22 5.66 (9.6%) 6th/[0.0923,0.3751) 20.53 13.14 12.72 7.91 8.19 9.67 7.99 7.36 6.79 5.71 (10.5%) 7th/[0.3751,0.6523) 22.91 14.15 11.50 9.01 8.75 8.36 6.88 7.28 5.92 5.26 (9.6%) 8th/[0.6523,1.026) 23.30 13.66 11.80 8.82 10.42 7.46 6.43 7.43 5.24 5.44 (10.2%) 9th/[1.026,1.443) 23.94 13.65 12.25 9.47 10.13 7.32 6.33 6.25 5.30 5.36 (10.2%) 10th/[1.443,15] 29.20 15.34 12.26 8.70 8.11 6.47 5.45 4.82 5.19 4.45 (10.3%) *Prior score deciles can be uneven depending upon the prior score distribution
  8. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 ELA SGP, Grade 6 (N = 89,764) SGP Deciles by Prior Achievement Level (ELA) Level 5 (8.7%) Prior Achievement Level Level 4 (31.9%) Level 3 (26%) Level 2 (18.5%) Level 1 (14.9%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 65 Extremes (Range) Highest OSS: [3.349,3.536) 0.0 0.0 4.3 0.0 0.0 0.0 0.0 4.3 8.7 4.3 21.7 56.5 N = 23 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.652) 12.80 10.85 11.42 10.42 10.22 9.62 8.95 8.65 8.54 8.55 (9.9%) 100 2nd/[−1.652,−1.034) 14.53 11.94 11.46 11.10 9.29 9.59 9.04 8.42 7.88 6.76 (9.7%) Prior Scale Score Decile/Range * 3rd/[−1.034,−0.6413) 15.36 12.85 11.03 10.40 9.52 9.80 8.71 8.47 7.32 6.54 (9.8%) Empirical SGP Distribution 4th/[−0.6413,−0.2949) 15.58 13.81 11.96 11.11 9.64 9.44 8.44 7.18 6.70 6.15 (10.6%) 5th/[−0.2949,0.0302) 15.76 14.29 12.67 10.78 10.14 8.92 8.14 7.28 6.66 5.36 (9.7%) 6th/[0.0302,0.2857) 17.74 13.79 12.05 10.51 9.88 9.01 8.08 6.98 6.14 5.81 (9.8%) 7th/[0.2857,0.588) 17.84 14.14 11.66 10.48 9.88 8.72 8.25 6.83 6.37 5.82 (9.2%) 8th/[0.588,0.9404) 19.88 13.89 12.34 10.25 9.03 8.36 8.09 7.19 6.11 4.86 (10.8%) 9th/[0.9404,1.461) 22.00 15.13 12.55 9.91 8.91 7.86 7.79 6.55 5.05 4.25 (10.5%) 10th/[1.461,15] 24.18 15.32 12.04 9.61 9.38 7.88 6.79 5.77 5.55 3.47 (10%) *Prior score deciles can be uneven depending upon the prior score distribution
  11. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 Mathematics SGP, Grade 5 (N = 89,200) SGP Deciles by Prior Achievement Level (Mathematics) Level 5 (8.7%) Prior Achievement Level Level 4 (34.6%) Level 3 (25%) Level 2 (19.7%) Level 1 (12%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [−15,−4.44] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 48 Extremes (Range) Highest OSS: [3.765,4.606) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.5 7.5 90.0 N = 40 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.536) 19.60 14.12 13.36 9.39 10.16 7.64 7.87 7.09 6.10 4.67 (9.7%) 100 2nd/[−1.536,−1.065) 20.13 13.97 12.86 11.41 8.99 8.80 8.03 5.97 5.11 4.71 (9.3%) Prior Scale Score Decile/Range * 3rd/[−1.065,−0.6694) 22.57 16.70 11.73 10.83 9.09 7.35 7.13 5.20 4.79 4.62 (10.5%) Empirical SGP Distribution 4th/[−0.6694,−0.3653) 24.21 15.13 12.06 10.18 8.74 7.65 6.61 5.44 4.87 5.11 (10.1%) 5th/[−0.3653,−0.0344) 24.34 15.12 11.83 9.49 9.19 6.98 5.77 7.01 4.63 5.65 (10.2%) 6th/[−0.0344,0.2259) 24.38 15.32 11.81 8.89 8.28 7.28 6.60 6.28 5.52 5.64 (9.8%) 7th/[0.2259,0.5214) 25.20 14.52 11.19 8.84 8.15 7.00 6.65 6.61 5.86 5.98 (9.8%) 8th/[0.5214,0.8636) 24.61 13.63 11.29 8.24 7.71 8.14 7.49 6.17 6.27 6.44 (10.5%) 9th/[0.8636,1.291) 22.94 13.52 10.62 9.09 8.27 8.49 6.91 6.79 6.60 6.78 (9.6%) 10th/[1.291,15] 19.06 12.69 9.57 9.05 8.90 9.41 6.51 7.60 7.60 9.61 (10.5%) *Prior score deciles can be uneven depending upon the prior score distribution
  12. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 Mathematics SGP, Grade 6 (N = 88,633) SGP Deciles by Prior Achievement Level (Mathematics) Level 5 (3.6%) Prior Achievement Level Level 4 (34%) Level 3 (29%) Level 2 (19.6%) Level 1 (13.8%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 200 Extremes (Range) Highest OSS: [3.384,4.065) 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.0 0.0 14.8 18.5 63.0 N = 27 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.643) 10.92 12.91 12.90 12.44 11.67 10.19 9.89 8.58 5.99 4.51 (8.8%) 100 2nd/[−1.643,−1.056) 15.59 14.99 14.19 12.24 10.60 9.10 8.04 6.30 5.16 3.79 (10.6%) Prior Scale Score Decile/Range * 3rd/[−1.056,−0.7248) 20.49 16.96 13.04 11.62 9.05 7.72 6.85 5.54 4.64 4.08 (9.7%) Empirical SGP Distribution 4th/[−0.7248,−0.3645) 23.90 16.74 12.94 10.31 8.86 7.47 6.35 4.98 4.32 4.13 (9.8%) 5th/[−0.3645,−0.0413) 24.09 16.21 13.12 9.81 8.81 7.47 6.55 5.32 4.57 4.05 (10.8%) 6th/[−0.0413,0.1864) 23.40 15.60 12.95 10.12 8.51 7.81 6.91 5.66 4.92 4.13 (9%) 7th/[0.1864,0.5255) 23.43 15.34 11.62 10.33 9.09 7.21 6.97 5.93 5.33 4.76 (10.8%) 8th/[0.5255,0.8381) 24.13 14.89 11.92 9.85 8.79 6.87 6.96 5.91 5.31 5.38 (9.6%) 9th/[0.8381,1.321) 23.33 14.64 11.27 10.49 8.87 7.32 6.77 6.27 5.81 5.22 (10.7%) 10th/[1.321,15] 20.16 13.51 11.22 9.86 8.91 9.25 7.58 7.27 6.19 6.06 (10.1%) *Prior score deciles can be uneven depending upon the prior score distribution
  13. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 Mathematics SGP, Grade 7 (N = 87,285) SGP Deciles by Prior Achievement Level (Mathematics) Level 5 (5.7%) Prior Achievement Level Level 4 (26.6%) Level 3 (28.8%) Level 2 (27.6%) Level 1 (11.3%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [−15,−3.83] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 48 Extremes (Range) Highest OSS: [4.134,4.825) 0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.3 8.3 29.2 12.5 41.7 N = 24 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.542) 9.38 11.29 11.09 11.81 11.40 11.10 10.10 8.99 7.69 7.15 (9.7%) 100 2nd/[−1.542,−1.139) 11.60 13.33 12.65 11.45 11.50 10.49 9.47 7.92 6.46 5.13 (9.3%) Prior Scale Score Decile/Range * 3rd/[−1.139,−0.8564) 13.75 14.50 13.06 11.79 10.66 9.02 8.56 7.53 6.16 4.97 (9.3%) Empirical SGP Distribution 4th/[−0.8564,−0.5544) 15.16 14.51 13.06 11.18 10.52 8.97 8.45 6.86 6.42 4.86 (10.8%) 5th/[−0.5544,−0.3049) 16.92 15.20 12.23 10.20 9.78 8.39 8.58 7.56 6.17 4.97 (9.6%) 6th/[−0.3049,0.017) 16.92 14.55 12.41 10.63 10.06 9.12 7.92 7.56 6.35 4.49 (10.1%) 7th/[0.017,0.3388) 18.54 14.81 11.71 11.44 9.89 8.49 8.36 7.10 5.67 3.98 (10.1%) 8th/[0.3388,0.749) 20.80 15.85 12.44 11.18 9.57 8.28 7.42 6.30 4.74 3.41 (10.5%) 9th/[0.749,1.286) 22.75 17.28 13.02 10.40 8.91 7.67 6.57 5.05 4.69 3.68 (10.6%) 10th/[1.286,15] 25.61 15.12 12.04 9.78 9.52 8.11 5.92 5.77 4.67 3.46 (10%) *Prior score deciles can be uneven depending upon the prior score distribution
  14. Student Growth Percentile Goodness−of−Fit Descriptives 2020−2021.2 Mathematics SGP, Grade 8 (N = 87,180) SGP Deciles by Prior Achievement Level (Mathematics) Level 5 (3.4%) Level 4 (23.6%) Prior Achievement Level Level 3 (30.1%) Level 2 (28.8%) Level 1 (14%) Median Student Growth Percentile Ceiling / Floor Effects Test Student Growth Percentile Range Current Observed Score 5 9 to to to to to to to to to to to to 10 20 30 40 50 60 70 80 90 95 Lowest OSS: [ −15 ] 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N = 70 Extremes (Range) Highest OSS: [4.086,4.104) 0.0 0.0 0.0 0.0 0.0 4.5 0.0 4.5 0.0 36.4 18.2 36.4 N = 22 Student Growth Percentile Range QQ−Plot: Student Growth Percentiles 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 9 to to to to to to to to to to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1st/[−15,−1.622) 11.57 13.53 11.79 11.50 10.78 10.70 8.75 8.09 6.85 6.44 (9.7%) 100 2nd/[−1.622,−1.171) 13.35 13.98 13.05 11.04 10.22 9.79 9.02 7.47 6.66 5.42 (9.2%) Prior Scale Score Decile/Range * 3rd/[−1.171,−0.8296) 14.24 14.83 12.13 11.82 10.30 9.73 8.27 7.27 6.39 5.03 (9.9%) Empirical SGP Distribution 4th/[−0.8296,−0.5464) 15.95 15.21 12.25 11.22 10.47 9.53 7.96 7.11 5.84 4.46 (9.7%) 5th/[−0.5464,−0.297) 17.85 14.45 13.26 10.78 9.92 8.51 8.09 6.48 5.89 4.75 (10.7%) 6th/[−0.297,0.0185) 17.91 15.31 13.11 11.17 9.72 8.50 7.32 6.82 5.84 4.32 (10.8%) 7th/[0.0185,0.2955) 19.02 15.38 12.67 10.76 9.74 8.81 7.15 6.15 5.33 5.00 (9.5%) 8th/[0.2955,0.6893) 21.81 14.32 12.70 10.12 9.25 8.30 7.20 6.30 5.21 4.81 (10.1%) 9th/[0.6893,1.154) 20.13 15.61 12.88 10.90 8.67 8.20 7.26 6.13 5.67 4.55 (10%) 10th/[1.154,15] 19.25 14.31 11.78 10.33 8.55 8.38 8.20 7.20 6.99 5.01 (10.4%) *Prior score deciles can be uneven depending upon the prior score distribution
  15. School level impact • We are currently in the process of finalizing school level impact analyses and want to share some of the initial looks at these results. • To calculate impact at the school level we calculate for each school: Unadjusted Impact = Median SGP2019- Median SGP2021 • We adjust for regression to the mean for this difference score by using pre-pandemic (2017 to 2019) school level median SGP scores. • Results forthcoming shortly Pandemic Related Academic Impact in Illinois 7