COVID-19 Academic Impact in Indiana Damian Betebenner & Adam Van Iwaarden Center for Assessment Indiana TAC Meeting January 27th, 2022

Update on Academic Impact Analysis and Report • We last met in late October and discussed findings. • The July Board of Education presentation indicated overall academic impact was moderate/large in ELA and large in mathematics. • Both status-based analyses as well as growth-based analyses support these findings. • Since then we have finalized results associated with: • ILEARN spring 2021 summative assessment • WIDA-ACCESS 2021 summative assessment • Today we want to share results associated with the analyses as well as some supplementary results requested by the TAC during the previous meeting. Pandemic Related Academic Impact in Indiana 2

How do we quantify academic impact using test data • The pandemic was like a “headwind” that (presumably) slowed down the learning of students due to a variety of factors. • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth • NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical growth to compare to. • Next best way: Look at how much 2021 status changed: • ILEARN: Use 2019 status (or some adjustment thereof) as a proxy. • ACCESS: Use 2020 status (or some adjustment thereof) as a proxy. • Adjustments include (potentially) imputation for missing data as well as propensity score matching for • NOTE: When providing 2021 data, users will (correctly) want to compare their data to 2019 (or 2020 with ACCESS). The task is to make sure that such a comparison is an apples-to-apples comparison. Pandemic Related Academic Impact in Indiana 3

How do we quantify academic impact using test data • Historical consecutive and skip-year growth norms were created for ILEARN and ACCESS in May, 2021 for use with 2021 ILEARN and ACCESS data. • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • ILEARN: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth • NWEA-MAP: Provides growth data based upon historical norms but we currently don’t have historical growth to compare to. • Next best way: Look at how much 2021 status changed: • ILEARN: Use 2019 status (or some adjustment thereof) as a proxy. • ACCESS: Use 2020 status (or some adjustment thereof) as a proxy. • Adjustments include imputation for missing data as well as propensity score matching for changes in demographic characteristics of the population. • Indiana participation rates were normal, with participation exceeding 95% in all grades and content areas. Pandemic Related Academic Impact in Indiana 4

ILEARN Mathematics Grade 7 Indiana COVID⌧19 Academic Impact All Students Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary 2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score Academic Impact: Overall and by Quintile Grade 7 Scaled/Standardized Score and Achievement Level Cuts 2017 to 2019 Mean Conditional Status 2019 to 2021 Mean Conditional Status (Observed) 2.5 2019 Percentiles (Observed) 2021 Percentiles (Observed) Overall Observed + Imputed 2019 to 2021 Mean Conditional Status (+ Imputed) 2019 to 2021 Academic Impact 2021 Percentiles (+ Imputed) 2019 to 2021 Academic Impact High 2019 and 2021 Mean Scaled Scores Observed 2021 Grade 7 Scaled Score Decile Cuts Moderate High High High High 2021 Grade 7 Scaled Score Participation 60 60 0 0 Total Status Growth 2019: 41.4% Proficient 2021: 30.5% Proficient 2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Deciles (2019 Norm) 2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled/Standardized Scores by Quintile Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 7 Students 50 50 50 50 50 50 2019 Grade 7 Students 6395 (0.00) 6486 (0.00) 6540 (0.00) 6591 (0.00) 6667 (0.00) 6536 (0.00) 2021 Grade 7 Students 37 33 28 27 34 32 2021 Grade 7 Students 6377 (−0.23) 6463 (−0.23) 6514 (−0.23) 6564 (−0.23) 6647 (−0.23) 6513 (−0.23) Academic Impact (Observed) −13.0 −17.0 −22.0 −23.0 −16.0 −18.0 Decrease (Observed) −17.6 (−0.18) −23.9 (−0.24) −25.8 (−0.26) −27.1 (−0.28) −19.7 (−0.20) −23 (0.23) Academic Impact (+ Imputed) −13.0 −17.0 −22.0 −23.0 −16.0 −18.0 Decrease (+ Imputed) −17.8 (−0.18) −24.1 (−0.25) −26.0 (−0.27) −27.4 (−0.28) −19.8 (−0.20) −23 (0.24) Count Observed 2019 15,509 15,848 15,723 15,393 16,289 78,762 Count Observed 2019 17,122 16,722 17,128 16,693 16,913 84,578 Count Observed 2021 15,372 15,914 15,720 15,739 15,816 78,561 Count Observed 2021 16,749 16,626 16,391 16,561 16,556 82,883 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 2021 to 2019 Grade 7 Status Change (Observed) 30 2021 to 2019 SGP Difference (+ Imputed) 30 0.30 2021 to 2019 Grade 7 Status Change (+ Imputed) 0.8: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 0.69: Correlation SGP and Conditional Status Difference 2019 to 2021 2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Rank and Scaled Scores (2019 Norm)

Qualifying Academic Impact To help summarize the vast quantity of results for stakeholders, we created categories of academic impact based upon both effect size standards. Pandemic Related Academic Impact in Indiana 6

Review of results: ILEARN • Academic impacts in mathematics were larger than those in ELA, though impacts were large in both content areas. • Academic impacts were consistent in size across grades. • Across demographic subgroups, academic impacts were fairly uniform. Pandemic Related Academic Impact in Indiana 7

Review of results: WIDA-ACCESS • For English Language Learners taking the WIDA-ACCESS English • Academic impacts were large for students in the elementary grades. • Academic impacts were moderate for students in middle school grades. • Academic impacts were modest to none for students in the high school grades. Pandemic Related Academic Impact in Indiana 8

Indiana COVID⌧19 Academic Impact Overview ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate High High High High High Free/reduced Lunch Students High Moderate Moderate Moderate Moderate Moderate High High High High High High Non Free/reduced Lunch Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate High High High High Asian Students Moderate Moderate Moderate Moderate Moderate Modest to None Moderate Moderate High Severe Severe High African American Students High High Moderate High Moderate Moderate Severe Severe Severe Severe High High Hispanic Students High Moderate High High Moderate Moderate High High High Severe High High White Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate High High High High ELL Students High Moderate High High Moderate High High High High High Moderate High Special Education Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate High Moderate Moderate Moderate Moderate Female Students Moderate Moderate Moderate Moderate Moderate Moderate High High High Severe High High Male Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate High High High High

Indiana COVID⌧19 Academic Impact Overview ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Free/reduced Lunch Students Non Free/reduced Lunch Students Asian Students African American Students Hispanic Students White Students ELL Students Special Education Students Female Students Male Students

WIDA IN COVID⌧19 Academic Impact Overview Reading Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate High Moderate High High Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement

WIDA IN COVID⌧19 Academic Impact Overview Reading Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students

Questions Arizona State Board of Education Meeting, October 25th, 2021

About this copy

COVID-19 Academic Impact in Indiana — Indiana Technical Advisory Committee, January 27, 2022.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.