Report on COVID-19 Academic Impact in Indiana An Overview Damian Betebenner, Adam Van Iwaarden & Nathan Dadey Center for Assessment Indiana Assessment Technical Advisory Committee May 13, 2021

Report on pandemic related academic impact in Indiana • Early in 2021 Indiana passed legislation (Public Law 211-2021 added IC 20-26-5-40.6) requiring: • The DOE shall, in collaboration with the state board, conduct a learning loss study regarding students in kindergarten through grade 10 for the 2020-2021 school year and the 2021-2022 school year. • The DOE contracted with the Center for Assessment to conduct the study. • The purpose of the report is to understand the academic impact of student in Indiana due to the pandemic, NOT accountability. Report on COVID-19 Academic Impact in Indiana

Report on pandemic related academic impact in Indiana • Legislation requires the report address several questions including: • Which specific content areas have significant learning loss for students in kindergarten through grade 10. • What are the overall student learning loss or gaps in education in Indiana due to the disruption in student education caused by the coronavirus disease (COVID-19). • What learning loss differences exist for specific subgroups, including students with disabilities and English language learners. Report on COVID-19 Academic Impact in Indiana

Research Questions • Question #1: Who are the students in Indiana? • Overall description of student population. • Overall description of students for whom assessment data has been collected. Who do we not have data? • Question #2: To what extent did COVID-19 disruptions impact student academic performance & learning outcomes? • Overall academic impact • Disparities in academic impact • Content area academic impact (E/La & math; English language proficiency) • Question #3: What is the anticipated academic recovery time based on the experienced COVID-19 disruptions? Report on COVID-19 Academic Impact in Indiana

Research Questions – Areas of Consideration • Student racial/ethnic subgroup • Special education status • English learner status • Poverty status • Community type (urban, suburban, rural) • School type (traditional public, charter, nonpublic/choice) • Mode of instruction • Student attendance Report on COVID-19 Academic Impact in Indiana

Defining pandemic related academic impact • COVID-19 pandemic related academic impact is called by several names include “learning loss”, “unfinished learning”, “COVID slide”. • Throughout this presentation and report we will simply refer to it as pandemic related academic impact. • This is not to suggest or minimize other ways that students were impacted including social-emotional, physical, and financial. • Terminology associated with academic impact is contentious. Report on COVID-19 Academic Impact in Indiana

Defining pandemic related academic impact • For most students (based upon preliminary interim assessment data), the pandemic functioned as an “academic headwind”, showing down student progress. • Defining academic impact corresponds to the difference between where the student (or group of students) would have been (absent the pandemic) versus where they ended up being due to the pandemic. • The non-pandemic counterfactual is estimated in general, in two ways: • Status comparison between 2019 and 2021 • Growth comparison between 2019 and 2021 Report on COVID-19 Academic Impact in Indiana

Defining pandemic related academic impact Report on COVID-19 Academic Impact in Indiana

Four Academic Impact Scenarios with MCAS 2.0 Report on COVID-19 Academic Impact in Indiana

Data available for academic impact analyses • Indiana has an excellent set of data for 2021 analyses including. • ILEARN state summative assessment in grades 3 to 8 in ELA and Mathematics. No data available in 2020. • WIDA-ACCESS English Language Proficiency assessment data in grades 1 to 12. Data available in 2020. • NWEA MAP interim assessment data in grades 1 to 10 in English Literacy and Mathematics. Data available from Fall 2020-2021 and Winter 2020-2021. Report on COVID-19 Academic Impact in Indiana

Academic Impact Data Analysis Plans for 2021 • Understanding pandemic related impact requires looking back to pre- pandemic results (the counter-factual) • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • The first step toward supporting these comparisons is to have a complete understanding is who constitutes the missing data. Report on COVID-19 Academic Impact in Indiana

ILEARN Academic Impact Data Analysis Plans for 2021 • Status comparisons: • Compare 2021 results with 2019 results with the possibility of adjusting for missing data in 2021 (MNAR/ MAR/MCAR). • Compare 2021 results with 2019 results by reducing 2019 results to match with 2021 results (Andrew Ho’s Fair Trend analysis). • Status comparisons possible for all grades 3 to 8 in ELA and mathematics. • Status comparisons possible for aggregated subgroups including demographic subgroups, schools, and corporations. • Growth comparisons: • Calculate both cohort referenced (using 2021 growth norms) and baseline referenced (2019/pre-COVID growth norms) skip-year SGPs (2019 to 2021) • Note: ILEARN SGPs only available for Grades 5 to 8. • ILEARN: Compare mean/median SGP 2021 results with mean/median 2019 results with the possibility of adjusting for missing data (MNAR/MAR/MCAR). • Baseline referenced SGPs can be used to show overall impacts as well as differential impact. • Cohort referenced SGPs can be used to show only differential impact. Report on COVID-19 Academic Impact in Indiana

WIDA-ACCESS Academic Impact Data Analysis Plans for 2021 • NOTE: WIDA-ACCESS English Language Proficiency assessment was not canceled in spring 2020 (administered before the pandemic). • Status comparisons: • Compare 2021 results with 2020 results with the possibility of adjusting for missing data (MNAR/MAR/ MCAR). • Status comparisons possible for all grades 1 to 12 in ELA. • Status comparisons possible for aggregated subgroups including demographic subgroups, schools, and corporations. • Growth comparisons: • Calculate consecutive year (2020 to 2021) cohort referenced and baseline referenced SGPs (2019 to 2021) • Calculate skip-year SGPs as an aside investigation to show how well skip-year growth aligns with consecutive year growth (this will help inform ILEARN analyses as we only have skip-year growth and status comparisons available) • Compare mean/median 2021 SGP 2021s with mean/median 2020 results with the possibility of adjusting for missing data (MNAR/MAR/MCAR). • Baseline referenced SGPs can be used to show overall impacts as well as differential impact. • Cohort referenced SGPs can be used to show only differential impact. Report on COVID-19 Academic Impact in Indiana

NWEA MAP Academic Impact Data Analysis Plans for 2021 • NOTE: The NWEA MAP assessment is given to approximately 40% of all students in grades K to 8, and approximately 15% of students in grades 9 and 10. • Currently, Indiana DOE only has data for Fall 2020-2021 and Winter 2020-2021. This data includes both status information as well as growth (SGP) information. • Status comparisons: • Because we have no pre-COVID NWEA-MAP status information, we will be unable to calculate status- based impact analyses. • Indiana just signed an agreement with NWEA to receive historical Indiana data. • Growth comparisons: • Fall 2020-2021 data includes SGPs from Fall 2019-2020 to Fall 2020-2021 as well as Winter 2019-2020 to Fall 2020-2021. These SGPs are based upon historical norms (baseline) and thus can be used to investigate overall academic impact circa Fall 2020-2021. • Winter 2020-2021 data includes SGPs from Fall 2020-2021 to Winter 2020-2021 as well as Winter 2019-2020 to Winter 2020-2021. These SGPs are based upon historical norms and can be used to investigate overall academic impact (and recover) circa Winter 2020-2021. Report on COVID-19 Academic Impact in Indiana

SGP Analysis Plans for 2021 • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Mean/median will by 50 for all grade x content area analyses • Cannot investigate overall impact of the pandemic on students. • Can investigate differential impact of the pandemic on students. • Does not require scale stability across years. • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related academic impact. • Mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact and differential impact. Report on COVID-19 Academic Impact in Indiana

Growth and Status Analysis Plans for 2021 • An issue impacting the interpretation of growth and status results in 2021 is participation in the state assessment. • Low participation doesn’t impact calculation of cohort referenced growth norms nor use of baseline referenced growth norms. • Low participation does impact comparisons of group level results with historical (e.g., 2019) results based upon higher participation rates. • For example, how should one compare growth and status results for school in 2019 and 2021 with very different rates of participation. Report on COVID-19 Academic Impact in Indiana

Growth and Status Analysis Plans for 2021 • Understanding who is missing is key. • State education data is exemplary in terms of identifying who should test. • For example, we could create worst/best case scenarios by substituting LOSS/ HOSS values for missing students. • More precisely, we empirically examine whether missing data is MCAR, MAR, or MNAR. • Apply imputation procedures based upon findings to create inferred values. • Indiana’s high participation rates suggest that such inferential procedures will not be used frequently. • We are currently testing out various procedures and have strong evidence that inferences about group aggregates can be faithfully recovered (not for accountability). Report on COVID-19 Academic Impact in Indiana

Growth and Status Analysis Plans for 2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • Demographic subgroups including ethnicity, poverty • English Language Learner impact can be evaluated using WIDA-ACCESS results. Report on COVID-19 Academic Impact in Indiana

SGP Analysis Plans post 2021 (2022 and beyond) • Spring 2021 represents an ideal time to get a snapshot of student attainment as we are presumably at or near the point of maximal impact on student attainment. • That is, recovery has not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery! Report on COVID-19 Academic Impact in Indiana

Four Academic Recovery Scenarios with MCAS 2.0: V-, U-, L-, and No- recovery Report on COVID-19 Academic Impact in Indiana

Communication of Results in 2021 • Technical challenges to the calculation of growth in 2021 are likely minor compared to communication challenges. • State reporting is, in general, highly aligned with accountability. • Even though analyses outlined herein deal with status/growth, they don’t necessarily align with accountability narratives. • It is critical for leadership (at the highest level) to map out a communication plan and then build analytics and reporting to support that plan. • Without a well considered, disciplined and coordinated communication plan, accountability narratives are likely to take over. Report on COVID-19 Academic Impact in Indiana

Questions Report on COVID-19 Academic Impact in Indiana

About this copy

Report on COVID-19 Academic Impact in Indiana: An Overview — Indiana Department of Education, May 13, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.