Pandemic Related Academic Impact in 2021 What do state summative assessments tell us Damian Betebenner & Adam Van Iwaarden Center for Assessment NAGB Dissemination & Reporting Meeting September 21st, 2021

Pandemic Related Academic Impact • As we are all aware, the pandemic greatly disrupted the lives of people worldwide over the last 18 months. • The impact on education was dramatic with millions of students receiving education in new ways. • A concern of many is to what extent the pandemic impacted students: Academically, emotionally, physically. • The Center has been working with a dozen states to investigate the extent to which the pandemic has impacted student learning. • As part of that effort, we’ve produced some resources that states can use to help understand academic impact. • Early Academic Impact Findings from Spring 2021 Assessments • Understanding Pandemic Learning Loss and Learning Recovery NAGB Dissemination and Reporting Meeting

Pandemic Related Academic Impact • The pandemic and all the ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two impediments manifest themselves (roughly) with assessments as: • Decrease in student growth (decrease in learning velocity). • Decrease in student attainment (decrease in amount learned). • In some states, understanding this phenomenon requires special consideration of low participation/changing demographics. NAGB Dissemination and Reporting Meeting

Pandemic related academic impact • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The natural impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • Supporting these comparisons requires: • Doing so in a technically rigorous manner. • Communicating the results/comparisons clearly so non-technical audiences get the main message. This is the hardest part. NAGB Dissemination and Reporting Meeting

Data to assess academic impact • During the 2020-2021 academic year, student achievement data from achievement vendors (primarily Renaissance Star, NWEA MAP, and Curriculum Associates iReady) was all that was available to investigate pandemic related academic impact. • States summative assessments (cancelled in 2020) were broadly administered nationwide in 2021. • Participation rates ranged from normal (> 95%) to poor. • Given their technical quality, alignment with state standards, coverage of the state student population, and long-term history, for nearly all states, 2021 state summative assessment data provides excellent data on which to begin investigating academic impact. • Many states have ELP assessments from 2020 and 2021 with which to investigate academic impact on the ELL subgroup. NAGB Dissemination and Reporting Meeting

Analyses for 2020-2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • English Language Learner impact can be evaluated using WIDA-ACCESS results. NAGB Dissemination and Reporting Meeting

Student attainment • Across the dozen states we’ve examined thus far: • Decreases in percent proficient are most often reported (though percent proficient can mask changes in the distribution) • Decreases in ELA generally in the upper single digits but sometimes in the teens. • Decreases in Math generally in the lower teens. • In terms of mean scale score decreases, effect sizes are large ranging from 0.2 to 0.4. • These are large effect sizes, especially when seen at the state level. NAGB Dissemination and Reporting Meeting

Student growth • States we’ve worked with utilize SGPs to measure growth. • States created baseline, skip-year growth norms using pre-pandemic student data. • By grade/content area median SGPs: • ELA between low 40s to low 30s. • Mathematics between mid 30s to low 20s. • Academic impact by grade is relatively uniform. This differs from interim assessment results which show nominal impact at middle school levels. • These decreases in student learning (i.e., growth) corroborate losses seen in attainment. • Decreases of this magnitude in student learning are unprecedented and will likely require unprecedented support and effort going forward to overcome. • Multiple year time frame • Substantial support beyond just getting kids back into the classroom. NAGB Dissemination and Reporting Meeting

Missing Data/Changing Demographics • The issue of lower than usual test participation in states has led to much debate on whether state assessment results are of use this year. • For all state analyses we have conducted we also performed multiple imputation analysis on the missing data and examined changes in student population beyond missing data (demographic shifts). • In all instances at the state level and almost all instances at the district and school level, imputation suggests results would have been worse had all students tested. • Because the vast majority of academic impact results examined are bad, what we are unsure about is just how bad (i.e., bad or really bad?) • To use an analogy, we know the patient is seriously sick with cancer, we don’t know whether it is stage IIIA, IIIB, or IIIC. NAGB Dissemination and Reporting Meeting

Demographic/Academic Subgroup Impact • By ethnicity, academic impacts are, in general, relatively uniform. • Asian students showed slightly larger impacts in mathematics • ELL status on state summative show fairly uniform academic impact. • On ELP assessments, larger impact in lower grades than in middle/ upper school grades. • Special education/IEP status in some states showed lesser impact that their non-special education counterparts • In middle school mathematics, several states show higher achievers demonstrating larger impacts than low achievers. NAGB Dissemination and Reporting Meeting

Demographic/Academic Subgroup Impact • States collecting data of mode of education during the pandemic (e.g., in-person, hybrid, remote) show more impact for students receiving remote instruction versus in-person instruction. • Given these findings many states have prioritized efforts to get students back to in-person learning as quickly and safely as possible. NAGB Dissemination and Reporting Meeting

Schools and Districts • Obviously, when overall state academic impacts are large, the average impact seen by schools and districts is also large. • However, there is wide variability in the impact on schools and districts. • For many, the impacts were so large we felt it necessary to add the category “severe” to characterize the impact seen. • In a small minority of cases schools showed modest to no impact or even some improvement. NAGB Dissemination and Reporting Meeting

Looking forward to 2021-2022 • As we (hopefully) transition from pandemic to recovery, data from 2021-2022 will provide an excellent basis to understand whether what type of recovery systems are demonstrating: • V-shaped, U-Shaped, L-Shaped, or no recovery • States with compromised 2021 assessments (low participation, fall testing, no testing) will have tradeoffs to make in terms of using 2021-2022 data to assess recovery. • Using data for these purposes is not the typical use of data for “accountability” and requires states to promote results in ways not related to accountability. This can be a delicate balancing act. • Overall, I’m optimistic about the use of 2020-2021 data for academic impact as well as to inform recovery in 2021-2022. NAGB Dissemination and Reporting Meeting

Questions NAGB Dissemination and Reporting Meeting

About this copy

Pandemic Related Academic Impact in 2021: What do state summative assessments tell us — NAGB, November 4, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.