Pandemic Related Academic Impact in 2021 What do state summative assessments tell us Damian Betebenner & Adam Van Iwaarden Center for Assessment WERA Professional Development Session Series October 12th, 2021

The Presentation is on Tuesday, Oct. 12

Pandemic Related Academic Impact • As we are all aware, the pandemic greatly disrupted the lives of people worldwide over the last 18 months. • The impact on education was dramatic with millions of students receiving education in new ways. • A concern of many is to what extent the pandemic impacted students: Academically, emotionally, physically. • The Center has been working with a dozen states to investigate the extent to which the pandemic has impacted student learning. • My colleagues and I wrote a blog detailing some early findings: https://www.nciea.org/blog/covid-19-disruptions/assessing- academic-impact-covid-19-summer-2021 WERA Profession Develment Session Series

Pandemic Related Academic Impact • The pandemic and all the ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two outcomes represent themselves as: • Decrease in student growth. • Decrease in student attainment. • Special consideration must be given to accommodate low participation/ changing demographics (multiple imputation and propensity score matching) WERA Profession Develment Session Series

Pandemic related academic impact • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • Supporting these comparisons requires having a complete understanding which individuals constitute the missing data. WERA Profession Develment Session Series

Data to assess academic impact • During the 2020-2021 academic year, student achievement data from achievement vendors (primarily Renaissance Star, NWEA MAP, and Curriculum Associates iReady) was all that was available to investigate pandemic related academic impact. • States summative assessments (cancelled in 2020) were broadly administered nationwide in 2021. • Participation rates ranged from normal (> 95%) to poor. • Given their technical quality, alignment with state standards, coverage of the state student population, and long-term history, for nearly all states, 2021 state summative assessment data provides excellent data on which to begin investigating academic impact. • Many states have ELP assessments from 2020 as 2021 with which to investigate academic impact on the ELL subgroup. WERA Profession Develment Session Series

Addressing test score validity:
In our district, I noticed results of Fall remote assessments for K-1 test takers were heavily inflated (students were 6-7 months ahead of the previous year’s “normal” cohort. After surveying the field, we found that this must have been due to parent assistance with the assessments despite instruction given to parents to not assist their child in any way during testing.
This provided a challenge to measure growth for year since by the time the same students took the assessment in the Spring, they were back in school and taking the test in their classrooms (without their parent’s help) which could paint a picture of diminished, if not, negative growth for year.

I mention this because folks may have questions about how you may have controlled for this if you used remote assessment data to help measure growth.

Analyses for 2020-2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • English Language Learner impact can be evaluated using WIDA-ACCESS results. WERA Profession Develment Session Series

Are there any visuals that would bring the most interesting findings to a clearer light? I really liked the idea of comparing the pandemic’s impact on learning to that of the impact hurricane Katrina had on the students living the communities it hit.

Student attainment • Across the dozen states we’ve examined thus far: • Decreases in percent proficient are most often reported (though percent proficient can mask changes in the distribution) • Decreases in ELA generally in the upper single digits but sometimes in the teens. • Decreases in Math generally in the lower teens. • In terms of mean scale score decreases, effect sizes are large ranging from 0.2 to 0.4. • These are large effect sizes, especially when seen at the state level. WERA Profession Develment Session Series

Student growth • States we’ve worked with utilize SGPs to measure growth. • States created baseline growth norms using pre-pandemic student data. • By grade/content area median SGPs in ELA between low 40s to low 30s. In Mathematics between mid 30s to low 20s. • Academic impact by grade is relatively uniform. • These decreases in student learning (i.e., growth) corroborate losses seen in attainment. • Decreases of this magnitude in student learning will likely require multiple years WERA Profession Develment Session Series

Demographic/Academic Subgroup Impact • By ethnicity, academic impacts are, in general, relatively uniform. • Asian students showed slightly larger impacts in mathematics • ELL status on state summative show fairly uniform academic impact. • On ELP assessments, larger impact in lower grades than in middle/ upper school grades. • Special education/IEP status in some states showed lesser impact that their non-special education counterparts • In middle school mathematics, several states show higher achievers demonstrating larger impacts than low achievers. WERA Profession Develment Session Series

Demographic/Academic Subgroup Impact • States collecting data of mode of education during the pandemic (e.g., in-person, hybrid, remote) show more impact for students receiving remote instruction versus in-person instruction. • Several states have found very noticeable differential impact by prior achievement. • Given these findings many states have prioritized efforts to get students back to in-person learning as quickly and safely as possible. WERA Profession Develment Session Series

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 ELA New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 ELA New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 ELA New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 ELA New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 Mathematics New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 Mathematics New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 Mathematics New Meridian SGP Calculations 2021

Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 Mathematics New Meridian SGP Calculations 2021

Academic Impact to Schools • Not surprisingly, there is wide variability in terms of the academic impact to schools. • The following bubble plots show academic impact to schools based upon school performance (status and growth) in 2019. • It is a challenge to help people to not confuse status with academic impact. • Many high and low achieving schools (and high and low growth schools) were severely impacted by the pandemic. WERA Profession Develment Session Series

School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 3.1% Higher Growth School Size 50 Students 100 Students 9.9% 4.8% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 11.4% Higher Growth School Size 50 Students 100 Students 22.4% 19.7% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 29.3% Higher Growth School Size 50 Students 100 Students 32.9% 35.9% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 38.7% Higher Growth School Size 50 Students 100 Students 28.9% 28.5% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 17.5% Higher Growth School Size 50 Students 100 Students 5.9% 11.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 3.4% Higher Growth School Size 50 Students 100 Students 7.8% 6.1% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 7.9% Higher Growth School Size 50 Students 100 Students 16.4% 16.3% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 20.2% Higher Growth School Size 50 Students 100 Students 35.9% 32.1% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 24.8% Higher Growth School Size 50 Students 100 Students 24.2% 19.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 43.7% Higher Growth School Size 50 Students 100 Students 15.6% 26.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series

Looking forward to 2021-2022 • As we (hopefully) transition from pandemic to recovery, data from 2021-2022 will provide an excellent basis to understand whether what type of recovery systems are demonstrating: • V-shaped, U-Shaped, L-Shaped, or no recovery • States with compromised 2021 assessments (low participation, fall testing, no testing) will have tradeoffs to make in terms of using 2021-2022 data to assess recovery. • Using data for these purposes is not the typical use of data for “accountability” and requires states to promote results in ways not related to accountability. This can be a delicate balancing act. • Overall, I’m optimistic about the use of 2020-2021 data for academic impact as well as to inform recovery in 2021-2022. WERA Profession Develment Session Series

Is there anything you noticed in the data specific for WA State that is different from the other states in your analysis? What differences? And should districts in our state be focusing on remediation of particular subjects/grade levels over others? For example, should districts in WA state invest more resources to catch students up in middle school math or elementary reading? It would be nice to see some kind of ranking of the grade levels with subject areas that appear to have been negatively impacted the most.

In my district, we are seeing the lowest scores (and maturity levels) of our 1st and 2nd graders, who have not had a lot of support at home.

Questions WERA Profession Develment Session Series

About this copy

Pandemic Related Academic Impact in 2021: What do state summative assessments tell us — Washington Educational Research Association, October 11, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.

• Talk page
• Download PDF
• Original file
• All talks