Pandemic Related Academic Impact in 2021: What do state summative assessments tell us

Washington Educational Research Association · October 11, 2021

Pandemic Related Academic Impact in 2021: What do state summative assessments tell us — Washington Educational Research Association, October 11, 2021.
Published

October 11, 2021

Event
Conference
Slides
31
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Pandemic Related Academic Impact in 2021 What do state summative assessments tell us Damian Betebenner & Adam Van Iwaarden Center for Assessment WERA Professional Development Session Series October 12th, 2021
  2. Pandemic Related Academic Impact • As we are all aware, the pandemic greatly disrupted the lives of people worldwide over the last 18 months. • The impact on education was dramatic with millions of students receiving education in new ways. • A concern of many is to what extent the pandemic impacted students: Academically, emotionally, physically. • The Center has been working with a dozen states to investigate the extent to which the pandemic has impacted student learning. • My colleagues and I wrote a blog detailing some early findings: https://www.nciea.org/blog/covid-19-disruptions/assessing- academic-impact-covid-19-summer-2021 WERA Profession Develment Session Series
  3. Pandemic Related Academic Impact • The pandemic and all the ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two outcomes represent themselves as: • Decrease in student growth. • Decrease in student attainment. • Special consideration must be given to accommodate low participation/ changing demographics (multiple imputation and propensity score matching) WERA Profession Develment Session Series
  4. Pandemic related academic impact • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • Supporting these comparisons requires having a complete understanding which individuals constitute the missing data. WERA Profession Develment Session Series
  5. Data to assess academic impact • During the 2020-2021 academic year, student achievement data from achievement vendors (primarily Renaissance Star, NWEA MAP, and Curriculum Associates iReady) was all that was available to investigate pandemic related academic impact. • States summative assessments (cancelled in 2020) were broadly administered nationwide in 2021. • Participation rates ranged from normal (> 95%) to poor. • Given their technical quality, alignment with state standards, coverage of the state student population, and long-term history, for nearly all states, 2021 state summative assessment data provides excellent data on which to begin investigating academic impact. • Many states have ELP assessments from 2020 as 2021 with which to investigate academic impact on the ELL subgroup. WERA Profession Develment Session Series
  6. Analyses for 2020-2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • English Language Learner impact can be evaluated using WIDA-ACCESS results. WERA Profession Develment Session Series
  7. Student attainment • Across the dozen states we’ve examined thus far: • Decreases in percent proficient are most often reported (though percent proficient can mask changes in the distribution) • Decreases in ELA generally in the upper single digits but sometimes in the teens. • Decreases in Math generally in the lower teens. • In terms of mean scale score decreases, effect sizes are large ranging from 0.2 to 0.4. • These are large effect sizes, especially when seen at the state level. WERA Profession Develment Session Series
  8. Student growth • States we’ve worked with utilize SGPs to measure growth. • States created baseline growth norms using pre-pandemic student data. • By grade/content area median SGPs in ELA between low 40s to low 30s. In Mathematics between mid 30s to low 20s. • Academic impact by grade is relatively uniform. • These decreases in student learning (i.e., growth) corroborate losses seen in attainment. • Decreases of this magnitude in student learning will likely require multiple years WERA Profession Develment Session Series
  9. Demographic/Academic Subgroup Impact • By ethnicity, academic impacts are, in general, relatively uniform. • Asian students showed slightly larger impacts in mathematics • ELL status on state summative show fairly uniform academic impact. • On ELP assessments, larger impact in lower grades than in middle/ upper school grades. • Special education/IEP status in some states showed lesser impact that their non-special education counterparts • In middle school mathematics, several states show higher achievers demonstrating larger impacts than low achievers. WERA Profession Develment Session Series
  10. Demographic/Academic Subgroup Impact • States collecting data of mode of education during the pandemic (e.g., in-person, hybrid, remote) show more impact for students receiving remote instruction versus in-person instruction. • Several states have found very noticeable differential impact by prior achievement. • Given these findings many states have prioritized efforts to get students back to in-person learning as quickly and safely as possible. WERA Profession Develment Session Series
  11. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 ELA New Meridian SGP Calculations 2021
  12. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 ELA New Meridian SGP Calculations 2021
  13. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 ELA New Meridian SGP Calculations 2021
  14. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 ELA New Meridian SGP Calculations 2021
  15. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 Mathematics New Meridian SGP Calculations 2021
  16. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 Mathematics New Meridian SGP Calculations 2021
  17. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 Mathematics New Meridian SGP Calculations 2021
  18. Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 Mathematics New Meridian SGP Calculations 2021
  19. Academic Impact to Schools • Not surprisingly, there is wide variability in terms of the academic impact to schools. • The following bubble plots show academic impact to schools based upon school performance (status and growth) in 2019. • It is a challenge to help people to not confuse status with academic impact. • Many high and low achieving schools (and high and low growth schools) were severely impacted by the pandemic. WERA Profession Develment Session Series
  20. School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 3.1% Higher Growth School Size 50 Students 100 Students 9.9% 4.8% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  21. School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 11.4% Higher Growth School Size 50 Students 100 Students 22.4% 19.7% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  22. School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 29.3% Higher Growth School Size 50 Students 100 Students 32.9% 35.9% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  23. School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 38.7% Higher Growth School Size 50 Students 100 Students 28.9% 28.5% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  24. School Level COVID Academic Impact ELA 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 17.5% Higher Growth School Size 50 Students 100 Students 5.9% 11.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  25. School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 3.4% Higher Growth School Size 50 Students 100 Students 7.8% 6.1% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  26. School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 7.9% Higher Growth School Size 50 Students 100 Students 16.4% 16.3% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  27. School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 20.2% Higher Growth School Size 50 Students 100 Students 35.9% 32.1% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  28. School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 24.8% Higher Growth School Size 50 Students 100 Students 24.2% 19.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  29. School Level COVID Academic Impact Mathematics 2019 Growth by Achievement Higher Achievement Higher Achievement Lower Growth 43.7% Higher Growth School Size 50 Students 100 Students 15.6% 26.2% 250 Students 500 Students 2019 Percent Proficient COVID Impact 2021 SGP − 2019 SGP Improvement (> 5) 40 Modest to None (−5 to 5) Moderate (−15 to −5) Large (−25 to −15) Achievement Severe (< −25) Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2019 Median Student Growth Percentile WERA Profession Develment Session Series
  30. Looking forward to 2021-2022 • As we (hopefully) transition from pandemic to recovery, data from 2021-2022 will provide an excellent basis to understand whether what type of recovery systems are demonstrating: • V-shaped, U-Shaped, L-Shaped, or no recovery • States with compromised 2021 assessments (low participation, fall testing, no testing) will have tradeoffs to make in terms of using 2021-2022 data to assess recovery. • Using data for these purposes is not the typical use of data for “accountability” and requires states to promote results in ways not related to accountability. This can be a delicate balancing act. • Overall, I’m optimistic about the use of 2020-2021 data for academic impact as well as to inform recovery in 2021-2022. WERA Profession Develment Session Series
  31. Questions WERA Profession Develment Session Series