Pandemic Related Academic Impact in 2021: A look at results from Arizona

Arizona Department of Education · October 25, 2021

Pandemic Related Academic Impact in 2021: A look at results from Arizona — Arizona Department of Education, October 25, 2021.
Published

October 25, 2021

Event
State agency
Slides
22
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Pandemic Related Academic Impact in 2021 A look at results from Arizona Damian Betebenner & Adam Van Iwaarden Center for Assessment Arizona State Board of Education Meeting October 25th, 2021
  2. Pandemic Related Academic Impact • We are all aware that the pandemic greatly disrupted the lives of people worldwide over the last 18 months. • Students were dramatically impacted as education was delivered in many new ways to students nationwide. • A concern of many is to what extent the pandemic impacted students: Academically, emotionally, physically. • The Center for Assessment has been working with a dozen states to investigate the extent to which the pandemic has impacted student learning. Arizona State Board of Education Meeting, October 25th, 2021
  3. What do we mean by “Academic Impact”? • The pandemic and all the ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two outcomes represent themselves as: • Decrease in student growth. • Decrease in student attainment. • Today’s presentation depicts academic impact vis-à-vis student growth. Arizona State Board of Education Meeting, October 25th, 2021
  4. AzM2 Growth Analyses in 2021 • Given the cancellation of testing in spring 2020, student growth in 2021 is calculated across a two year span: 2019 to 2021. • In order to examine the slow down due to the pandemic, growth is compared to pre-pandemic student growth from 2017 to 2019 – these are baseline norms. • Baseline growth in each grade and content area has a mean/median of 50. • The extent to which growth from 2019 to 2021 is below 50 is indicative of slowdown in student learning due to the pandemic. Arizona State Board of Education Meeting, October 25th, 2021
  5. A Note about Missing Data • Low participation on state assessments is a threat to the validity of academic impact results in 2021. • In some schools/districts, participation on AzM2 assessments was below that traditionally observed. • Would results for the school/district/state been the same if all students had tested? Would academic impact appear the same? • As part of Arizona’s academic impact analysis we employed multiple imputation to determine to what extent missing data impacts summary results, flagging results that are likely different due to student non-testing. • Overall, multiple imputation analyses indicate that current results represent a best case scenario for Arizona students. That is, had all students participated, the results would likely be lower. Arizona State Board of Education Meeting, October 25th, 2021
  6. Analyses for 2020-2021 • Today we show pandemic related academic impact: • At the state level by grade and content area • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school Arizona State Board of Education Meeting, October 25th, 2021
  7. AzM2 Median/Mean Skip-Year SGPs by Grade ELA Mathematics Grade 2019 2021 Loss 2019 2021 Loss 3 NA NA NA NA 4 NA NA NA NA 10 50.0 (52,181) 36.0 (50,370) -14.0 NA NA NA Arizona State Board of Education Meeting, October 25th, 2021
  8. Discussion of Content Area by Grade Impact • The decrease in student learning (i.e., growth) in both ELA and Mathematics is large. • Growth decreases in math exceeded those in ELA, but both were large. • Growth impact was uniformly large across grades. • The decreases in growth, especially at the state level, are unprecedented. • Recovery from the losses will similarly be unprecedented --- likely requiring multiple-years with additional supports Arizona State Board of Education Meeting, October 25th, 2021
  9. AzM2 Median/Mean Skip-Year SGPs by Ethnicity ELA Mathematics Grade 2019 2021 Loss 2019 2021 Loss American Indian Black/African American Hispanic or Latino Asian 63.0 (10,008) 46.0 (7,415) -17.0 65.0 (7,647) 46.0 (7,415) -19.0 White 52.0 (132,952) 38.0 (113,094) -14.0 53.0 (107,281) 35.0 (92,466) -18.0 Multiple Races 51.0 (11,066) 37.0 (10,213) -14.0 52.0 (9,451) 31.0 (8,754) -21.0 Arizona State Board of Education Meeting, October 25th, 2021
  10. Discussion of Content Area by Ethnicity Impact • Note that prior to COVID there were disparities in student growth rates by ethnicity • The decrease in student learning (i.e., growth) in both ELA and Mathematics is large across all ethnic subgroups. • Despite the large impact, some ethnic subgroups faired worse that others: • Hispanic and African American students in mathematics were impacted more than their counterparts. • Ethnicity by Grade analyses show similar disparities. Arizona State Board of Education Meeting, October 25th, 2021
  11. AzM2 Median/Mean Skip-Year SGPs by free/reduced lunch status ELA Mathematics Grade 2019 2021 Loss 2019 2021 Loss Free Reduced 32.0 Lunch: Yes (133,137) Free Reduced Lunch: No Arizona State Board of Education Meeting, October 25th, 2021
  12. Discussion of Content Area by Free/Reduced Lunch Status Impact • Note that prior to COVID there were disparities in student growth rates by free/reduced lunch status • The decrease in student learning (i.e., growth) in both ELA and Mathematics is large across both free/reduced lunch groups. • The decrease in math associated with students receiving free/ reduced price lunch (-25.0) is quite large. Arizona State Board of Education Meeting, October 25th, 2021
  13. AzM2 Median/Mean Skip-Year SGPs by English learner status ELA Mathematics Grade 2019 2021 Loss 2019 2021 Loss English 45.0 32.0 45.0 24.0 Learner: Yes (19,614) (17,814) (18,575) (16,612) English Learner: No Arizona State Board of Education Meeting, October 25th, 2021
  14. Discussion of Content Area by English learner Status Impact • Note that prior to COVID there were disparities in student growth rates by English learner status • Impacts were large and fairly uniform across English learners/ non-English learners. Arizona State Board of Education Meeting, October 25th, 2021
  15. AzM2 Median/Mean Skip-Year SGPs by Special Education status ELA Mathematics Grade 2019 2021 Loss 2019 2021 Loss Special Education: No Special 40.0 32.0 42.0 29.0 Education: Yes (39,347) (32,714) (35,066) (28,799) Arizona State Board of Education Meeting, October 25th, 2021
  16. Discussion of Content Area by Special Education Status Impact • Note that prior to COVID there were disparities in student growth rates by special education status. • Pandemic related academic impact was less for special education students than for non-special education students. • It is not clear what is responsible for this disparity. Two possibilities: • Better support for special education students throughout the pandemic. • Impoverished pre-pandemic rates of learning for special education students. Arizona State Board of Education Meeting, October 25th, 2021
  17. Academic Impact by School • Average academic impact across students in Arizona was large. • However, there was substantial variation in academic impact at the school level. • The follow slides show academic impact for ELA and Mathematics based upon the School’s 2019 A-F letter grade. • Note, that high performing schools are not immune to academic impact from the pandemic. Arizona State Board of Education Meeting, October 25th, 2021
  18. ELA: School Academic Impact by 2019 Letter Grade Arizona State Board of Education Meeting, October 25th, 2021
  19. Mathematics: School Academic Impact by 2019 Letter Grade Arizona State Board of Education Meeting, October 25th, 2021
  20. Summary of Academic Impacts • Academic impacts to students over the course of the pandemic have been unprecedented and large nationwide and in Arizona. • Thought there is some variability in terms of who is impacted, there are no groups that have been immune to academic impacts including: • High and low achieving students • Students from different ethnic groups • Special education, regular education, English Language Learner students. • The large size of the impacts indicate lengthy time (> 1 year) to recover, likely requiring education supports beyond just a norm return to schooling. Arizona State Board of Education Meeting, October 25th, 2021
  21. Looking forward to 2021-2022 • As we (hopefully) transition from pandemic to recovery, AzM2 data from 2021-2022 will provide an excellent basis to understand what type of recovery students are demonstrating: • V-shaped, • U-Shaped, • L-Shaped, or • No recovery • Using data for these purposes is not the typical use of data for “accountability” and requires using results in ways not related to accountability. This can be a delicate balancing act. • I’m optimistic about ADE’s use of AzM2 2020-2021 data to evaluate academic impact and subsequently to monitor and inform recovery in Arizona State Board of Education Meeting, October 25th, 2021
  22. Questions Arizona State Board of Education Meeting, October 25th, 2021