Academic Impact and Recovery

Center for Assessment Reidy Interactive Lecture Series · September 2022

Academic Impact and Recovery — Center for Assessment Reidy Interactive Lecture Series, September 2022.
Published

September 22, 2022

Event
Colloquium
Slides
17
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Academic Impact and Recovery Conceptualization and Results Damian Betebenner — Center for Assessment Reidy Interactive Lecture Series Portsmouth, New Hampshire September 21st, 2022
  2. COVID-19 Pandemic • The COVID-19 Pandemic lead to, arguably, the largest educational disruption in the history of the United States. • Two-and-a-half years into the pandemic and the impacts of the pandemic are still being investigated. • The pandemic impacted students academically, emotionally, physically, and financially. • In the brief overview I’ll focus on academic impact and discuss what is meant by academic impact and recovery as well as present results compiled across multiple states. Rhode Island SGP Growth Calculations in 2021 & 2022
  3. What is Academic Impact? • The pandemic and all the ensuing disruptions functioned/ functions as an “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • In education these two impediments manifest as: ▪ Decrease in student growth (i.e. decrease in velocity = deceleration). ▪ Decrease in student attainment. Pandemic Related Academic Impact in Indiana 3
  4. Learning acceleration/deceleration • Summarization of assessment data (either for diagnostic or accountability purposes) emphasizes two types of data results: Status (i.e. student attainment) and Growth (i.e., student academic progress) • With the pandemic, emphasis has changed to look at academic impact • We emphasize that Impact is synonymous with decleration and Recovery is synonymous with acceleration ▪ Deceleration, by definition, is the change in (i.e. decrease) velocity (i.e. growth) ▪ Acceleration, by definition, is the change in (i.e., increase) velocity (i.e., growth ▪ To understand whether and the extent to which recovery is occurring, one must understand the magnitude of impact. Pandemic Related Academic Impact in Indiana 4
  5. Assessment Data • To investigate impact we utilized two primary sources of data • State summative assessment data • State English Language Proficiency Data • We investigated academic impact by using two complementary analyses: • Change in attainment: Equi-percentile scale score change (all grades) • Change in growth: Baseline referenced student growth percentile change (grades with growth) Pandemic Related Academic Impact in Indiana 5
  6. ILEARN Mathematics Grade 7 Indiana 2019 to 2021 COVID⌧19 Academic Impact All Students Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary 2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score Academic Impact: Overall and by Quintile Grade 7 Scaled/Standardized Score and Achievement Level Cuts 2017 to 2019 Mean Conditional Status 2.5 2019 Percentiles (Observed) Overall 2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed) Observed + Imputed 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large 2019 to 2021 Growth Impact 2019 to 2021 Status Impact 2019 and 2021 Mean Scaled Scores Observed 2021 Grade 7 Scaled Score Decile Cuts 1.5 Large Large Large Large Large 2021 Grade 7 Scaled Score Participation Total Status Growth 2019: 41.4% Proficient 2021: 30.5% Proficient Growth/Status Change Legend 2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 7 Students 50.0 50.0 50.0 50.0 50.0 50.0 2019 Grade 7 Students 6394 (−1.45) 6486 (−0.51) 6540 ( 0.04) 6591 ( 0.57) 6667 ( 1.35) 6536 (0.00) 2021 Grade 7 Students 37.0 33.0 28.0 27.0 34.0 32.0 2021 Grade 7 Students 6377 (−1.63) 6462 (−0.76) 6514 (−0.22) 6564 ( 0.29) 6647 ( 1.14) 6513 (−0.23) Calculations suppressed Academic Impact (Observed) −13.0 −17.0 −22.0 −23.0 −16.0 −18.0 Change (Observed) −17 (−0.18) −24 (−0.25) −26 (−0.26) −27 (−0.27) −20 (−0.20) −23 (−0.23) Academic Impact (+ Imputed) −13.0 −17.0 −22.0 −23.0 −16.0 −18.0 Change (+ Imputed) −17 ( 2.22) −24 ( 1.45) −26 ( 1.00) −27 ( 0.58) −20 (−0.03) −23 (−0.24) due to small group N (< 50) Count Observed 2019 15,509 15,848 15,723 15,831 15,851 78,762 Count Observed 2019 16,896 16,948 16,777 17,044 16,913 84,578 Count Observed 2021 15,622 15,664 15,720 15,739 15,816 78,561 Count Observed 2021 16,535 16,496 16,735 16,561 16,556 82,883 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 2021 to 2019 Grade 7 Status Change (Observed) 30 2021 to 2019 SGP Difference (+ Imputed) 30 0.30 2021 to 2019 Grade 7 Status Change (+ Imputed) 0.83: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 0.66: Correlation SGP and Conditional Status Difference 2019 to 2021 2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Ranks and Scaled Scores (2019 Norm) 6
  7. 2020-2021: Academic Impact ILEARN ELA and Mathematics Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Economically Disadvantaged Students Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Large Non−Economically Disadvantaged Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large African American Students Large Large Moderate Large Moderate Moderate Severe Severe Severe Severe Large Large Asian Students Moderate Moderate Moderate Moderate Moderate Modest to None Moderate Moderate Large Severe Severe Large White Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Hispanic Students Large Moderate Large Large Moderate Moderate Large Large Large Severe Large Large ELL Students Large Moderate Large Large Moderate Large Large Large Large Large Moderate Large Special Education Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Female Students Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Severe Large Large Male Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Pandemic Related Academic Impact in Indiana 7
  8. 2020-2021: Academic Impact ELA And Mathematics Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Students Non−Economically Disadvantaged Students African American Students Asian Students White Students Hispanic Students ELL Students Special Education Students Female Students Male Students Pandemic Related Academic Impact in Indiana 8
  9. 2020-2021: A Year of Disruptions WIDA ACCESS WIDA⌧ACCESS Composite WIDA IN 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement WIDA⌧ACCESS Composite WIDA IN 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Pandemic Related Academic Impact in Indiana 9
  10. 2020-2021: Multi-State Impact ELA And Mathematics Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area: All Students Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 State A Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe State B Moderate Moderate Large Large Large Large Large Moderate Severe Severe Large Severe State C Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State D Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe State G Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State H Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State I Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate Pandemic Related Academic Impact in Indiana 10
  11. 2020-2021: Multi-State Impact ELA And Mathematics Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Growth/Status Change Legend NOTE: Quintiles for each state are based upon quintile Modest Calculations suppressed cuts for that state. Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile State A State B State C State D State E State F State G State H State I Pandemic Related Academic Impact in Indiana 11
  12. 2020-2021: Multi-State Impact (WIDA-ACCESS) WIDA⌧ACCESS Composite Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area: All Students Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State G Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None Pandemic Related Academic Impact in Indiana 12
  13. Academic Recovery 2021-2022 • Education during the 2021-2022 academic year was much closer to normal for students. • With school closer to normal, an explicit goal was to try and help students recover from losses incurred during the initial phase of the pandemic. • Our primary goal with analyses in 2021-2022 is to accurately distinguish between three situations: • Deterioration: Continued decreased rates of learning • Stabilization: A return to normal rates of learning • Recovery: Acceleration to above normal rates of learning sufficient to catch students up from last year’s impacts. Pandemic Related Academic Impact in Indiana 13
  14. 2021-2022: Recovery (or not) ILEARN ELA and Mathematics Indiana 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.55 −0.07 0.1 0.2 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization African American Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Asian Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization ELL Stabilization Stabilization Stabilization Deterioration Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Special Education Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Male Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Pandemic Related Academic Impact in Indiana 14
  15. Recovery • We are in the middle of compiling 2022 results across multiple states looking at academic recovery • Most results thus far for states suggest a return to typical rates of learning which suggests something between stabilization bordering on recovery to stabilization bordering on deterioration. • This does not mean students are returning to levels of achievement they would have reached under typical learning scenarios • Acceleration of learning necessary to get students to recover will be difficult to produce without dramatic interventions Pandemic Related Academic Impact in Indiana 15
  16. Cautions going forward • Maintaining scales to pre-pandemic years will increasingly become challenging for states. • Two views of returning to “normal” • Is the system returning to normal. • Are students impacted by the pandemic returning to normal. • System can return to “normal” without student returning to normal — students eventually leave the system. • Status comparison increasingly become misleading: Grade 3 students in 2022-2023 were in kindergarten in 2019-2020 — Pandemic Related Academic Impact in Indiana 16
  17. State Leaders • Darin Nielsen, Utah Department of Education • Robert Lee, Massachusetts Department of Elementary and Secondary Education • Charity Flores, Indiana Department of Education