New Hampshire 2023 Academic Impact Recovery Results

New Hampshire Department of Education · July 22, 2023

New Hampshire 2023 Academic Impact Recovery Results — New Hampshire Department of Education, July 22, 2023.
Published

July 22, 2023

Event
State agency
Slides
27
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. COVID-19 2023 Impact & Recovery Background, Results, and Next Steps Damian Betebenner & Adam VanIwaarden Center for Assessment July 21st, 2023
  2. Background Introductory Concepts and Terminology
  3. COVID-19 Pandemic • The COVID-19 Pandemic led to, arguably, the largest educational disruption in the history of the United States. • Nearly 3 years into the pandemic and the impacts of the pandemic are still being felt. • The pandemic impacted students academically, emotionally, physically, and financially. • In this overview, we address three things: • Academic Impact: What is it and what happened to students in New Hampshire? • Academic Recovery: What is it and what is happening to students in New Hampshire currently? • Now What?: How can impact and recovery data be leveraged to help students going forward? Evidence-based investigations of what is working.
  4. What is Academic Impact? What is Academic Recovery? • The pandemic and all the ensuing disruptions functioned/functions as an academic headwind, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • In education odometer is to attainment as speedometer is to growth. • Academic impact is analogous to a headwind, decreasing the rate of student learning (deceleration) which, in turn leads to lower attainment
  5. What is Academic Impact? What is Academic Recovery? • Academic recovery is analogous to an academic tailwind, increasing the rate of student learning (acceleration) which, in turn leads to higher attainment. • During Fall 2021 to present, under the presumption that student education got back to normal, the hope was that rates of learning stabilized to pre-pandemic rates or, even better, increased to rates exceeding pre-pandemic rates allowing students to catch-up. • Examination of academic impact/recovery are opposite sides of the same coin and utilize the same analytics.
  6. Impact & Recovery • We investigate impact/recovery in two ways: • Change in same grade/same cohort attainment: We examine equi-percentile change in attainment. Impact: 2019 to 2021, Recovery 2021 to 2022 • Change in student growth: We examine change in baseline (pre-pandemic) student growth (SGPs). Impact: 2019 to 2021 (using 2017 to 2019 growth norms), Recovery: 2021 to 2022 (using 2018 to 2019 growth norms).
  7. Assessment Data • We investigated the impacts in achievement and growth of the pandemic across two time frames: • Impact: Spring 2020 through Summer 2021 • Recovery: Fall 2021 through Summer 2022 • To investigate impact we utilized two primary sources of data • State summative assessment data (NH-SAS) • Impact: NH-SAS Spring 2019 to Spring 2021 (NH-SAS testing cancelled in • Recovery: NH-SAS Spring 2021 to Spring 2022 • State English Language Proficiency Data (ACCESS) • Impact: ACCESS Winter 2020 to Winter 2021 • Recovery: ACCESS Winter 2021 to Winter 2022
  8. Impact From Spring 2020 to Summer 2021 NH-SAS: Spring 2019 and Spring 2021 Assessments ACCESS: Winter 2020 and Winter 2021 Assessments
  9. SAS Mathematics Grade 7 New Hampshire 2019 to 2021 COVID⌧19 Academic Impact All Students Growth: Grade 5 2019 to Grade 7 2021 Status: Grade 7 2019 to Grade 7 2021 Summary 2019 Grade 5 Scaled Score Decile Cuts 2019 Grade 7 Scaled Score Academic Impact: Overall and by Quintile Grade 7 Scaled/Standardized Score and Achievement Level Cuts 2.5 Overall Observed + Imputed 2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed) 2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Large 2019 to 2021 Growth Impact 2019 to 2021 Status Impact 2021 Grade 7 Scaled Score Decile Cuts Observed 2019 and 2021 Mean Scaled Scores Large Large Large Severe Severe 2021 Grade 7 Scaled Score 70 70 550 Participation 50 −0.29 Total Status Growth 2019: 47.1% Proficient 2021: 34.1% Proficient −3 Growth/Status Change Legend 2019 Grade 5 Scaled Score with Achievement Level Cuts Grade 7 Overall Percentile Ranks Modest Severe Large Moderate to Improvement 2019 & 2021 Grade 7 Median SGPs by Quintile 2019 & 2021 Grade 7 Mean Scaled (Standardized) Scores by Quintile None Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 7 Students 50.0 50.0 49.0 50.0 51.0 50.0 2019 Grade 7 Students 457 (−1.46) 517 (−0.45) 547 ( 0.05) 575 ( 0.52) 622 ( 1.32) 544 (0.00) 2021 Grade 7 Students 33.0 30.0 27.0 24.0 25.0 28.0 2021 Grade 7 Students 440 (−1.76) 501 (−0.73) 530 (−0.23) 558 ( 0.24) 605 ( 1.03) 527 (−0.29) Calculations suppressed Academic Impact (Observed) −17.0 −20.0 −22.0 −26.0 −26.0 −22.0 Change (Observed) −18 (−0.30) −16 (−0.28) −17 (−0.29) −17 (−0.28) −17 (−0.29) −17 (−0.29) Academic Impact (+ Imputed) −18.0 −19.0 −22.0 −25.0 −26.0 −21.8 Change (+ Imputed) −16 ( 0.21) −16 (−0.14) −17 (−0.32) −17 (−0.48) −17 (−0.76) −20 (−0.35) due to small group N (< 50) Count Observed 2019 2,155 2,185 2,215 2,208 2,178 10,941 Count Observed 2019 2,422 2,435 2,494 2,469 2,447 12,267 Count Observed 2021 1,518 1,677 1,713 1,772 1,818 8,498 Count Observed 2021 2,061 1,998 2,078 2,035 2,052 10,224 2021 to 2019 Scaled/Standardized Score Difference 2021 to 2019 SGP Difference (Observed) 0.55 2021 to 2019 Grade 7 Status Change (Observed) 30 2021 to 2019 SGP Difference (+ Imputed) 30 0.50 2021 to 2019 Grade 7 Status Change (+ Imputed) 0.31: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 2021 to 2019 Conditional Status Difference (Observed) 0.45 0.68: Correlation SGP and Conditional Status Difference 0.40 2019 to 2021 2019 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 7 Percentile Ranks and Scaled Scores (2019 Norm)
  10. Summary We summarize the academic impact both overall and by achievement quintile For Grade 7 mathematics, the overall academic impact for students was large (z decrease = -0.29 and median SGP decrease = Academic impact was not uniform across students. For this grade/content area higher achieving students were more severely impacted than lower achieving student.
  11. SAS ELA and Mathematics New Hampshire 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large Economically Disadvantaged Students Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Moderate Non−Economically Disadvantaged Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large Special Education Students Large Moderate Modest to None Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate African American Students Large Moderate Modest to None Modest to None Large Moderate Large Moderate Severe Large Moderate Moderate Asian Students Modest to None Moderate Modest to None Moderate Moderate Modest to None Large Large Large Large Large Large White Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large Hispanic Students Large Large Moderate Moderate Large Moderate Large Severe Severe Large Large Large ELL Students Moderate Moderate Moderate Moderate Large Moderate Large Large Large Large Large Large Female Students Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large Male Students Large Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
  12. WIDA ACCESS WIDA⌧ACCESS Composite WIDA NH 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
  13. All Students Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 New Hampshire Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State A Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate State B Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State C Modest to None Modest to None Moderate Moderate Moderate Modest to None Moderate Moderate Moderate Large Large Moderate State D Moderate Moderate Large Large Large Large Moderate Moderate Large Large Large Large State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe State G Moderate Moderate Moderate Moderate Large Moderate State H Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State I Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State J Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe
  14. All Students Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
  15. Recovery Years 1 and 2 Year 1: From Fall 2021 to Summer 2022, Year 2: From Fall 2022 to Summer 2023 NH-SAS: Spring 2021, Spring 2022 and Spring 2023 Assessments ACCESS: Winter 2021, Winter 2022 and Winter 2023 Assessments
  16. Recovery Phase • During the 2021-2022 academic year, school returned to normal for most students. • The 2022-2023 academic year saw continued that trend. • The hope was that a return to normal would lead recovery of the academic losses incurred by students. • Because learning slowed during the initial phase of the pandemic, to make up for those losses, learning must accelerate to rates above normal in order to catch-up for those loses. • Data from two sources was utilized to examine whether recovery is occurring: • NH-SAS data from 2021, 2022 and 2023 • WIDA-ACCESS data from 2021, 2022 and 2023
  17. Recovery Phase: Cautions in Interpretation • Interpretation of “recovery” based upon outcomes from tests can be misleading. • Since NCLB, interpretation of education system quality is often based upon attainment of students (e.g., Percentage of proficient students) • Because student attainment dropped precipitously in New Hampshire (and nationwide) and has not yet recovered to pre-pandemic percentages, there is a common misunderstanding that the education system continues to suffer from the pandemic. • To provide the appropriate nuance to understand academic recovery we distinguish between: • Student Recovery: Are students back to levels of attainment they would have been had the pandemic not occurred? • Education System Recovery: Is the system back to supporting rates of learning observed before the pandemic? • NOTE: student learning (i.e., student growth) precedes student attainment. Higher/lower rates of learning lead to high/low attainment.
  18. Student Level Recovery: ELA • Though it is impossible to know exactly what level of attainment students today would have reached had the pandemic not occurred, a conservative basis of comparison would be the 2019 student scores (mean scaled score & % proficient) Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 2020 No Testing
  19. Student Level Recovery: Mathematics • Though it is impossible to know exactly what level of attainment students today would have reached had the pandemic not occurred, a conservative basis of comparison would be the 2019 student scores (mean scaled score & % proficient) Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 2020 No Testing
  20. System Level Recovery: ELA • Based upon pre-pandemic rates of student learning, we can compare rates of learning following the pandemic to determine the extent to which the system has returned to “normal”. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 2019 NA 50 50 50 50 50 2020 No Testing 2021 NA NA 49 45 49 46 2022 NA 51 52 54 48 51 2023 NA 52 58 55 55 59
  21. System Level Recovery: Mathematics • Based upon pre-pandemic rates of student learning, we can compare rates of learning following the pandemic to determine the extent to which the system has returned to “normal”. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 2019 NA 50 50 50 50 50 2020 No Testing 2021 NA NA 37 35 35 36 2022 NA 55 56 56 53 54 2023 NA 52 51 54 52 54
  22. Four Academic Recovery Scenarios with NH- SAS: V-, U-, L-, and No-recovery
  23. 2021-2022 & 2022-23: Recovery Phase • Overall results suggest that New Hampshire students are experiencing something resembling a U- shaped recovery. • Depending upon how gradual the U-shaped recovery, the time-span needed to fully recovery may extend beyond the amount of time • The education system has recovered and is actually performing slightly better than it had historically. in that it has returned to rates of learning that it produced pre-pandemic. • It is critical to note that system level recovery is not the same as student recovery. • Because student learning was significantly depressed between 2019 and 2021. Student recovery will require rates of learning well in excess of what was typical pre-pandemic. • Compared to results the Center has modeled in a dozen other states, New Hampshire is at or near the top in terms of how it’s system and students are faring. • Averages can obscure important variability. Whereas some students are faring well, others struggle.
  24. SAS Reading and Mathematics New Hampshire 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) Reading Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization African American Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Recovery Asian Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Recovery Stabilization Stabilization Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Limited English Proficient Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Male Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
  25. SAS ELA and Mathematics New Hampshire 2022 to 2023 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Non−Economically Disadvantaged Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization African American Stabilization Deterioration Stabilization Stabilization Stabilization Recovery Recovery Stabilization Deterioration Recovery Recovery Recovery Asian Stabilization Stabilization Recovery Stabilization Stabilization Recovery Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization White Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Limited English Proficient Stabilization Stabilization Recovery Stabilization Recovery Recovery Stabilization Stabilization Stabilization Recovery Stabilization Recovery Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Female Stabilization Stabilization Recovery Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Male Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
  26. 2021-2022 & 2022-23: Recovery Phase • Results from 2023 following those in 2022 suggest New Hampshire is on the right path. • In both ELA and mathematics, rates of learning returned to historical (pre-pandemic norms) and in many grades exceeded pre-pandemic rates of learning. This increase in rates of learning beyond pre-pandemic is helping students recover and to gradually catch-back-up. • The education system has recovered and is actually performing slightly better than it had historically. in that it has returned to rates of learning that it produced pre-pandemic. • It is critical to note that system level recovery is not the same as student recovery. • Because student learning was significantly depressed between 2019 and 2021. Student recovery will require rates of learning well in excess of what was typical pre-pandemic. • Students not making sufficient progress eventually run out of time needed to catch-up.
  27. Questions