COVID-19 Impact & Recovery in Utah

Utah Technical Advisory Committee · January 24, 2024

COVID-19 Impact & Recovery in Utah — Utah Technical Advisory Committee, January 24, 2024.
Published

January 24, 2024

Event
Technical advisory committee
Slides
40
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. COVID-19 Impact & Recovery in Utah 2023 Update Damian Betebenner & Adam VanIwaarden Center for Assessment November 8th, 2023
  2. Background Introductory Concepts and Terminology
  3. Academic Recovery in 2023 • The availability of 2023 RISE/UA+ data allows Utah to update COVID-19 recovery analyses. • Spring 2023 marks the third year since the pandemic began and this allows for the third set of analyses to be presented: • Summer 2021: Academic Impact • Summer 2022: Academic Recovery Year 1 • Summer 2023: Academic Recovery Year 2
  4. Two Types of Recovery in 2023: System & Student • We elaborate on the results from the previous recovery analyses in two ways: • System level recovery: Is the education system supporting rates of learning that are comparable to what it was prior to the pandemic? • Student level recovery: Are students achieving at levels that are comparable to what they were prior to the pandemic? • Originally, we focused mostly on system-level recovery: examining rates of learning in 2021 and 2022 compared to those prior to the pandemic. • We continue examining system level analyses but also focus on student-level recovery. • The reason for providing both is to counter a prevailing narrative that the education system continues to suffer because students lag behind their pre-pandemic levels of attainment.
  5. Two Types of Recovery: System & Student • System-level recovery • Comparison of 2019 and 2023 same-grade student growth: We examined the change in baseline student growth (SGPs) for 2019 students and 2023 students utilizing coefficient matrices derived from pre-pandemic cohorts of students. • Student-level recovery • Comparison of 2019 to 2023 same-grade achievement: We examine the equi-percentile change in achievement for 2019 students and 2023 students at same-grade
  6. System-level recovery • It is better to understand the health of the education system via the rates of student learning that it supports rather than the level of student attainment that are evident. • Because students’ rates of learning slowed so dramatically during the pandemic, rates of learning necessary to catch students up to where they would have been would have to be equally dramatic (in the opposite direction) • That is, the system could return to producing normal rates of learning without students recovering.
  7. Student-level recovery • Student level recovery is the most common representation of recovery presented in the media: Where are students now compared to where they were in 2019. • Comparison to 2019 attainment levels is a “low bar” for comparison: • Attainment levels in 2023, had the pandemic not occurred, would almost certainly have been higher than 2019 levels (based on “typical” rates of yearly improvement) • In the following we provide data indicating the extent to which student attainment is behind 2019 attainment levels.
  8. System Recovery: Cohort-Based Comparison (RISE) Compare observed versus expected attainment for a cohort of students Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8
  9. Student Recovery: Cross-Sectional Comparison (RISE) Compare same grade/content area attainment across different years Grade 3 No Testing due to Grade 4 Grade 5 Grade 6 COVID-19 Pandemic Grade 7 Grade 8
  10. Why Baseline SGPs? • Why does one need to look at baseline referenced SGPs instead of, for example, scale score gains. There are several reasons. • SGPs are the growth metric that the state of Utah has been using for more than a decade. • SGPs are readily interpretable probabilistically whereas gain scores are not. • Quantile regression (on which SGPs are based) is invariant to mono-tone transformations of scale (equi-variant to mono-tone transformations of scale). OLS regression, for example, is not. See Koenker (2005, p. 38) • Gain scores (which can be calculated since RISE is vertically scaled) are dependent upon: Content area, grade, prior level of achievement. • Gain scores were largely discredited as an effective means of communicating change. For example, no state uses them as a growth model.
  11. Impact From Spring 2020 to Spring 2021 RISE: Spring 2019 and Spring 2021 Assessments
  12. Impact Phase • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Impact was categorized as ranging from Severe to Modest to None. • For almost all content areas and grades, impact was categorized as moderate (-0.25 <= effect size <= -0.1/ -15 <= median SGP decrease <=
  13. RISE ELA and Mathematics Utah 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate Economically Disadvantaged Students Large Large Moderate Moderate Modest to None Moderate Moderate Moderate Large Large Moderate Moderate Non−Economically Disadvantaged Students Moderate Large Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Special Education Students Large Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate African American Students Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Large Large Large Large Asian Students Moderate Moderate Moderate Moderate Moderate Modest to None Moderate Moderate Large Large Moderate Large White Students Moderate Large Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Hispanic Students Large Large Moderate Moderate Modest to None Modest to None Moderate Moderate Large Large Moderate Moderate ELL Students Large Severe Moderate Moderate Modest to None Moderate Large Large Large Large Moderate Moderate Female Students Moderate Large Modest to None Moderate Modest to None Modest to None Moderate Moderate Large Large Moderate Moderate Male Students Moderate Large Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate
  14. ELA And Mathematics Utah 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change NOTE: Quintiles for each student −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile Modest Calculations suppressed cuts defined for All Students. Severe Large Moderate to Improvement None due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Students Non−Economically Disadvantaged Students Special Education Students African American Students Asian Students White Students Hispanic Students ELL Students Female Students Male Students
  15. Recovery Year 1 From Fall 2021 to Summer 2022 RISE: Spring 2021 and Spring 2022 Assessments
  16. Recovery Phase Year 1 • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Achievement (Student Recovery) and Growth (System Recovery) were characterized in three ways: • Lagging: Behind pre-pandemic levels (attainment or growth) • On Par: About the same as pre-pandemic levels (attainment or growth) • Leading: Ahead of pre-pandemic levels (attainment or growth) • Because learning slowed during the initial phase of the pandemic (2020 to 2021), to make up for those losses, learning must accelerate to rates above normal (i.e., leading)
  17. RISE ELA and Mathematics Utah 2019 to 2022 COVID⌧19 Academic Student Recovery Grade & Content Area Lagging Leading Student Recovery characterizes the extent Status Change −0.38 −0.09 0.12 0.38 to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 All Students Lagging Lagging On Par On Par On Par On Par On Par On Par Lagging Economically Disadvantaged Lagging Lagging Lagging Lagging On Par Lagging Lagging Lagging Lagging Non−Economically Disadvantaged Lagging Lagging On Par Lagging On Par Lagging On Par Lagging Lagging African American Lagging On Par On Par On Par On Par Lagging On Par On Par On Par Asian On Par On Par On Par On Par On Par On Par On Par On Par On Par White On Par Lagging On Par On Par On Par On Par On Par On Par Lagging Hispanic Lagging Lagging Lagging On Par On Par Lagging Lagging Lagging Lagging ELL Lagging Lagging Lagging Lagging On Par Lagging Lagging Lagging Lagging Special Education Lagging Lagging Lagging On Par On Par Lagging On Par Lagging On Par Female Lagging Lagging On Par On Par On Par On Par Lagging Lagging Lagging Male Lagging Lagging On Par On Par On Par On Par On Par On Par On Par
  18. RISE ELA and Mathematics Utah 2019 to 2022 COVID⌧19 Academic Student RecoveryGrade, Content Area & Achievement Quintile Lagging Leading Student Recovery characterizes the extent Status Change to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged African American Asian White Hispanic ELL Special Education Female Male
  19. RISE ELA and Mathematics Utah 2021 to 2022 COVID⌧19 Academic System Recovery Grade & Content Area Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students On Par Leading Leading Leading On Par On Par On Par On Par Leading Leading Leading On Par Economically Disadvantaged On Par On Par Leading Leading On Par On Par On Par On Par Leading Leading Leading On Par Non−Economically Disadvantaged On Par On Par Leading Leading On Par On Par On Par On Par Leading Leading Leading On Par African American On Par Leading Leading Leading On Par On Par On Par Leading Leading Leading Leading Leading Asian On Par On Par Leading Leading On Par On Par Leading On Par Leading Leading Leading Leading White On Par Leading Leading Leading Leading On Par On Par On Par Leading Leading Leading On Par Hispanic On Par Leading Leading On Par On Par On Par On Par Leading Leading Leading Leading On Par ELL On Par Leading Leading On Par On Par On Par On Par Leading Leading Leading On Par On Par Special Education On Par On Par Leading On Par On Par On Par On Par On Par Leading On Par On Par On Par Female On Par Leading Leading On Par On Par On Par On Par On Par Leading Leading Leading Leading Male On Par Leading Leading Leading On Par On Par On Par On Par Leading Leading Leading On Par
  20. RISE ELA and Mathematics Utah 2021 to 2022 COVID⌧19 Academic System Recovery Grade, Content Area & Achievement Quintile Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged African American Asian White Hispanic ELL Special Education Female Male
  21. Recovery Year 1 • Analysis of 2022 RISE data indicated several positive trends in Utah to be happy about. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Utah comes out as the best in terms of pandemic recovery. • Despite a robust recovery, students remain slightly behind 2019 levels in ELA and more in mathematics.
  22. Recovery Year 2 From Fall 2022 to Summer 2023 RISE: Spring 2022 and Spring 2023 Assessments
  23. Recovery Phase Year 2 • Student growth (baseline referenced to pre-pandemic rates of learning) was examined by content area and grade as well as for the major student demographic subgroups. • Achievement (Student Recovery) and Growth (System Recovery) were characterized in three ways: • Lagging: Behind pre-pandemic levels (attainment or growth) • On Par: About the same as pre-pandemic levels (attainment or growth) • Leading: Ahead of pre-pandemic levels (attainment or growth) • Because learning slowed during the initial phase of the pandemic (2020 to 2021), to make up for those losses, learning must accelerate to rates above normal (i.e., leading)
  24. 2023 Status Results • Percent Proficient/Standardized Score (pooled standard deviation)/Count for RISE: ELA Mathematics Grade
  25. 2023 SGP Results • Mean/Median/Count for baseline referenced SGPs in ELA and mathematics are as follows: ELA Mathematics Grade
  26. Status Recovery Black Line: 2019 (pre-covid) scaled-score distribution. Red line: 2023 scaled-score distribution. The impact of the pandemic on student achievement (decrease) is illustrated as the area between the black line (bottom) and the red line (top) X-axis: Grade 8 percentile rank The horizontal distance between the curves at the proficiency cut level (horizontal dashed blue line) illustrates the increase in percent proficient. Decrease of 7.2% proficient and decrease of 0.16 standard deviation
  27. Status Recovery Red line: 2023 - 2019 scaled-score distribution difference. The recovery from the pandemic (difference in the distributions) on student achievement (increase) is indicated by the red line. X-axis: Grade 8 percentile rank Average impact z = -0.16 standard units. Larger impact for Q3 and Q4 students.
  28. Growth Recovery Black Line: (pre-covid) conditional scaled-score distribution — Grade 8/2019 conditioned on Grade 7/2018 Red line: conditional scaled-score distribution — Grade 8/2023 conditioned on Grade 7/2022 The lag from the pandemic on student growth (increase) is illustrated as the area between the black line (top) and the red line (bottom) X-axis: Grade 7 Scaled Score and achievement level cuts. Fixing a level on the x-axis, the vertical distance between the curves is the average decrease in scale score associated with the impact for a student at that 8th-grade level.
  29. Growth Recovery Red line: SGP Difference Grade 8/2023 - Grade 8/2019 conditional scaled score distribution X-axis: Grade 7 Scaled Score and associated deciles and achievement level cuts Median SGP 2019 Grade 8: 50 Median SGP 2023 Grade 8: 46 Higher achieving groups (above the 40th percentile) demonstrating lower rates of learning than pre- COVID.
  30. Summary We summarize the academic recovery in terms of Growth (System Level Recovery) and Status (Student Level Recovery) both overall and by achievement quintile Growth is, overall, on par with the pre-covid growth. For the higher quintiles, growth is on-par bordering on lagging. Status is, overall, lagging. That is, student attainment has not returned to pre- pandemic levels. This reaffirms the point made two years ago: For students to recover, learning rates will have to
  31. UA+ ELA and Mathematics and Science Utah 2019 to 2023 COVID⌧19 Academic Student Recovery Grade & Content Area Lagging Leading Student Recovery characterizes the extent Status Change −0.38 −0.09 0.12 0.38 to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Science Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 All Students Lagging Lagging On Par Lagging On Par On Par On Par On Par On Par On Par Lagging Lagging On Par Lagging Lagging Lagging On Par On Par On Par Lagging Lagging Economically Disadvantaged Lagging Lagging Lagging Lagging On Par On Par On Par On Par Lagging Lagging Lagging Lagging On Par Lagging Lagging Lagging On Par On Par Lagging Lagging Lagging Non−Economically Disadvantaged On Par Lagging On Par Lagging On Par On Par On Par On Par On Par On Par Lagging Lagging On Par Lagging Lagging Lagging On Par On Par On Par On Par Lagging African American Lagging Lagging On Par On Par On Par Lagging On Par On Par On Par On Par On Par Lagging Lagging Lagging Lagging Lagging On Par On Par Lagging Lagging Lagging Asian On Par Lagging On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Leading Lagging On Par Lagging On Par On Par Lagging On Par Lagging White On Par Lagging On Par On Par On Par On Par On Par On Par On Par On Par Lagging On Par On Par Lagging Lagging Lagging On Par On Par On Par On Par Lagging Hispanic Lagging Lagging On Par Lagging On Par Lagging On Par On Par Lagging Lagging Lagging Lagging On Par Lagging Lagging Lagging On Par On Par Lagging Lagging Lagging ELL Lagging Lagging Lagging Lagging On Par Lagging On Par On Par Lagging Lagging Lagging Lagging On Par Lagging On Par On Par Lagging On Par Lagging Lagging Lagging Special Education Lagging Lagging Lagging On Par On Par Lagging On Par On Par On Par On Par On Par On Par On Par Lagging On Par Lagging On Par On Par Lagging Lagging Lagging Female Lagging Lagging On Par Lagging On Par On Par On Par On Par Lagging Lagging Lagging Lagging On Par On Par Lagging Lagging On Par On Par Lagging Lagging Lagging Male On Par Lagging On Par On Par On Par On Par Lagging On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par On Par On Par On Par Lagging
  32. UA+ ELA and Mathematics and Science Utah 2019 to 2023 COVID⌧19 Academic Student RecoveryGrade, Content Area & Achievement Quintile Lagging Leading Student Recovery characterizes the extent Status Change −0.38 −0.09 0.12 0.38 to which student attainment has returned to pre−pandemic levels of attainment. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Science Grade 3Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9Grade 10 Grade 3Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9Grade 10 Grade 6Grade 7Grade 8Grade 9Grade 10 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged African American Asian White Hispanic ELL Special Education Female Male
  33. RISE/UA+ ELA and Mathematics and Science Utah 2019 to 2023 COVID⌧19 Academic System Recovery Grade & Content Area Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Science Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9 Grade 10 Grade 4Grade 5Grade 6Grade 7Grade 8Grade 9 Grade 10 Grade 7Grade 8Grade 9 Grade 10 All Students On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par On Par Lagging Lagging Economically Disadvantaged On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par Lagging Lagging Lagging Non−Economically Disadvantaged On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par On Par On Par Lagging African American On Par Leading On Par On Par Lagging On Par On Par On Par On Par On Par Lagging Lagging Lagging Lagging On Par Lagging Lagging Lagging Asian On Par Leading On Par On Par On Par On Par On Par On Par On Par On Par Leading Lagging On Par Lagging On Par Lagging On Par Lagging White On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par On Par On Par Lagging Hispanic Lagging On Par On Par On Par Lagging On Par On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par Lagging Lagging Lagging ELL Lagging On Par On Par On Par Lagging On Par On Par On Par On Par On Par On Par Lagging On Par On Par On Par Lagging Lagging Lagging Special Education Lagging On Par On Par On Par Lagging On Par On Par On Par On Par On Par On Par Lagging On Par Lagging On Par Lagging Lagging Lagging Female On Par Leading On Par On Par On Par On Par On Par On Par On Par On Par On Par On Par Lagging Lagging On Par Lagging Lagging Lagging Male On Par On Par On Par On Par On Par Lagging On Par On Par On Par On Par On Par Lagging Lagging Lagging On Par On Par On Par Lagging
  34. RISE/UA+ ELA and Mathematics and Science Utah 2019 to 2023 COVID⌧19 Academic System Recovery Grade, Content Area & Achievement Quintile Lagging Leading System Recovery characterizes the extent Growth Change −30 −7 10 30 to which rates of student learning (i.e., growth) have returned to pre−pandemic rates of learning. Calculations suppressed Lagging On Par Leading due to small group N (< 50) ELA Mathematics Science Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 7 Grade 8 Grade 9 Grade 10 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged African American Asian White Hispanic ELL Special Education Female Male
  35. UA+ Science Grade 10 Utah 2019 to 2023 COVID⌧19 Academic Recovery All Students Growth: Grade 9/2018 to 10/2019 versus Grade 9/2022 to 10/2023 Status: Grade 10/2019 versus Grade 10/2023 Summary 2018 Grade 9 Scaled Score Decile Cuts 2019 Grade 10 Scaled Score Grade 10 Scaled/Standardized Score and Achievement Level Cuts 10 20 30 40 50 60 70 80 90 100 150 200 250300 Growth/Status Recovery Legend 2018 to 2019 Mean Conditional Status 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 3 2023 Percentiles (Observed) Observed + Imputed 2019 to 2023 Mean Conditional Status (+ Imputed) 2023 Percentiles (+ Imputed) −0.38 −0.09 0.12 0.38 275 2019 to 2023 Growth Impact 275 2019 to 2023 Status Impact 2019 Grade 10 Scaled Score Decile Cuts 2019 and 2023 Mean Scaled Scores Observed 2 Lagging On Par Leading 2019 Grade 10 Scaled Score 225 225 Calculations suppressed 0.5 due to small group N (< 50) −1 Participation −1.5 Total Status Growth −2 2019: 33.0% Proficient 2019 42,847 42,847 (100.0%) 34,461 (80.4%) 2023: 32.2% Proficient 2023 43,171 43,171 (100.0%) 35,170 (81.5%) 2018 Grade 9 Scaled Score with Achievement Level Cuts Grade 10 Overall Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 10 Median SGPs by Quintile 2019 & 2023 Grade 10 Mean Scaled (Standardized) Scores by Quintile Lagging Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 10 Students 50.0 50.0 50.0 50.0 50.0 50.0 2019 Grade 10 Students 158 (−1.42) 185 (−0.49) 201 ( 0.04) 215 ( 0.52) 238 ( 1.32) 199 (0.00) Q1 Q2 Q3 Q4 Q5 2023 Grade 10 Students 29.0 26.0 32.0 39.0 53.0 36.0 2023 Grade 10 Students 138 (−2.09) 179 (−0.69) 197 (−0.09) 211 ( 0.40) 235 ( 1.23) 192 (−0.25) Lagging Lagging Lagging Lagging On Par Academic Impact (Observed) −21.0 −24.0 −18.0 −11.0 3.0 −14.0 Change (Observed) −20 (−0.67) −6 (−0.20) −4 (−0.13) −4 (−0.12) −3 (−0.09) −7 (−0.25) Academic Impact (+ Imputed) −24.0 −26.0 −20.0 −11.5 3.0 −17.0 Change (+ Imputed) −20 ( 0.10) −6 (−0.63) −4 (−1.09) −4 (−1.51) −3 (−2.19) −7 (−0.25) Count Observed 2019 6,927 6,789 7,043 6,843 6,859 34,461 Count Observed 2019 8,328 8,877 8,404 8,694 8,544 42,847 Count Observed 2023 6,942 7,082 6,974 6,994 7,178 35,170 Count Observed 2023 8,492 9,036 8,099 8,998 8,546 43,171 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 1.10 2023 to 2019 Grade 10 Status Change (Observed) Overall 30 30 1.05 2023 to 2019 Grade 10 Status Change (+ Imputed) 2023 to 2019 SGP Difference (+ Imputed) 1.00 0.95 0.39: Correlation SGP and Status Percentile Change Lagging Conditional Status/Growth Change 2023 to 2019 Conditional Status Difference (Observed) 0.90 0.36: Correlation SGP and Conditional Status Difference 0.80 15 15 0.50 Lagging Lagging Lagging Lagging Lagging 2019 to 2023 2018 Grade 9 Scaled Score with Decile & Achievement Level Cuts Grade 10 Percentile Ranks and Scaled Scores (2019 Norm)
  36. RISE ELA Grade 4 Utah 2019 to 2023 COVID⌧19 Academic Recovery ELL Students Growth: Grade 3/2018 to 4/2019 versus Grade 3/2022 to 4/2023 Status: Grade 4/2019 versus Grade 4/2023 Summary 2018 Grade 3 Scaled Score Decile Cuts All Students 10 20 30 40 50 60 70 80 90 2019 Grade 4 Scaled Score Grade 4 Scaled/Standardized Score and Achievement Level Cuts ELL Students 125 300 400 500 675 Growth/Status Recovery Legend 2018 to 2019 Mean Conditional Status 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 3.5 2023 Percentiles (Observed) 2019 to 2023 Mean Conditional Status (+ Imputed) 2023 Percentiles (+ Imputed) −0.38 −0.09 0.12 0.38 Observed + Imputed 2019 to 2023 Growth Impact 3 2019 to 2023 Status Impact 2019 and 2023 Mean Scaled Scores 2.5 Lagging On Par Leading 2019 Grade 4 Scaled Score Decile Cuts Observed 2019 Grade 4 Scaled Score 90 Calculations suppressed 80 due to small group N (< 50) 30 −0.67 Participation Total Status Growth 10 2019: 19.9% Proficient 2019 6,379 6,379 (100.0%) 5,912 (92.7%) −2.5 2023: 12.4% Proficient 2023 5,599 5,599 (100.0%) 4,277 (76.4%) 2018 Grade 3 Scaled Score with Achievement Level Cuts Grade 4 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 4 Median SGPs by Quintile 2019 & 2023 Grade 4 Mean Scaled (Standardized) Scores by Quintile Lagging Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 4 Students 43.0 39.0 41.0 39.0 45.0 41.0 2019 Grade 4 Students 256 (−0.67) 325 (−0.67) 369 (−0.67) 411 (−0.67) 470 (−0.67) 318 (−0.67) Q1 Q2 Q3 Q4 Q5 2023 Grade 4 Students 35.0 37.0 33.0 31.0 34.0 35.0 2023 Grade 4 Students 223 (−1.09) 312 (−1.09) 363 (−1.09) 407 (−1.09) 461 (−1.09) 285 (−1.09) Lagging On Par Lagging Lagging Lagging Academic Impact (Observed) −8.0 −2.0 −8.0 −8.0 −11.0 −6.0 Change (Observed) −32 (−0.42) −13 (−0.17) −6 (−0.08) −4 (−0.06) −9 (−0.12) −33 (−0.42) Academic Impact (+ Imputed) −8.0 −2.0 −8.0 −8.0 −11.0 −10.0 Change (+ Imputed) −32 (−0.49) −13 (−0.36) −6 (−0.34) −4 (−0.37) −9 (−0.51) −33 (−0.42) Count Observed 2019 2,469 1,524 1,054 554 311 5,912 Count Observed 2019 2,793 1,588 1,052 595 351 6,379 Count Observed 2023 1,938 1,148 691 391 109 4,277 Count Observed 2023 2,828 1,440 763 413 155 5,599 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 2023 to 2019 Grade 4 Status Change (Observed) Overall 30 2023 to 2019 SGP Difference (+ Imputed) 30 2023 to 2019 Grade 4 Status Change (+ Imputed) 0.35 0.88: Correlation SGP and Status Percentile Change Lagging Conditional Status/Growth Change 2023 to 2019 Conditional Status Difference (Observed) 0.99: Correlation SGP and Conditional Status Difference 0.30 15 15 0.20 Lagging Lagging Lagging Lagging Lagging 2019 to 2023 2018 Grade 3 Scaled Score with Decile & Achievement Level Cuts Grade 4 Percentile Ranks and Scaled Scores (2019 Norm)
  37. UA+ ELA Grade 9 Utah 2019 to 2023 COVID⌧19 Academic Recovery Female Students Growth: Grade 8/2018 to 9/2019 versus Grade 8/2022 to 9/2023 Status: Grade 9/2019 versus Grade 9/2023 Summary 2018 Grade 8 Scaled Score Decile Cuts All Students 10 20 30 40 50 60 70 80 90 2019 Grade 9 Scaled Score Grade 9 Scaled/Standardized Score and Achievement Level Cuts Female Students 300 Growth/Status Recovery Legend 2018 to 2019 Mean Conditional Status 3.5 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 2023 Percentiles (Observed) Observed + Imputed 2019 to 2023 Mean Conditional Status (+ Imputed) 2023 Percentiles (+ Imputed) −0.38 −0.09 0.12 0.38 275 2019 to 2023 Growth Impact 275 2019 to 2023 Status Impact 2019 Grade 9 Scaled Score Decile Cuts 2019 and 2023 Mean Scaled Scores Observed Lagging On Par Leading 2019 Grade 9 Scaled Score 225 225 1 Calculations suppressed 70 0.5 due to small group N (< 50) Participation Total Status Growth 2019: 53.9% Proficient 2019 21,787 21,787 (100.0%) 19,671 (90.3%) −3 2023: 48.4% Proficient 2023 21,713 21,713 (100.0%) 18,437 (84.9%) 2018 Grade 8 Scaled Score with Achievement Level Cuts Grade 9 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 9 Median SGPs by Quintile 2019 & 2023 Grade 9 Mean Scaled (Standardized) Scores by Quintile On Par Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 9 Students 51.0 50.0 48.0 47.0 49.0 49.0 2019 Grade 9 Students 163 (0.16) 187 (0.16) 201 (0.16) 214 (0.16) 235 (0.16) 204 (0.16) Q1 Q2 Q3 Q4 Q5 2023 Grade 9 Students 55.0 47.0 46.0 45.0 46.0 48.0 2023 Grade 9 Students 159 (0.02) 183 (0.02) 199 (0.02) 212 (0.02) 233 (0.02) 200 (0.02) On Par On Par On Par On Par On Par Academic Impact (Observed) 4.0 −3.0 −2.0 −2.0 −3.0 −1.0 Change (Observed) −4 (−0.16) −4 (−0.16) −3 (−0.10) −2 (−0.07) −2 (−0.07) −3 (−0.13) Academic Impact (+ Imputed) 4.0 −3.0 −2.0 −1.6 −3.0 −2.0 Change (+ Imputed) −4 ( 0.16) −4 (−0.64) −3 (−1.09) −2 (−1.52) −2 (−2.23) −3 (−0.13) Count Observed 2019 2,743 3,670 4,216 4,346 4,696 19,671 Count Observed 2019 3,190 4,061 4,688 4,766 5,082 21,787 Count Observed 2023 2,836 3,514 3,885 3,977 4,225 18,437 Count Observed 2023 3,385 4,311 4,465 4,483 5,069 21,713 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 1.20 2023 to 2019 Grade 9 Status Change (Observed) Overall 30 2023 to 2019 SGP Difference (+ Imputed) 30 1.15 2023 to 2019 Grade 9 Status Change (+ Imputed) 1.05 −0.25: Correlation SGP and Status Percentile Change Lagging Conditional Status/Growth Change 2023 to 2019 Conditional Status Difference (Observed) 1.00 0.97: Correlation SGP and Conditional Status Difference 0.90 15 15 0.60 Lagging Lagging Lagging On Par On Par 2019 to 2023 2018 Grade 8 Scaled Score with Decile & Achievement Level Cuts Grade 9 Percentile Ranks and Scaled Scores (2019 Norm)
  38. UA+ ELA Grade 9 Utah 2019 to 2023 COVID⌧19 Academic Recovery Male Students Growth: Grade 8/2018 to 9/2019 versus Grade 8/2022 to 9/2023 Status: Grade 9/2019 versus Grade 9/2023 Summary 2018 Grade 8 Scaled Score Decile Cuts All Students 10 20 30 40 50 60 70 80 90 2019 Grade 9 Scaled Score Grade 9 Scaled/Standardized Score and Achievement Level Cuts Male Students 300 Growth/Status Recovery Legend 2018 to 2019 Mean Conditional Status 3.5 2019 Percentiles (Observed) −30 −7 10 30 2022 to 2023 Mean Conditional Status (Observed) 2023 Percentiles (Observed) Observed + Imputed 2019 to 2023 Mean Conditional Status (+ Imputed) 2023 Percentiles (+ Imputed) −0.38 −0.09 0.12 0.38 275 2019 to 2023 Growth Impact 275 2019 to 2023 Status Impact 2019 Grade 9 Scaled Score Decile Cuts 2019 and 2023 Mean Scaled Scores Observed Lagging On Par Leading 2019 Grade 9 Scaled Score 225 225 1 Calculations suppressed 70 0.5 due to small group N (< 50) Participation Total Status Growth 2019: 43.1% Proficient 2019 22,554 22,554 (100.0%) 20,600 (91.3%) −3 2023: 39.9% Proficient 2023 23,793 23,793 (100.0%) 20,650 (86.8%) 2018 Grade 8 Scaled Score with Achievement Level Cuts Grade 9 Overall and Subgroup Percentile Ranks Growth Recovery: Overall and by Quintile Overall 2019 & 2023 Grade 9 Median SGPs by Quintile 2019 & 2023 Grade 9 Mean Scaled (Standardized) Scores by Quintile Lagging Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 9 Students 50.0 50.0 52.0 53.0 51.0 51.0 2019 Grade 9 Students 160 (−0.14) 187 (−0.14) 201 (−0.14) 214 (−0.14) 234 (−0.14) 196 (−0.14) Q1 Q2 Q3 Q4 Q5 2023 Grade 9 Students 50.0 44.0 43.0 42.0 44.0 45.0 2023 Grade 9 Students 157 (−0.23) 183 (−0.23) 199 (−0.23) 212 (−0.23) 232 (−0.23) 194 (−0.23) On Par On Par Lagging Lagging Lagging Academic Impact (Observed) 0.0 −6.0 −9.0 −11.0 −7.0 −6.0 Change (Observed) −3 (−0.12) −4 (−0.15) −2 (−0.09) −2 (−0.07) −2 (−0.08) −2 (−0.09) Academic Impact (+ Imputed) 0.0 −6.0 −9.0 −11.0 −7.0 −7.0 Change (+ Imputed) −3 ( 0.27) −4 (−0.62) −2 (−1.09) −2 (−1.52) −2 (−2.20) −2 (−0.09) Count Observed 2019 5,309 4,356 3,858 3,741 3,336 20,600 Count Observed 2019 5,723 4,444 4,367 4,169 3,851 22,554 Count Observed 2023 4,973 4,269 4,014 3,801 3,593 20,650 Count Observed 2023 5,805 4,888 4,602 4,233 4,265 23,793 Status Recovery: Overall and by Quintile 2023 to 2019 Scaled/Standardized Score Difference 2023 to 2019 SGP Difference (Observed) 1.20 2023 to 2019 Grade 9 Status Change (Observed) Overall 30 2023 to 2019 SGP Difference (+ Imputed) 30 1.15 2023 to 2019 Grade 9 Status Change (+ Imputed) 1.05 −0.65: Correlation SGP and Status Percentile Change On Par Conditional Status/Growth Change 2023 to 2019 Conditional Status Difference (Observed) 1.00 0.99: Correlation SGP and Conditional Status Difference 0.90 15 15 0.60 Lagging Lagging On Par On Par On Par 2019 to 2023 2018 Grade 8 Scaled Score with Decile & Achievement Level Cuts Grade 9 Percentile Ranks and Scaled Scores (2019 Norm)
  39. Recovery Year 2 • Analysis of 2023 RISE data indicates two things: • Rates of learning, on average, are slightly exceeding or at pre-pandemic rates of growth. That is, the system has recovered, and then some. • Student attainment, on average, is still lagging behind because rates of learning have not accelerated enough to catch students up. That is, students have not fully “recovered” but levels are, in most grades, close to pre-pandemic levels. • In both ELA and mathematics, learning acceleration from the lows of the pandemic would be considered, on average. That is, the system has stabilized.
  40. Questions