Pandemic Related Academic Impact in 2021: What do state summative assessments tell us

Rhode Island Technical Advisory Committee · November 3, 2021

Pandemic Related Academic Impact in 2021: What do state summative assessments tell us — Rhode Island Technical Advisory Committee, November 3, 2021.
Published

November 3, 2021

Event
Technical advisory committee
Slides
46
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Pandemic Related Academic Impact in 2021 What do state summative assessments tell us Damian Betebenner & Adam Van Iwaarden Center for Assessment Rhode Island TAC Meeting November 3rd, 2021
  2. Conceptualizing Academic Impact • The pandemic and all the ensuing disruptions functioned/ functions as an “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • And by slowing one’s rate of progress they lead to less distance being travelled. • In education these two impediments manifest as: • Decrease in student growth. • Decrease in student attainment. RI Academic Impact Discussion 2021
  3. RICAS SGP Operational Considerations in 2021 • Due to no RICAS testing in Spring 2020, RICAS growth analyses will be skip-year/gap year growth from 2019 to 2021. • Two types of SGP analyses were conducted for RICAS data: • Cohort referenced analyses: These analyses create growth norms from the 2019 to 2021 students. These growth norms yield cohort means/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2019 and prior data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact • Projections (lagged and straight) were calculated for 2021 data using baseline growth norms in case they were needed. RI Academic Impact Discussion 2021
  4. WIDA-ACCESS SGP Operational Considerations in 2021 • Two types of SGP analyses were conducted for WIDA-ACCESS data: • Cohort referenced analyses: These analyses create growth norms from the 2020 to 2021 students. These growth norms yield cohort mean/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2020 and prior data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact. • Use composite score from WIDA-ACCESS data. No analyses performed using sub-scores. • Projections (lagged and straight) were calculated for 2021 data using both cohort and baseline growth norms in case they were needed. RI Academic Impact Discussion 2021
  5. SGP Summary of Cohort versus Baseline Referenced • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Mean/median will by 50 for all grade x content area analyses • Cannot investigate overall impact of the pandemic on students. • Can investigate differential impact of the pandemic on students. • Does not require scale stability across years. • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related academic impact. • Mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact and differential impact • Requires scale stability across years. RI Academic Impact Discussion 2021
  6. SGP Assumptions • SGP analyses presume valid scale scores • Students forming the growth norm group are presumed to have taken the same (or equivalent) test at the same time. Scores have to mean the same thing. • SGPs, however, are often used as a post-hoc check of scale score validity/ comparability via Reductio ad absurdum • In 2021, check form comparability and mode-of-administration comparability. • RIDE has extensive experience (e.g., PARCC paper/pencil versus online) investigating and accommodating mode-effects. • If scores are not comparable then we either attempt to make them comparable or run separate analyses. • We assume something close to population invariance with respect to IRT scaling. • The population of students tested in 2021 and differences to 2019 is a concern so we begin with data on the population of students tested. RI Academic Impact Discussion 2021
  7. RICAS Participation ELA Mathematics Grade RI Academic Impact Discussion 2021
  8. WIDA-ACCESS Participation Overall Overall Grade Grade RI Academic Impact Discussion 2021
  9. SGP Analyses for 2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • Demographic subgroups including ethnicity, poverty • Academic subgroups (special education, low/high achievers) • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • English Language Learner impact can be evaluated using WIDA-ACCESS results. RI Academic Impact Discussion 2021
  10. RICAS Percent Proficient by Grade ELA Mathematics Grade RI Academic Impact Discussion 2021
  11. RICAS Percent Proficient by Grade ELA Mathematics Grade 2019 2021 Difference Effect-Size 2019 2021 Difference Effect-Size RI Academic Impact Discussion 2021
  12. RICAS Median/Mean Skip-Year SGPs by Grade ELA Mathematics Grade 3 NA NA NA NA 4 NA NA NA NA RI Academic Impact Discussion 2021
  13. Discussion • Median baseline SGPs in ELA were in the mid 30s and median baseline SGPs in mathematics were lower (upper teens to lower 30s). • Deviation from normal growth (i.e., 50) represents the extent to which academic growth slowed down from 2019 to 2021, that is, academic impact. • Larger academic impact in ELA than mathematics. • Academic impact is fairly uniform across grades. • Academic impact is very large likely requiring multiple years and additional educational supports/interventions to recover. • Recovery implies getting back (at a minimum) to where 50th percentile growth would have led to. • Median/mean growth below 50 must be averaged out with median/mean growth above 50 going forward. E.g. A mean two-year SGP of 35 would need to be followed by mean two-year SGP of 65 in order to get an average of 50. RI Academic Impact Discussion 2021
  14. WIDA-ACCESS Median/Mean SGPs by Grade Overall Overall Grade Grade K NA NA 6 50.0/50.2 32.0/38.4 (887) RI Academic Impact Discussion 2021
  15. Discussion of Grade Level Results • Median baseline SGPs are low in elementary school grades and are close to 50 in the high school grades. • Deviation from normal growth (i.e., 50) represents the extent to which academic growth slowed down from 2020 to 2021, that is, academic impact. • Academic impact is very large in elementary and middle school and close to non-existent in high school. • Lack of impact is consistent with: • Great academic support for the group during the pandemic. • Impoverished learning conditions prior to the pandemic such that the pandemic didn’t alter the status quo. RI Academic Impact Discussion 2021
  16. RICAS Median/Mean Skip-Year SGPs by Ethnicity ELA Mathematics Ethnicity Am. In./AL Native 44.0/44.3 (259) 29.5/35.2 (218) 43.0/45.3 (263) 18.0/26.2 (221) Asian 54.0/53.5 (1,175) 41.0/44.6 (1,081) 56.0/55.0 (1,204) 33.0/39.1 (1,095) African American 41.0/43.6 (3,145) 31.0/35.8 (2,890) 42.0/45.1 (3,202) 20.0/27.0 (2,901) Hispanic or Lat. 46.0/46.8 (9,875) 31.0/36.2 (9,001) 44.0/46.1 (10,136) 20.0/27.5 (9,116) Multiple Ethnicities 47.0/47.8 (1,755) 34.0/37.3 (1,675) 47.0/47.9 (1,763) 23.0/30.6 (1,651) Native HI/PI 46.0/46.5 (51) 33.0/35.3 (49) 53.0/50.4 (53) 24.0/30.7 (47) No Race reported 47.0/45.9 (8) 28.5/33.4 (132) 46.5/41.7 (6) 14.0/21.3 (128) White 53.0/52.4 (22,316) 39.0/42.3 (18,733) 54.0/52.6 (22,358) 29.0/35.2 (18,598) RI Academic Impact Discussion 2021
  17. Discussion of Ethnicity Results • Broken out by ethnicity, 2021 baseline SGP results show similar rank ordering (i.e., growth gaps) as seen in normal years. • Student academic growth is lower in mathematics than in ELA for all ethnic subgroups. • Even though growth is lower for ethnic minorities, academic impacts (growth difference with 2019) are uniform across ethnicity subgroups. RI Academic Impact Discussion 2021
  18. RICAS Median/Mean Skip-Year SGPs by Free/Reduced Lunch Status ELA Mathematics Free/Reduced Lunch Status FRL Yes 44.0/45.9 (18,477) 30.0/35.0 (14,479) 45.0/46.4 (18,820) 20.0/27.2 (14,541) FRL No 55.0/53.7 (20,107) 41.0/43.4 (19,300) 55.0/53.6 (20,165) 30.0/36.0 (19,216) RI Academic Impact Discussion 2021
  19. Discussion of Free/Reduced Lunch Results • Baseline SGP results show similar growth gaps in 2021 (students indicated as receiving free/reduced lunch have lower SGPs than students indicated as not receiving free/reduced lunch. • Academic impacts are uniform across the subgroups. RI Academic Impact Discussion 2021
  20. RICAS Median/Mean Skip-Year SGPs by IEP Status ELA Mathematics IEP Status Non-IEP 51.0/50.9 (31,158) 37.0/40.8 (26,913) 51.0/50.8 (31,552) 25.0/32.3 (26,939) IEP 42.0/44.3 (5,398) 29.0/35.0 (4,802) 43.0/45.0 (5,408) 25.0/31.3 (4,780) Students with 504 52.0/50.8 (2,028) 35.0/39.2 (2,064) 53.0/52.0 (2,025) 25.0/32.4 (2,038) RI Academic Impact Discussion 2021
  21. Discussion of IEP Results • Baseline SGP results show similar growth gaps in 2021 (students indicated as receiving free/reduced lunch have lower SGPs than students indicated as not receiving free/reduced lunch. • Academic impacts are, in general, uniform across the subgroups: • Academic impact for IEP group in mathematics is smaller than the other two groups. RI Academic Impact Discussion 2021
  22. RICAS Median/Mean Skip-Year SGPs by ELL Status ELA Mathematics ELL Status ELL No 51.0/50.3 (36,261) 36.0/40.3 (30,942) 51.0/50.4 (36,249) 25.0/32.5 (30,607) ELL Yes 42.0/44.7 (2,323) 30.0/34.9 (2,837) 43.0/45.5 (2,736) 23.0/28.8 (3,150) RI Academic Impact Discussion 2021
  23. Discussion of ELL Results • Baseline SGP results show somewhat similar growth gaps for ELA subgroups between 2019 and 2021. • The results for both ELA and mathematics show slightly less of an impact for ELL students than for their non-ELL counterparts, though the difference is practically insignificant. RI Academic Impact Discussion 2021
  24. RICAS Median/Mean Skip-Year SGPs by Gender ELA Mathematics Gender Female 54.0/53.0 (18,917) 38.0/41.4 (16,438) 51.0/50.6 (19,137) 25.0/32.5 (16,375) Male 46.0/47.1 (19,659) 34.0/38.4 (17,167) 49.0/49.5 (19,842) 25.0/32.0 (17,205) Other --- 32.5/38.1 (42) --- 16.0/18.1 (49) RI Academic Impact Discussion 2021
  25. Discussion of Gender Results • Growth gender gaps historically exist (favoring females) in both ELA and mathematics. • Growth gender gaps are historically larger in ELA than mathematics. • Gaps are smaller in 2021 between males and females suggesting the females might have been slightly more academically impacted than males RI Academic Impact Discussion 2021
  26. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 ELA RI Academic Impact Discussion 2021
  27. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 ELA RI Academic Impact Discussion 2021
  28. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 ELA RI Academic Impact Discussion 2021
  29. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 ELA RI Academic Impact Discussion 2021
  30. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 5 Mathematics RI Academic Impact Discussion 2021
  31. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 6 Mathematics RI Academic Impact Discussion 2021
  32. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 7 Mathematics RI Academic Impact Discussion 2021
  33. RICAS Median/Mean Skip-Year SGPs by Prior Achievement: Grade 8 Mathematics RI Academic Impact Discussion 2021
  34. Discussion of Growth by Prior Achievement Results • In some grades and content areas median SGP baselines vary by the prior achievement level of students, sometimes substantially. • For example, in Grade 6 ELA lower achieving students had much lower baseline SGPs than higher achieving students. This implies that those students had much greater academic impact. • In Grade 7 mathematics higher achieving students had lower baseline SGPs than lower achieving students implying that higher achieving students were more impacted. • In general, there is a fairly uniform impact by prior achievement level. RI Academic Impact Discussion 2021
  35. Participation Rates and Interpretation • An issue impacting the interpretation of growth (and status) results in 2021 is participation in the state assessment. • Low participation doesn’t impact calculation of cohort referenced growth norms nor use of baseline referenced growth norms. • Low participation does impact comparisons of group level results with historical (e.g., 2019) results based upon higher participation rates. • For example, how should one compare growth and status results for school in 2019 and 2021 with very different rates of participation. RI Academic Impact Discussion 2021
  36. School Level Results • Due to larger fluctuations in participation rates a the school level, growth comparisons between 2019 and 2021 must be approached more cautiously. • In general we have found via multiple imputation that comparisons of status and growth results from 2019 to 2021 are defensible. • Whether a school’s results support cross-year comparison depends more on the size of the school than on the participation rate/enrollment change. • The following are academic impact results at the school level: • For grades 3 and 4, academic impact was determined via status change analyses. • For grades 5, 6, 7, and 8 academic impact was determined via growth change. • Summarization across grades within a school was done by averaging ordinal levels of impact (Severe 5, Large 4, Moderate 3, Modest to None 2, and Improvement 1) RI Academic Impact Discussion 2021
  37. Academic Impact by 2018-2019 Star Rating: ELA/Count Severe Large Moderate Modest/None Improvement 1 Star 2 Star 3 Star 4 Star 5 Star RI Academic Impact Discussion 2021
  38. Academic Impact by 2018-2019 Star Rating: ELA/ Percentage Severe Large Moderate Modest/None Improvement 1 Star 2 Star 3 Star 4 Star 5 Star RI Academic Impact Discussion 2021
  39. Academic Impact by 2018-2019 Star Rating: Mathematics/ Count Severe Large Chart Title Moderate Modest/None Improvement 1 Star 2 Star 3 Star 4 Star 5 Star RI Academic Impact Discussion 2021
  40. Academic Impact by 2018-2019 Star Rating: Mathematics/ Percentage Severe Large Chart Title Moderate Modest/None Improvement 1 Star 2 Star 3 Star 4 Star 5 Star RI Academic Impact Discussion 2021
  41. SGP Analysis Plans post 2021 (2022 and beyond) • Spring 2021 represents an ideal time to get a snapshot of student attainment as we are hopefully/presumably at or near the point of maximal impact on student attainment. • That is, recovery has not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery! RI Academic Impact Discussion 2021
  42. Summary of Findings • Academic impact analyses were successfully conducted for both RICAS and WIDA-ACCESS. • Results of the analyses suggest large academic impacts on students due to the pandemic. • Results are larger in math and in ELA • Results are uniformly large across most demographic subgroups. • Results show variation across academic subgroups (e.g., high versus low achievers in some content areas). • Academic impact varied greatly by schools and also varied by 2018-2019 STAR Rating. • Recovery from impacts this large will likely take multiple years and require additional supports beyond that provided by normal schooling. RI Academic Impact Discussion 2021
  43. SGP Goodness of Fit Plots: RICAS ELA RI Academic Impact Discussion 2021
  44. SGP Goodness of Fit Plots: RICAS ELA RI Academic Impact Discussion 2021
  45. SGP Goodness of Fit Plots: RICAS Mathematics RI Academic Impact Discussion 2021
  46. SGP Goodness of Fit Plots: RICAS Mathematics RI Academic Impact Discussion 2021