Pandemic Related Academic Impact in 2021
What do state summative assessments tell us
Damian Betebenner & Adam Van Iwaarden
Center for Assessment
Rhode Island TAC Meeting
November 3rd, 2021/Updated December 9th, 2021
Conceptualizing Academic Impact
• The pandemic and all the ensuing disruptions functioned/
functions as an “academic headwind”, impeding (in general) the
academic progress of students.
• Headwinds impede progress in two ways:
• They slow one’s rate of progress
• And by slowing one’s rate of progress they lead to less distance being
travelled.
• In education these two impediments manifest as:
• Decrease in student growth.
• Decrease in student attainment.
RI Academic Impact Discussion 2021
RICAS SGP Operational Considerations in 2021
• Due to no RICAS testing in Spring 2020, RICAS growth analyses will be
skip-year/gap year growth from 2019 to 2021.
• Two types of SGP analyses were conducted for RICAS data:
• Cohort referenced analyses: These analyses create growth norms from the
2019 to 2021 students. These growth norms yield cohort means/median
SGPs of 50 and obscure COVID-19 academic impact.
• Baseline referenced analyses: Historical growth norms were created using
2019 and prior data. These growth growth analyses don’t have mean/median
SGPs of 50 and can be used to investigate overall COVID-19 academic impact
• Projections (lagged and straight) were calculated for 2021 data using
baseline growth norms in case they were needed.
RI Academic Impact Discussion 2021
WIDA-ACCESS SGP Operational Considerations in 2021
• Two types of SGP analyses were conducted for WIDA-ACCESS data:
• Cohort referenced analyses: These analyses create growth norms from the
2020 to 2021 students. These growth norms yield cohort mean/median SGPs
of 50 and obscure COVID-19 academic impact.
• Baseline referenced analyses: Historical growth norms were created using
2020 and prior data. These growth growth analyses don’t have mean/median
SGPs of 50 and can be used to investigate overall COVID-19 academic impact.
• Use composite score from WIDA-ACCESS data. No analyses performed
using sub-scores.
• Projections (lagged and straight) were calculated for 2021 data using
both cohort and baseline growth norms in case they were needed.
RI Academic Impact Discussion 2021
SGP Summary of Cohort versus Baseline Referenced
• Cohort referenced SGPs
• Growth norms created from the current cohort of students.
• Mean/median will by 50 for all grade x content area analyses
• Cannot investigate overall impact of the pandemic on students.
• Can investigate differential impact of the pandemic on students.
• Does not require scale stability across years.
• Baseline referenced SGPs.
• Growth norms created from historical (i.e., pre-COVID) cohorts of students.
• Mean/median 50 for historical cohort, not necessarily 50 for current cohort.
• Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related
academic impact.
• Mean SGP deviation of 5 corresponds to an effect size of 0.18.
• Can investigate BOTH overall impact and differential impact
• Requires scale stability across years.
RI Academic Impact Discussion 2021
SGP Assumptions
• SGP analyses presume valid scale scores
• Students forming the growth norm group are presumed to have taken the same (or
equivalent) test at the same time. Scores have to mean the same thing.
• SGPs, however, are often used as a post-hoc check of scale score validity/
comparability via Reductio ad absurdum
• In 2021, check form comparability and mode-of-administration comparability.
• RIDE has extensive experience (e.g., PARCC paper/pencil versus online) investigating
and accommodating mode-effects.
• If scores are not comparable then we either attempt to make them comparable or
run separate analyses.
• We assume something close to population invariance with respect to IRT
scaling.
• The population of students tested in 2021 and differences to 2019 is a concern
so we begin with data on the population of students tested.
RI Academic Impact Discussion 2021
RICAS Participation
ELA Mathematics
Grade
RI Academic Impact Discussion 2021
WIDA-ACCESS Participation
Overall Overall
Grade Grade
RI Academic Impact Discussion 2021
SGP Analyses for 2021
• With comparable results we can quantify pandemic related
academic impact:
• At the state level
• Demographic subgroups including ethnicity, poverty
• Academic subgroups (special education, low/high achievers)
• By region/district/school
• By mode of education (remote/hybrid/in-person).
• Attendance
• English Language Learner impact can be evaluated using WIDA-ACCESS
results.
RI Academic Impact Discussion 2021
RICAS Percent Proficient by Grade
ELA Mathematics
Grade
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Grade
ELA Mathematics
Grade
3 NA NA NA NA
4 NA NA NA NA
RI Academic Impact Discussion 2021
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: ELA Grade 5
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8670)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
RI Academic Impact Discussion 2021
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: Mathematics Grade 5
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8737)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
RI Academic Impact Discussion 2021
2018−2019 Scale Score with Achievement Level Cuts
Discussion
• Median baseline SGPs in ELA were in the mid 30s and median baseline SGPs in
mathematics were lower (upper teens to lower 30s).
• Deviation from normal growth (i.e., 50) represents the extent to which
academic growth slowed down from 2019 to 2021, that is, academic impact.
• Larger academic impact in ELA than mathematics.
• Academic impact is fairly uniform across grades.
• Academic impact is very large likely requiring multiple years and additional
educational supports/interventions to recover.
• Recovery implies getting back (at a minimum) to where 50th percentile growth would
have led to.
• Median/mean growth below 50 must be averaged out with median/mean growth
above 50 going forward. E.g. A mean two-year SGP of 35 would need to be followed
by mean two-year SGP of 65 in order to get an average of 50.
RI Academic Impact Discussion 2021
WIDA-ACCESS Median/Mean SGPs by Grade
Overall Overall
Grade Grade
K NA NA 6 50.0/50.2 32.0/38.4 (887)
RI Academic Impact Discussion 2021
Discussion of Grade Level Results
• Median baseline SGPs are low in elementary school grades and
are close to 50 in the high school grades.
• Deviation from normal growth (i.e., 50) represents the extent to
which academic growth slowed down from 2020 to 2021, that is,
academic impact.
• Academic impact is very large in elementary and middle school
and close to non-existent in high school.
• Lack of impact is consistent with:
• Great academic support for the group during the pandemic.
• Impoverished learning conditions prior to the pandemic such that the pandemic
didn’t alter the status quo.
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Ethnicity
ELA Mathematics
Ethnicity
Am. In./AL Native 44.0/44.3 (259) 29.5/35.2 (218) 43.0/45.3 (263) 18.0/26.2 (221)
Asian 54.0/53.5 (1,175) 41.0/44.6 (1,081) 56.0/55.0 (1,204) 33.0/39.1 (1,095)
African American 41.0/43.6 (3,145) 31.0/35.8 (2,890) 42.0/45.1 (3,202) 20.0/27.0 (2,901)
Hispanic or Lat. 46.0/46.8 (9,875) 31.0/36.2 (9,001) 44.0/46.1 (10,136) 20.0/27.5 (9,116)
Multiple Ethnicities 47.0/47.8 (1,755) 34.0/37.3 (1,675) 47.0/47.9 (1,763) 23.0/30.6 (1,651)
Native HI/PI 46.0/46.5 (51) 33.0/35.3 (49) 53.0/50.4 (53) 24.0/30.7 (47)
No Race reported 47.0/45.9 (8) 28.5/33.4 (132) 46.5/41.7 (6) 14.0/21.3 (128)
White 53.0/52.4 (22,316) 39.0/42.3 (18,733) 54.0/52.6 (22,358) 29.0/35.2 (18,598)
RI Academic Impact Discussion 2021
Discussion of Ethnicity Results
• Broken out by ethnicity, 2021 baseline SGP results show similar
rank ordering (i.e., growth gaps) as seen in normal years.
• Student academic growth is lower in mathematics than in ELA
for all ethnic subgroups.
• Even though growth is lower for ethnic minorities, academic
impacts (growth difference with 2019) are uniform across
ethnicity subgroups.
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Free/Reduced
Lunch Status
ELA Mathematics
Free/Reduced Lunch
Status
FRL Yes 44.0/45.9 (18,477) 30.0/35.0 (14,479) 45.0/46.4 (18,820) 20.0/27.2 (14,541)
FRL No 55.0/53.7 (20,107) 41.0/43.4 (19,300) 55.0/53.6 (20,165) 30.0/36.0 (19,216)
RI Academic Impact Discussion 2021
Discussion of Free/Reduced Lunch Results
• Baseline SGP results show similar growth gaps in 2021 (students
indicated as receiving free/reduced lunch have lower SGPs than
students indicated as not receiving free/reduced lunch.
• Academic impacts are uniform across the subgroups.
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by IEP Status
ELA Mathematics
IEP Status
Non-IEP 51.0/50.9 (31,158) 37.0/40.8 (26,913) 51.0/50.8 (31,552) 25.0/32.3 (26,939)
IEP 42.0/44.3 (5,398) 29.0/35.0 (4,802) 43.0/45.0 (5,408) 25.0/31.3 (4,780)
Students with 504 52.0/50.8 (2,028) 35.0/39.2 (2,064) 53.0/52.0 (2,025) 25.0/32.4 (2,038)
RI Academic Impact Discussion 2021
Discussion of IEP Results
• Baseline SGP results show similar growth gaps in 2021 (students
indicated as receiving free/reduced lunch have lower SGPs than
students indicated as not receiving free/reduced lunch.
• Academic impacts are, in general, uniform across the subgroups:
• Academic impact for IEP group in mathematics is smaller than the
other two groups.
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by ELL Status
ELA Mathematics
ELL Status
ELL No 51.0/50.3 (36,261) 36.0/40.3 (30,942) 51.0/50.4 (36,249) 25.0/32.5 (30,607)
ELL Yes 42.0/44.7 (2,323) 30.0/34.9 (2,837) 43.0/45.5 (2,736) 23.0/28.8 (3,150)
RI Academic Impact Discussion 2021
Discussion of ELL Results
• Baseline SGP results show somewhat similar growth gaps for ELA
subgroups between 2019 and 2021.
• The results for both ELA and mathematics show slightly less of
an impact for ELL students than for their non-ELL counterparts,
though the difference is practically insignificant.
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Gender
ELA Mathematics
Gender
Female 54.0/53.0 (18,917) 38.0/41.4 (16,438) 51.0/50.6 (19,137) 25.0/32.5 (16,375)
Male 46.0/47.1 (19,659) 34.0/38.4 (17,167) 49.0/49.5 (19,842) 25.0/32.0 (17,205)
Other --- 32.5/38.1 (42) --- 16.0/18.1 (49)
RI Academic Impact Discussion 2021
Discussion of Gender Results
• Growth gender gaps historically exist (favoring females) in both
ELA and mathematics.
• Growth gender gaps are historically larger in ELA than
mathematics.
• Gaps are smaller in 2021 between males and females suggesting
the females might have been slightly more academically
impacted than males
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 5 ELA
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 6 ELA
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 7 ELA
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 8 ELA
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 5 Mathematics
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 6 Mathematics
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 7 Mathematics
RI Academic Impact Discussion 2021
RICAS Median/Mean Skip-Year SGPs by Prior Achievement:
Grade 8 Mathematics
RI Academic Impact Discussion 2021
Discussion of Growth by Prior Achievement Results
• In some grades and content areas median SGP baselines vary by the
prior achievement level of students, sometimes substantially.
• For example, in Grade 6 ELA lower achieving students had much lower
baseline SGPs than higher achieving students. This implies that those
students had much greater academic impact.
• In Grade 7 mathematics higher achieving students had lower baseline
SGPs than lower achieving students implying that higher achieving
students were more impacted.
• In general, there is a fairly uniform impact by prior achievement level.
RI Academic Impact Discussion 2021
Participation Rates and Interpretation
• An issue impacting the interpretation of growth (and status) results in
2021 is participation in the state assessment.
• Low participation doesn’t impact calculation of cohort referenced
growth norms nor use of baseline referenced growth norms.
• Low participation does impact comparisons of group level results with
historical (e.g., 2019) results based upon higher participation rates.
• For example, how should one compare growth and status results for
school in 2019 and 2021 with very different rates of participation.
RI Academic Impact Discussion 2021
School Level Results
• Due to larger fluctuations in participation rates a the school level, growth
comparisons between 2019 and 2021 must be approached more cautiously.
• In general we have found via multiple imputation that comparisons of status
and growth results from 2019 to 2021 are defensible.
• Whether a school’s results support cross-year comparison depends more on
the size of the school than on the participation rate/enrollment change.
• The following are academic impact results at the school level:
• For grades 3 and 4, academic impact was determined via status change analyses.
• For grades 5, 6, 7, and 8 academic impact was determined via growth change.
• Summarization across grades within a school was done by averaging ordinal levels of
impact (Severe 5, Large 4, Moderate 3, Modest to None 2, and Improvement 1)
RI Academic Impact Discussion 2021
Academic Impact by 2018-2019 Star Rating: ELA/Count
Severe Large Moderate Modest/None Improvement
1 Star 2 Star 3 Star 4 Star 5 Star
RI Academic Impact Discussion 2021
Academic Impact by 2018-2019 Star Rating: ELA/
Percentage
Severe Large Moderate Modest/None Improvement
1 Star 2 Star 3 Star 4 Star 5 Star
RI Academic Impact Discussion 2021
Academic Impact by 2018-2019 Star Rating: Mathematics/
Count
Severe Large Moderate Modest/None Improvement
1 Star 2 Star 3 Star 4 Star 5 Star
RI Academic Impact Discussion 2021
Academic Impact by 2018-2019 Star Rating: Mathematics/
Percentage
Severe Large Moderate Modest/None Improvement
1 Star 2 Star 3 Star 4 Star 5 Star
RI Academic Impact Discussion 2021
SGP Analysis Plans post 2021 (2022 and beyond)
• Spring 2021 represents an ideal time to get a snapshot of student
attainment as we are hopefully/presumably at or near the point
of maximal impact on student attainment.
• That is, recovery has not really begun for most students in
earnest.
• It’s impossible to talk about recovery sensibly without
understanding how big the impact is.
• Monitoring recovery will be at least as critical as monitoring the impact of
the pandemic on student achievement.
• Beware the K-shaped recovery!
RI Academic Impact Discussion 2021
Summary of Findings
• Academic impact analyses were successfully conducted for both RICAS and
WIDA-ACCESS.
• Results of the analyses suggest large academic impacts on students due to
the pandemic.
• Results are larger in math and in ELA
• Results are uniformly large across most demographic subgroups.
• Results show variation across academic subgroups (e.g., high versus low
achievers in some content areas).
• Academic impact varied greatly by schools and also varied by 2018-2019 STAR
Rating.
• Recovery from impacts this large will likely take multiple years and require
additional supports beyond that provided by normal schooling.
RI Academic Impact Discussion 2021
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: ELA Grade 6
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8277)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: ELA Grade 7
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8386)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: ELA Grade 8
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8440)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: Mathematics Grade 6
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8280)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: Mathematics Grade 7
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8344)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
Rhode_Island RICAS 2020−2021 Status by 2018−2019 Prior Status
All Students: Mathematics Grade 8
2018−2019 & 2020−2021 Median SGPs by Scale Score Decile Overall (N=8391)
2018−2019 Scale Score Decile Cuts
2018−2019 to 2020−2021 Mean Conditional Status
2016−2016_2017 to 2018−2019 Mean Conditional Status
2.5 2018−2019 to 2020−2021 OLS Conditional Status
2018−2019 to 2020−2021 Baseline P50
2.0 2016_2017−2018 to 2018−2019 Mean Conditional Status
2020−2021 Scale Score
Academic Impact
2018−2019 to 2020−2021 Conditional Status Academic Impact
−0.4 2018−2019 to 2020−2021 Variance Adjusted Academic Impact
2018−2019 Scale Score with Achievement Level Cuts
About this copy
Pandemic Related Academic Impact in 2021: What do state summative assessments tell us — Rhode Island Department of Education, December 9, 2021.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.