COVID-19 Academic Impact in Rhode Island
Damian Betebenner & Adam Van Iwaarden
Center for Assessment
Rhode Island Weekly Call
February 17th, 2022
Update on Academic Impact Analysis and Report
• We last met in Fall, 2021 and discussed findings.
• The results from RICAS and WIDA-ACCESS in 2021 indicated overall academic
impact was moderate/large in ELA and large in mathematics.
• Both status-based analyses as well as growth-based analyses support these
findings.
• Since then we have finalized results associated with:
• RICAS spring 2021 summative assessment
• WIDA-ACCESS 2021 summative assessment
• Today we want to share results associated with the analyses.
Pandemic Related Academic Impact in Rhode Island 2
How do we quantify academic impact using test data
• The pandemic was like a “headwind” that (presumably) slowed down the learning of
students due to a variety of factors.
• We can use test data in two ways to quantify the impact of this headwind:
• Best way: Quantify how much the rate of learning (i.e., growth) slowed:
• RICAS: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth.
• ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth
• Next best way: Look at how much 2021 status changed:
• RICAS: Use 2019 status (or some adjustment thereof) as a proxy.
• ACCESS: Use 2020 status (or some adjustment thereof) as a proxy.
• Adjustments include (potentially) imputation for missing data.
• Growth is to Speedometer as Status is to Odometer
• Growth deceleration/acceleration will show up as status decreases/increases and vice-versa
Pandemic Related Academic Impact in Rhode Island 3
How do we quantify academic impact using test data
• Status (odometer) comparisons are the most prevalent way of quantifying academic impact:
• Usually looking at decrease in percent proficient (e.g., grade 3 ELA 2019 to grade 3 ELA 2021)
• This isn’t wrong conceptually, just not the most refined metric to use.
• Strength: Status comparisons can be calculated for all grades and content areas and groups.
• Weakness: Status comparisons look at two distinct groups of students and don’t take account of system
improvement.
• Growth (speedometer) comparisons are less prevalent way of quantifying academic impact:
• Historical consecutive and skip-year growth norms were created for RICAS and ACCESS in May, 2021 for
use with 2021 RICAS and ACCESS data.
• Strength: Better accuracy as it only looks at one cohort of students.
• Weakness: Not available for all students since it requires a prior score.
• Rhode Island participation rates were below normal, with participation in elementary grades near 90%
decreasing to 80% in middle school grades.
• Participation (as we’ll see) has a bigger impact upon status comparisons than growth.
Pandemic Related Academic Impact in Rhode Island 4
RICAS Mathematics Grade 6
Rhode Island 2019 to 2021 COVID⌧19 Academic Impact All Students
Growth: Grade 4 2019 to Grade 6 2021 Status: Grade 6 2019 to Grade 6 2021 Summary
2019 Grade 4 Scaled Score Decile Cuts 2019 Grade 6 Scaled Score
Academic Impact: Overall and by Quintile
Grade 6 Scaled/Standardized Score and Achievement Level Cuts
2017 to 2019 Mean Conditional Status
2019 Percentiles (Observed)
Overall
2019 to 2021 Mean Conditional Status (Observed) 2021 Percentiles (Observed)
Observed + Imputed
2019 to 2021 Mean Conditional Status (+ Imputed)
2021 Percentiles (+ Imputed) Severe
2019 to 2021 Academic Impact 2019 to 2021 Academic Impact
2 2019 and 2021 Mean Scaled Scores
Observed
2021 Grade 6 Scaled Score Decile Cuts
Large Severe Severe Severe Severe
2021 Grade 6 Scaled Score
0.5 Participation
0 0.00 Total Status Growth
450 450 2019: 27.9% Proficient
2021: 17.8% Proficient
Growth/Status Change Legend
2019 Grade 4 Scaled Score with Achievement Level Cuts Grade 6 Overall Deciles (2019 Norm)
Modest
Severe Large Moderate to Improvement
2019 & 2021 Grade 6 Median SGPs by Quintile 2019 & 2021 Grade 6 Mean Scaled (Standardized) Scores by Quintile None
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 6 Students 50.0 49.0 51.0 50.0 50.0 50.0 2019 Grade 6 Students 459 (−1.35) 478 (−0.48) 488 ( 0.03) 499 ( 0.53) 518 ( 1.42) 488 (0.00)
2021 Grade 6 Students 26.5 15.0 14.0 17.0 22.0 19.0 2021 Grade 6 Students 456 (−1.52) 470 (−0.86) 480 (−0.36) 492 ( 0.22) 512 ( 1.13) 481 (−0.32) Calculations suppressed
Academic Impact (Observed) −23.5 −34.0 −37.0 −33.0 −28.0 −31.0 Change (Observed) −4 (−0.17) −8 (−0.38) −8 (−0.39) −6 (−0.31) −6 (−0.29) −7 (0.32)
Academic Impact (+ Imputed) −23.0 −30.0 −32.7 −31.0 −28.0 −28.9 Change (+ Imputed) −4 ( 0.06) −8 (−0.20) −8 (−0.25) −7 (−0.22) −6 (−0.28) −7 (0.35)
due to small group N (< 50)
Count Observed 2019 2,059 2,184 1,945 1,592 1,733 9,513 Count Observed 2019 2,395 2,020 2,354 1,992 2,151 10,912
Count Observed 2021 1,548 1,628 1,814 1,550 1,740 8,280 Count Observed 2021 2,069 1,544 1,960 1,628 1,735 8,936
2021 to 2019 Scaled/Standardized Score Difference
2021 to 2019 SGP Difference (Observed) 0.65 2021 to 2019 Grade 6 Status Change (Observed)
30 2021 to 2019 SGP Difference (+ Imputed) 0.60 2021 to 2019 Grade 6 Status Change (+ Imputed)
0.55 0.84: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
2021 to 2019 Conditional Status Difference (Observed)
0.94: Correlation SGP and Conditional Status Difference 10
2019 to 2021
2019 Grade 4 Scaled Score with Decile & Achievement Level Cuts Grade 6 Percentile Rank and Scaled Scores (2019 Norm)
Qualifying Academic Impact
To help summarize the vast quantity of results for stakeholders, we created
categories of academic impact based upon both effect size standards.
Pandemic Related Academic Impact in Rhode Island 10
RICAS ELA and Mathematics
Rhode Island 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Moderate Moderate Large Moderate Moderate Large Large Large Large Severe Large Severe
Economically Disadvantaged Large Moderate Large Large Moderate Moderate Severe Severe Large Severe Large Severe
Non−Economically Disadvantaged Large Moderate Large Moderate Moderate Large Large Large Large Severe Large Severe
African American Students Moderate Moderate Moderate Large Moderate Moderate Severe Severe Large Large Large Severe
Asian Students Moderate Modest to None Large Moderate Moderate Large Large Large Large Severe Large Severe
White Students Moderate Moderate Large Moderate Moderate Large Large Large Large Severe Large Severe
Hispanic Students Large Moderate Large Large Moderate Moderate Severe Severe Large Severe Large Severe
ELL Students Moderate Moderate Large Large Moderate Moderate Large Large Large Large Large Severe
Special Education Students Modest to None Modest to None Moderate Large Moderate Moderate Moderate Moderate Moderate Large Large Large
Female Students Large Moderate Large Large Moderate Large Large Large Large Severe Large Severe
Male Students Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Severe Large Severe 11
ELA And Mathematics
Rhode Island 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Growth/Status Change Legend
NOTE: Quintiles for each student
subgroup are based upon quintile
Modest
Calculations suppressed cuts defined for All Students.
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Economically Disadvantaged
Non−Economically Disadvantaged
African American Students
Asian Students
White Students
Hispanic Students
ELL Students
Special Education Students
Female Students
Male Students 12
Review of results: RICAS
• Academic impacts in mathematics were larger than those in ELA,
though impacts were large in both content areas.
• Academic impacts were larger in middle school grades than
elementary school grades.
• Across demographic subgroups, academic impacts were generally
large to severe.
• When impacts for subgroups are small, questions about pre-pandemic
rates of learning come under suspicion.
Pandemic Related Academic Impact in Rhode Island 13
WIDA ACCESS Composite
WIDA RI 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Growth/Status Change Legend
Modest
Severe Large Moderate to Improvement
None
Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
All Students Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
Composite
WIDA RI 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Growth/Status Change Legend
Modest
Severe Large Moderate to Improvement
None
Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Review of results: WIDA-ACCESS
• For English Language Learners taking the WIDA-ACCESS English
• Academic impacts were moderate to large for students in the
elementary and middle school grades.
• Academic impacts were modest to none for students in the high
school grades.
• These results are fairly consistent with the seven other WIDA states
we examine.
Pandemic Related Academic Impact in Rhode Island 16
RICAS results for larger districts
• Like with demographic subgroups earlier, we can examine academic
impact by district.
• The following plots show academic impact results for several larger
districts.
• Given the large size of the academic impact overall, most districts
experienced large to severe impacts.
• There is, however, variability to be seen in the results.
Pandemic Related Academic Impact in Rhode Island 17
RICAS ELA and Mathematics
Rhode Island 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Moderate Moderate Large Moderate Moderate Large Large Large Large Severe Large Severe
District 01 Moderate Improvement Modest to None Moderate Large Large Moderate Modest to None Moderate Large Moderate Severe
District 06 Large Modest to None Moderate Moderate Large Severe Moderate Moderate Moderate Severe Large Severe
District 07 Large Moderate Large Large Moderate Large Severe Severe Severe Severe Large Severe
District 08 Modest to None Moderate Moderate Large Large Severe Large Large Large Severe Severe Severe
District 10 Large Moderate Large Large Moderate Large Severe Severe Large Severe Large Severe
District 26 Large Modest to None Moderate Large Moderate Large Severe Severe Severe Severe Moderate Severe
District 28 Moderate Moderate Large Moderate Moderate Moderate Severe Large Large Large Moderate Severe
District 35 Moderate Modest to None Large Large Large Modest to None Large Large Large Severe Severe Large
District 39 Moderate Moderate Moderate Moderate Modest to None Moderate Severe Severe Large Severe Moderate Large
ELA And Mathematics
Rhode Island 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Growth/Status Change Legend
NOTE: Quintiles for each student
subgroup are based upon quintile
Modest
Calculations suppressed cuts defined for All Students.
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
District 01
District 06
District 07
District 08
District 10
District 26
District 28
District 35
District 39 19
Baseline SGP: What is Recovery
• Given the negative pandemic impacts, any recovery implies learning
acceleration.
• However, not all acceleration can be considered recovery
V-Shaped Recovery (> 60)
60 U-Shaped Recovery (50 to 60)
Median 2021 Baseline SGP 50 Median 2022 Baseline SGP
L-Shaped “Recovery” (40 to 50)
Downward Spiral (< 40)
Note: Any median SGP above 30
20 in 2022 would be considered
10 acceleration. But only values
1 above 50 would be su cient to
catch up.
ffi
Questions
Arizona State Board of Education Meeting, October 25th, 2021
About this copy
COVID-19 Academic Impact in Rhode Island — Rhode Island Department of Education, February 17, 2022.
This is an image-based copy of the original PDF slides; builds and animations are flattened.