COVID-19 Impact & Recovery
October 2022 Update
Damian Betebenner— Center for Assessment
Indiana DOE Assessment TAC
October 13th, 2022
Impact & Recovery
• Recall:
• Academic impact is analogous to a headwind, decreasing the rate of student learning
(deceleration) which, in turn leads to lower attainment
• Academic recovery is analogous to a tailwind, increasing the rate of student learning
(acceleration) which, in turn leads to higher attainment
• 2019 to 2021, under the strong presumption that educational disruptions between spring 2020 to
spring 2021 led to learning deceleration, we examined academic impact
• 2021 to 2022, under the assumption/desire that student education approximating normal, led to
learning stabilization/recovery, we examined academic recovery.
• Examination of academic impact/recovery are opposite sides of the same coin and utilize the same
analytics.
• It is important to note that any meaningful discussion about recovery presumes an understanding
of the magnitude of impact.
Impact & Recovery
• We investigate impact/recovery in two ways:
• Change in same grade/same cohort attainment: We examine equi-percentile change in
attainment.
• Change in student growth: We examine change in baseline student growth (SGPs) utilizing
coefficient matrices derived from pre-pandemic cohorts of students. 2017 to 2019 growth for
2019 to 2021
• As Rich pointed out at the last TAC meeting, attainment is to odometer as growth is to
speedometer.
• Across multiple states the correlation (grades/content area/state) between these two measures of
impact are 0.81.
• Low participation the largest cause of outliers between the two indicators
• We utilize change in student growth for grades/cohorts with that quantity and change in
attainment (mapped to change in student growth) for grades/cohorts that don’t.
2019 to 2021: Impact
• ELA
• Decreases in percent proficient ranged from upper single digits in lower grades to lower single
digits in higher grades.
• Median (baseline) SGPs ranged from upper 30s to lower 40s
• Impacts were categorized as moderate
• Mathematics
• Decreases in percent proficient was in the lower teens
• Median (baseline) SGPs ranged from the low to mid 30s.
• Impacts were categorized as large
• Moderate, large, and severe impacts likely require multiple years of above average (pre-pandemic)
growth to recover to where students would have been had the pandemic not occurred.
ILEARN ELA and Mathematics
Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Deterioration Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large
Economically Disadvantaged Large Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large Large
Non−Economically Disadvantaged Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large
African American Large Large Moderate Large Moderate Moderate Severe Severe Severe Severe Large Large
Asian Moderate Moderate Moderate Moderate Moderate Modest to None Moderate Moderate Large Severe Severe Large
White Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large
Hispanic Large Moderate Large Large Moderate Moderate Large Large Large Severe Large Large
ELL Large Moderate Large Large Moderate Large Large Large Large Large Moderate Large
Special Education Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate
Female Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Severe Large Large
Male Moderate Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large
ELA And Mathematics
Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Deterioration Recovery
−40 −25 −15 −5 5 20 Growth Change
NOTE: Quintiles for each student
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile
Modest
Calculations suppressed
cuts defined for All Students.
Severe Large Moderate to Improvement
None due to small group N (< 50) Box width indicates group proportion.
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Economically Disadvantaged
Non−Economically Disadvantaged
African American
Asian
White
Hispanic
ELL
Special Education
Female
Male
ELA And Mathematics
Indiana 2019 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Deterioration Recovery
−40 −25 −15 −5 5 20 Growth Change
NOTE: Quintiles for each student
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change subgroup are based upon quintile
Modest
Calculations suppressed
cuts defined for All Students.
Severe Large Moderate to Improvement
None due to small group N (< 50) Box width indicates group proportion.
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Economically Disadvantaged
Non−Economically Disadvantaged
African American
Asian
White
Hispanic
ELL
Special Education
Female
Male
ILEARN Mathematics Grade 8
Indiana 2019 to 2021 COVID⌧19 Academic Impact Special Education Students
Growth: Grade 6 2019 to Grade 8 2021 Status: Grade 8 2019 to Grade 8 2021 Summary
2019 Grade 6 Scaled Score Decile Cuts
All Students
10 20 30 40 50 60 70 80 90 2019 Grade 8 Scaled Score
Academic Impact: Overall and by Quintile
Grade 8 Scaled/Standardized Score and Achievement Level Cuts
Special Education Students
Overall
Observed + Imputed
2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (Observed) 2 2021 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Moderate
Observed
2019 to 2021 Growth Impact 2019 to 2021 Status Impact
2019 and 2021 Mean Scaled Scores
2021 Grade 8 Scaled Score Decile Cuts
Modest to None Moderate Moderate Moderate Moderate
2021 Grade 8 Scaled Score
Participation
60 0 Total Status Growth
6400 6400 2019: 8.6% Proficient
2021: 6.1% Proficient
Growth/Status Change Legend
2019 Grade 6 Scaled Score with Achievement Level Cuts Grade 8 Overall and Subgroup Percentile Ranks
Modest
Severe Large Moderate to Improvement
2019 & 2021 Grade 8 Median SGPs by Quintile 2019 & 2021 Grade 8 Mean Scaled (Standardized) Scores by Quintile None
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 8 Students39.0 36.0 35.0 40.0 45.0 38.0 2019 Grade 8 Students 6382 (−0.92) 6487 (−0.92) 6551 (−0.92) 6609 (−0.92) 6695 (−0.92) 6451 (−0.92)
2021 Grade 8 Students36.0 30.0 23.0 27.0 32.5 33.0 2021 Grade 8 Students 6369 (−1.05) 6455 (−1.05) 6515 (−1.05) 6580 (−1.05) 6671 (−1.05) 6437 (−1.05) Calculations suppressed
Academic Impact (Observed)−3.0 −6.0 −12.0 −13.0 −12.5 −5.0 Change (Observed) −13 (−0.12) −33 (−0.30) −36 (−0.33) −29 (−0.27) −24 (−0.22) −14 (−0.13)
Academic Impact (+ Imputed)−3.0 −6.0 −12.0 −13.0 −12.5 −5.0 Change (+ Imputed) −13 (2.31) −33 (1.51) −36 (1.04) −29 (0.63) −24 (0.02) −14 (−0.13)
due to small group N (< 50)
Count Observed 20195,955 2,587 1,275 712 377 10,906 Count Observed 2019 6,537 2,833 1,364 701 423 11,858
Count Observed 20216,485 2,318 1,156 633 332 10,924 Count Observed 2021 5,835 2,859 1,536 859 443 11,532
2021 to 2019 Scaled/Standardized Score Difference
2021 to 2019 SGP Difference (Observed) 0.30 2021 to 2019 Grade 8 Status Change (Observed)
30 2021 to 2019 SGP Difference (+ Imputed) 30 2021 to 2019 Grade 8 Status Change (+ Imputed)
0.25 −0.02: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
2021 to 2019 Conditional Status Difference (Observed)
0.99: Correlation SGP and Conditional Status Difference
2019 to 2021
2019 Grade 6 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)
ILEARN ELA Grade 8
Indiana 2019 to 2021 COVID⌧19 Academic Impact All Students
Growth: Grade 6 2019 to Grade 8 2021 Status: Grade 8 2019 to Grade 8 2021 Summary
2019 Grade 6 Scaled Score Decile Cuts 2019 Grade 8 Scaled Score
Academic Impact: Overall and by Quintile
Grade 8 Scaled/Standardized Score and Achievement Level Cuts
Overall
Observed + Imputed
2017 to 2019 Mean Conditional Status 2019 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (Observed) 2.5 2021 Percentiles (Observed)
2019 to 2021 Mean Conditional Status (+ Imputed) 2021 Percentiles (+ Imputed) Moderate
Observed
2019 to 2021 Growth Impact 2019 to 2021 Status Impact
2021 Grade 8 Scaled Score Decile Cuts
2019 and 2021 Mean Scaled Scores
1.5 Large Large Moderate Moderate Modest to None
2021 Grade 8 Scaled Score
Participation
Total Status Growth
5400 5400 2019: 50.1% Proficient
2021: 43.9% Proficient
Growth/Status Change Legend
2019 Grade 6 Scaled Score with Achievement Level Cuts Grade 8 Overall Percentile Ranks
Modest
Severe Large Moderate to Improvement
2019 & 2021 Grade 8 Median SGPs by Quintile 2019 & 2021 Grade 8 Mean Scaled (Standardized) Scores by Quintile None
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 8 Students 50.0 50.0 50.0 50.0 50.0 50.0 2019 Grade 8 Students 5458 (−1.45) 5533 (−0.50) 5577 ( 0.05) 5619 ( 0.58) 5679 ( 1.34) 5573 (0.00)
2021 Grade 8 Students 35.0 34.0 39.0 45.0 50.0 40.0 2021 Grade 8 Students 5439 (−1.69) 5513 (−0.75) 5562 (−0.14) 5609 ( 0.46) 5675 ( 1.29) 5560 (−0.17) Calculations suppressed
Academic Impact (Observed) −15.0 −16.0 −11.0 −5.0 0.0 −10.0 Change (Observed) −19 (−0.24) −20 (−0.25) −15 (−0.19) −9 (−0.12) −4 (−0.05) −13 (−0.17)
Academic Impact (+ Imputed) −15.0 −16.0 −11.0 −5.0 0.0 −10.0 Change (+ Imputed) −19 ( 0.34) −20 (−0.46) −15 (−0.92) −9 (−1.35) −4 (−1.98) −13 (−0.17)
due to small group N (< 50)
Count Observed 2019 15,204 15,664 15,068 15,768 15,476 77,180 Count Observed 2019 16,523 16,799 16,614 16,601 16,454 82,991
Count Observed 2021 15,567 15,724 15,726 15,429 15,859 78,305 Count Observed 2021 16,619 16,442 16,477 16,580 16,483 82,601
2021 to 2019 Scaled/Standardized Score Difference
2021 to 2019 SGP Difference (Observed) 0.40 2021 to 2019 Grade 8 Status Change (Observed)
30 2021 to 2019 SGP Difference (+ Imputed) 30 2021 to 2019 Grade 8 Status Change (+ Imputed)
0.35 1.00: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
2021 to 2019 Conditional Status Difference (Observed)
1.00: Correlation SGP and Conditional Status Difference 0.30
2019 to 2021
2019 Grade 6 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2019 Norm)
WIDA ACCESS WIDA⌧ACCESS Composite
WIDA IN 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
All Students Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement
WIDA⌧ACCESS Composite
WIDA IN 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile
Impact Recovery
−40 −25 −15 −5 5 20 Growth Change
−0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
All Students
Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Indiana Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large
State A Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe
State B Moderate Moderate Moderate Moderate Large Moderate
State C Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large
State D Moderate Moderate Large Large Large Large Large Moderate Severe Severe Large Severe
State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large
State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe
State G Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large
State H Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate
State I Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate
WIDA⌧ACCESS Composite
Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area: All Students
Growth/Status Change Legend
Modest
Calculations suppressed
Severe Large Moderate to Improvement
None due to small group N (< 50)
Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
State A Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None
State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None
State C Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement
State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None
State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None
State F Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None
State G Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None
Four Academic
Recovery
Scenarios with
ILEARN : V-, U-, L-,
and No-recovery
Baseline SGP: What is Recovery
• Given the negative pandemic impacts, any recovery implies learning acceleration.
• However, not all acceleration can be considered recovery
• Acceleration versus 2021 versus acceleration versus 2019
• It is important to note that recovery of the system to normal rates of learning does not imply students have recovered
V-Shaped Recovery (> 60)
60 U-Shaped Recovery (50 to 60)
Median 2021 Baseline SGP 50 Median 2022 Baseline SGP
L-Shaped “Recovery” (40 to 50)
Downward Spiral (< 40) Note: Any median SGP above 30
in 2022 would be considered
acceleration. But only values
10 above 50 would be sufficient to
begin catching students up to
where they would have been
absent the pandemic.
ILEARN ELA and Mathematics
Indiana 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.55 −0.07 0.1 0.2 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
All Students Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Economically Disadvantaged Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Non−Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
African American Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization
Asian Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Hispanic Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Stabilization
ELL Stabilization Stabilization Stabilization Deterioration Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization
Special Education Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
Female Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization
Male Stabilization Stabilization Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization
ELA And Mathematics
Indiana 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile
Deterioration Recovery
−30 −7 10 30 Growth Change
NOTE: Quintiles for each student
−0.55 −0.07 0.1 0.2 Status Change subgroup are based upon quintile
Calculations suppressed
cuts defined for All Students.
Deterioration Stabilization Recovery
due to small group N (< 50) Box width indicates group proportion.
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Economically Disadvantaged
Non−Economically Disadvantaged
African American
Asian
White
Hispanic
ELL
Special Education
Female
Male
ELA And Mathematics
Indiana 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile
Deterioration Recovery
−30 −7 10 30 Growth Change
NOTE: Quintiles for each student
−0.55 −0.07 0.1 0.2 Status Change subgroup are based upon quintile
Calculations suppressed
cuts defined for All Students.
Deterioration Stabilization Recovery
due to small group N (< 50) Box width indicates group proportion.
ELA Mathematics
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Economically Disadvantaged
Non−Economically Disadvantaged
African American
Asian
White
Hispanic
ELL
Special Education
Female
Male
ILEARN ELA Grade 6
Indiana 2021 to 2022 COVID⌧19 Academic Recovery Hispanic Students
Growth: Grade 5 2019 to Grade 6 2022 Status: Grade 6 2021 to Grade 6 2022 Summary
2021 Grade 5 Scaled Score Decile Cuts
All Students
10 20 30 40 50 60 70 80 90 2021 Grade 6 Scaled Score
Academic Impact: Overall and by Quintile
Grade 6 Scaled/Standardized Score and Achievement Level Cuts
Hispanic Students
2018 to 2019 Mean Conditional Status
2021 Percentiles (Observed)
Overall
2021 to 2022 Mean Conditional Status (Observed) 2022 Percentiles (Observed)
2021 to 2022 Growth Impact 3 2021 to 2022 Status Impact Deterioration
2021 and 2022 Mean Scaled Scores
2022 Grade 6 Scaled Score Decile Cuts
Observed
Deterioration Stabilization Stabilization Stabilization Stabilization
2022 Grade 6 Scaled Score
Participation
Total Status Growth
5400 5400 −1.5 2021: 27.6% Proficient
2022: 26.9% Proficient
Growth/Status Change Legend
2021 Grade 5 Scaled Score with Achievement Level Cuts Grade 6 Overall and Subgroup Percentile Ranks
Deterioration Stabilization Recovery
2019 & 2022 Grade 6 Median SGPs by Quintile 2021 & 2022 Grade 6 Mean Scaled (Standardized) Scores by Quintile
Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall
2019 Grade 6 Students 53.0 49.0 47.0 47.0 45.0 49.0 2021 Grade 6 Students 5404 (−0.31) 5480 (−0.31) 5523 (−0.31) 5564 (−0.31) 5621 (−0.31) 5497 (−0.31)
2022 Grade 6 Students 36.0 43.0 43.0 41.0 43.0 41.0 2022 Grade 6 Students 5400 (−0.34) 5474 (−0.34) 5519 (−0.34) 5563 (−0.34) 5621 (−0.34) 5494 (−0.34) Calculations suppressed
Academic Impact (Observed) −17.0 −6.0 −4.0 −6.0 −2.0 −8.0 Change (Observed) −4 (−0.06) −7 (−0.09) −4 (−0.05) −1 (−0.01) 0 ( 0.00) −3 (−0.04)
Count Observed 2019 3,109 2,527 2,228 1,687 1,185 10,736 Count Observed 2021 3,230 2,764 2,268 1,833 1,310 11,405
due to small group N (< 50)
Count Observed 2022 3,126 2,528 2,143 1,681 1,140 10,618 Count Observed 2022 3,143 2,700 2,310 1,877 1,264 11,294
2022 to 2021 Scaled/Standardized Score Difference
2022 to 2021 SGP Difference (Observed) 0.40 2022 to 2021 Grade 6 Status Change (Observed)
2022 to 2021 Conditional Status Difference (Observed) 0.64: Correlation SGP and Status Percentile Change
Conditional Status/Growth Change
25 1.00: Correlation SGP and Conditional Status Difference 25
2021 to 2022
2021 Grade 5 Scaled Score with Decile & Achievement Level Cuts Grade 6 Percentile Ranks and Scaled Scores (2021 Norm)
WIDA ACCESS WIDA⌧ACCESS Composite
WIDA IN 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.55 −0.07 0.1 0.2 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
All Students Stabilization Deterioration Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Deterioration Deterioration Deterioration Deterioration
WIDA⌧ACCESS Composite
WIDA IN 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile
Deterioration Recovery
−30 −7 10 30 Growth Change
−0.55 −0.07 0.1 0.2 Status Change
Calculations suppressed
Deterioration Stabilization Recovery
due to small group N (< 50)
WIDA−ACCESS Composite
Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12
Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile
All Students
Nationwide
• We are in the middle of tabulating results for about a dozen states on the extent
of recovery occurring
• Overall, most states are stabilizing or stabilizing bordering on deterioration. There
is at least one state whose results indicate rates of learning that would lead to
student recovery
• I have seen examples from several states (some in great detail) which make me
question whether the psychometrics (e.g., equating/linking) are holding up over
the course of the pandemic.
• Analyses such as these will require a stable scale going forward.
Secondary Analyses:
Impact & Recovery
• Many states and research organizations are interested in examining student assessment data
to examine student impact/recovery in relation to pandemic mitigation efforts and pandemic
academic recovery efforts.
• This interest is particularly acute due to the large amounts of money being directed at
schools/districts to help with student academic recovery.
• Return on investment (ROI) analyses have been discussed with several states including
Indiana.
• ROI analyses present numerous significant challenges including:
• Inconsistent terminology used by schools/corporations/state (e.g., high-dosage tutoring)
• Funding sources either not detailed or coming from multiple sources
• Fidelity of implementation may vary greatly across sites
Exemplar Schools: Impact & Recovery
• ROI discussions in Indiana focused on two areas:
• Identifying outlier schools based upon impact/recovery patterns and doing in-depth
follow-up interviews to determine what, if any, activities these schools engaged in.
That is, switch from return on investment to return on activities. (post hoc
investigation)
• Schools receiving 1008 funding (a priori investigation)
• We have begun both of these analyses.
• For the outlier schools We investigated schools along two dimensions:
• Academic Impact (2019 to 2021 decrease in student learning)
• Academic Recovery (2021 to 2022 increase in student learning relative to pre-
pandemic 2018-2019 learning)
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Middle Schools by Poverty: ELA
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary−Middle Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
School Impact−Recovery: 2019 to 2021 and 2021 to 2022
Elementary Schools by Poverty: ELA & Math Combined
Recovery
+ Outliers
Academic Recovery 2021 to 2022
− Outliers
Deterioration
Free/Reduced Lunch: < 30%
Free/Reduced Lunch: 30% to 45%
Free/Reduced Lunch: 45% to 60%
Free/Reduced Lunch: 60% to 75%
Free/Reduced Lunch: > 75%
Severe Impact Academic Impact 2019 to 2021 No Impact
1008 Schools
• Indiana’s House Bill 1008, the student learning recovery grant program, put forth
up to $150,000,000 to establish:
• Student learning recovery grant program
• Student learning recovery grant program fund
• Entities eligible to receive grant money had to develop and submit a student
learning acceleration plan.
• Due to the large sums of money being devoted to such programs together with
the unprecedented academic situation students currently face, Indiana is keenly
interested in understanding whether recovery programs being implemented are
successful.
1008 Schools
• We have just begun investigating impact associated with 1008 schools.
• 1008 participation began summer 2021.
• Initially we investigated results for schools in Marion County (includes
Indianapolis)
• Data from Marion County schools included student level participation indicating
total minutes of participation in four categories: ELA, Mathematics, CCR (Career/
College Readiness), and Other as well as Total Minutes
• Additional statewide data was also just received. Quality of the data varies greatly
from corporation to corporation.
SGP Baseline: 2022 minus 2019 SGP Baseline: 2022 minus 2019
By 1008 Status ELA By Total Minutes 1008 Decile ELA
SGP Baseline SGP Baseline
FALSE TRUE (220,2100] (2100,3500] (3500,4200] (4200,5600] (5600,6650] (6650,7000] (7000,8050] (8050,8400] (8400,14400] (14400,34560]
1008 Status Decile Total Minutes 1008
SGP Baseline: 2022 minus 2019 SGP Baseline: 2022 minus 2019
By 1008 Status Mathematics By Total Minutes 1008 Decile Mathematics
SGP Baseline SGP Baseline
FALSE TRUE (220,2100] (2100,3500] (3500,4200] (4200,5600] (5600,6650] (6650,7000] (7000,8050] (8050,8400] (8400,15210] (15210,34560]
1008 Status Decile Total Minutes 1008
About this copy
COVID-19 Impact & Recovery — Indiana Technical Advisory Committee, October 13, 2022.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.