Using SGPs from Star Assessments to Understand COVID-19 Learning Loss Damian Betebenner Center for Assessment NCME National Conference May 27th, 2021 This work is licensed under a Creative Commons Attribution 4.0 International License.

Academic Impact and the COVID-19 Pandemic • As we are all aware, the COVID-19 pandemic has upended the lives of people both in the US and worldwide. • The pandemic has dramatically impacted the physical, emotional, a financial status of people both in the US and worldwide. • Students did not avoid the impact of the pandemic as their education changed dramatically in early 2020 and those changes reverberated into 2021. • From the first stages of the pandemic, there was a desire to understand the academic impact of the pandemic (comprising all the myriad ways the pandemic changed education) on students. • Like most aspects of life, student academic testing was severely disrupted during the first stages of the pandemic leaving little comprehensive data on student attainment • As the pandemic stretched into the 2020-2021 school year, data from interim assessment providers became available to help begin to assess the academic impact of the pandemic on students. NCME 2021: Using SGPs from STAR Assessments

What does COVID-19 academic impact mean? • Several terms have been used to discuss pandemic related academic impact including Learning Loss, Unfinished Learning, and COVID-slide • The terminology used is contentious. • If, in fact, the pandemic has had a negative impact, overall, on the academic attainment of students, whether the lower attainment is due to learning loss (skill atrophy), unfinished learning, or some combination is unknown. • We use the terms interchangeably without ascribing the underlying mechanism of lower attainment. NCME 2021: Using SGPs from STAR Assessments

What do we mean by “Learning Loss” • Understanding pandemic related academic impact/learning loss requires a counter-factual: Where would the student/group/system be if the pandemic hadn’t happened. • We can’t observe the counterfactual. • Using historical data, we can attempt to model the counterfactual. • For example, assuming the student would have demonstrated typical pre-COVID academic growth, how far behind (ahead) is the student? NCME 2021: Using SGPs from STAR Assessments

What do we mean by “Learning Loss” • A decrease in student growth leads to a decrease in attainment. • Similarly, lower attainment implies that the growth for the student was lower. • Deviation from historically typical growth can be used to quantify the academic impact of the pandemic. NCME 2021: Using SGPs from STAR Assessments

Renaissance STAR Assessments and Student Growth • Since 2010, Renaissance Learning has been calculating student growth percentiles (SGPs) for students on their STAR mathematics, reading, and early literacy assessments. • The RLI STAR student growth percentiles differ slightly from those used as part of state assessments: • STAR assessments are administered through the year in Fall (August to November), Winter (December to March) and Spring (April to June) assessment windows. • Growth is calculated across several permutations of the assessment windows: Fall-to- Fall, Fall-to-Winter, Fall-to-Spring, Winter-to-Spring, Spring-to-Spring, Spring-to-Fall. • Students do not take the assessments on the same date, so quantile regression modelling takes account of: • Up to two prior achievement scores • Time during the window when test is taken (i.e., number of days since beginning of window) • Time-lag to the prior test (i.e., number of days to the first prior test) • See this report for more details. NCME 2021: Using SGPs from STAR Assessments

Renaissance STAR Assessments and Student Growth • STAR SGPs grew out of Renaissance Learning’s efforts to create historical growth norms using gain scores from their vertical scale. • SGPs calculated using gain scores as the dependent variable are equivalent to SGPs calculated using the most recent score as the dependent variable. • Renaissance Learning generates SGPs in real-time for students using their cloud- based analytics platform and the SGP package based upon the R software environment (Betebenner, Van Iwaarden, Domingue, and Shang, 2021). • Generating SGPs in real-time requires pre-existing growth norms (i.e., coefficient matrices) to be used. • At present, coefficient matrices utilized to calculate SGPs for STAR assessments are based upon 2015-16, 2016-17, and 2017-18 STAR results and utilize between 500,000–and 1,000,000 students in each grade and content area. NCME 2021: Using SGPs from STAR Assessments

SGP interpretation • For the entire norm group of students on which the coefficient matrices (growth norms) are derived: • SGPs have mean/median of 50 • SGPs are uncorrelated with prior (initial) achievement. • For SGPs calculated using later cohorts of students: • Mean/median SGPs for subgroups (e.g., demographic subgroups, schools, districts) will usually differ from 50. That is, it is not uncommon to see growth gaps. • Mean SGP differences between groups of 5 correspond to an effect size of 0.18. • Covid related academic impact for groups can be determined by calculating: Mean SGPgroup, pre-Covid – Mean SGPgroup, Covid • That is, how much has growth (i.e. learning) slowed down due to COVID. • My co-presenters will be providing numerous results. NCME 2021: Using SGPs from STAR Assessments

SGP and the assessment of academic impact • Historically, the least utilized SGPs calculated by STAR was probably Fall-to-Fall growth. • With a pandemic striking the world in the spring, Fall-to-Fall growth took on particular relevance. • Fall 2019 to Fall 2020 SGPs allowed Renaissance to evaluate academic impact due to COVID. • Fall 2020 to Winter 2021 and Winter 2021 to Spring 2021 SGPs allow Renaissance to evaluate both impact and potentially initial recovery. NCME 2021: Using SGPs from STAR Assessments

Four types of recovery • Borrowing terminology from economics, STAR SGPs will broadly allow stakeholders to plan for and evaluate: • V-shaped recovery -- Immediate snap-back to non-pandemic trajectory • U-shaped recovery – More gradual move back toward non-pandemic trajectory • L-shaped recovery – Not a recovery but not falling further behind. • Academic downward spiral – Continue to fall further and further behind. • At a system level, beware the K-shaped recovery where high achievers/advantaged predominantly experience V- and U-shaped recoveries and low achievers/ disadvantaged don’t recover. NCME 2021: Using SGPs from STAR Assessments

Threats to interpretation • There are several threats to the validity of STAR interim assessment SGP results. • It was difficult to distinguish Fall 2020 test takers who took the exam at home and took the exam at school. • The students for whom we have growth data in 2021 might be different than those with growth data in 2020. • Students absent from testing in Fall 2020 • Changing customer base • Scale drift would undermine the use of historical growth norms (i.e., baseline SGP coefficient matrices) which presume no scale drift. NCME 2021: Using SGPs from STAR Assessments

Summary • Renaissance STAR Assessments included a built in growth measure capable of allowing stakeholders to begin assessing the academic impact of the pandemic. • Have worked with several states on this issue, states are not aware of how to utilize these data. • Multiple source of evidence are required before • As state summative data becomes available shortly, an important (dis)confirmatory piece of evidence will be available. NCME 2021: Using SGPs from STAR Assessments

Descriptive accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality Robert Linn, 2008, p. 21 NCME 2021: Using SGPs from STAR Assessments

References • Betebenner, D. W., Van Iwaarden, A. R., Domingue, B. and Shang, Y. SGP: Student Growth Percentiles & Percentile Growth Trajectories (2021). R package version 1.9-9.81, https://sgp.io. • Linn, R. L. (2008) Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. NCME 2021: Using SGPs from STAR Assessments

About this copy

Using SGPs from Star Assessments to Understand COVID-19 Learning Loss — NCME Annual Meeting, May 27, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.