Colorado SGP Growth Calculations in 2021 Operational and Investigative Considerations Damian Betebenner Center for Assessment Colorado Department of Education TAP Meeting May 25, 2021

Calculation of SGPs in 2021 • Not surprisingly 2021 presents numerous challenges in the administration of state assessments. • These challenges extend beyond just the administration of the assessments to the subsequent analysis of the data derived from the assessments. • As Andrew Ho said, if there was ever a “put up or shut up moment” for large-scale, state summative assessment, now is that moment. • The following slides detail considerations associated with SGP calculations in 2021 and demonstrate a path by which large scale state summative assessments can rise to the current moment and demonstrate essential value. CO SGP Growth Calculations in 2021

Operational vs Investigative: A Useful Distinction • Though SGPs are used almost exclusively for investigative analyses, there are numerous operational considerations in their calculation necessary to make sure growth scores (i.e. SGPs) are suitable for investigative analyses. • Investigative analyses generally presume data derived is of sufficient quality for investigations. 1 Table taken from Making Sense of 2021 Assessment Results by Dadey, Keng, Boyer, & Marion (2021) CO SGP Growth Calculations in 2021

SGP Operational Considerations in 2021 • Spring 2021 administration of CMAS presents numerous challenges that may impact the calculation of valid SGPs. • Requires skip-year growth (using 2019 as the prior year) due to no spring 2020 testing (this is not really an operational problem. CO and others states have experience calculating skip-year growth.) • Motivation effects on the exam. • Opt-in testing in off grades/content areas. • Spring 2021 administration of WIDA-ACCESS does not present these challenges. CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • SGP analyses presume valid scale scores • Students forming the growth norm group are presumed to have taken the same (or equivalent) test at the same time. Scores must mean the same thing. • SGPs, however, are often used as a post-hoc check of scale score validity/comparability via Reductio ad absurdum. • CDE has extensive experience (e.g., PARCC paper/pencil versus online) investigating and accommodating mode-effects. • Note: Low participation is considered an investigative challenge, not an operational challenge. We assume something close to population invariance with respect to IRT scaling. CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • CMAS • Cohort referenced and baseline referenced skip-year SGPs for CMAS. Baseline coefficient matrices are derived from pre-COVID CMAS data. • SGP reporting? • Baseline referenced results used for supporting investigative analyses. • WIDA-ACCESS • Cohort and baseline referenced (consecutive-year & skip-year) SGPs for WIDA-ACCESS. Baseline Coefficient matrices are derived from pre-COVID WIDA-ACCESS data. • Skip-year results on WIDA-ACCESS can be compared to consecutive-year results to see how well skip-year results capture the academic impact of the COVID-19 pandemic. • These results will help inform use of skip-year CMAS results where consecutive-year results are not available. CO SGP Growth Calculations in 2021

SGP Investigative Considerations in 2021 If we can overcome the operational challenges associated with administration of tests, the generation of valid scale scores, and the calculation of SGPs, the data derived from CMAS and WIDA-ACCESS will be invaluable to CDE in understanding what has occurred to students academically during the pandemic. CO SGP Growth Calculations in 2021

2021 Assessment and Analysis Fallacies • Administration of state assessments this year was particularly contentious. • “Better no data than bad data!” • “We already know where kids are and how to help them!” • “Interim assessments already tell us what we need to know!” • “The data won’t arrive soon enough to help!” • “Focusing on the academic impact (i.e., learning loss) promotes a deficit mindset!” • “You can’t compare results from 2021 to 2019!” • “People will mis-interpret the data!” CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Mean/median will by 50 for all grade x content area analyses • Cannot investigate overall impact of the pandemic on students. • Can investigate differential impact of the pandemic on students. • Does not require scale stability across years. • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related academic impact. • Mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact and differential impact • Requires scale stability across years. CO SGP Growth Calculations in 2021

Four Academic Impact Scenarios with CMAS CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change (growth or status) is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • The first step toward supporting these comparisons is to have a complete understanding is who constitutes the missing data. CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • An issue impacting the interpretation of growth (and status) results in 2021 is participation in the state assessment. • Low participation doesn’t impact calculation of cohort referenced growth norms nor use of baseline referenced growth norms. • Low participation does impact comparisons of group level results with historical (e.g., 2019) results based upon higher participation rates. • For example, how should one compare growth and status results for school in 2019 and 2021 with very different rates of participation? CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • Understanding who is missing is key. • State education data is exemplary in terms of identifying who should test. • For example, we could create worst/best case scenarios by substituting LOSS/ HOSS values for missing students. • More precisely, we empirically examine whether missing data is MCAR, MAR, or MNAR. • Apply multiple-imputation procedures based upon findings to create inferred values. • We are currently testing out various procedures and have strong evidence that inferences about group aggregates can be faithfully recovered. • Again, results are not intended to be used for accountability. CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • With comparable results we can quantify pandemic related academic impact: • At the state level • By region/district/school • By mode of education (remote/hybrid/in-person). • Attendance • Demographic subgroups including ethnicity, poverty • Academic impact on English Language Learner impact can be evaluated using WIDA-ACCESS results. CO SGP Growth Calculations in 2021

SGP Analysis Plans for 2021 • ESSER (Federal) State Plan released in late April requires states: “Understanding the Impact of the COVID-19 Pandemic: Describe how the SEA will support its LEAs in identifying the extent of the impact of the COVID-19 pandemic on student learning and student well-being, including identifying the groups of students most impacted by the pandemic.” • https://oese.ed.gov/files/2021/04/ARP-ESSER-State-Plan- Template-04-20-2021_130PM.pdf • The best data available to understand the academic impact of the pandemic comes from the state assessments: CMAS and ACCESS CO SGP Growth Calculations in 2021

SGP Analysis Plans post 2021 (2022 and beyond) • Spring 2021 represents an ideal time to get a snapshot of student attainment as we are presumably at or near the point of maximal impact on student attainment. • That is, recovery has not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery! CO SGP Growth Calculations in 2021

Four Academic Recovery Scenarios with CMAS: V-, U-, L-, and No-recovery CO SGP Growth Calculations in 2021

Communication of Results in 2021 • Technical challenges to the calculation of growth in 2021 are likely minor compared to communication challenges. • State reporting is, in general, highly aligned with accountability. • Even though analyses outlined herein deal with status/growth, they don’t necessarily align with accountability narratives. • It is critical for leadership (at the highest level) to map out a communication plan and then build analytics and reporting to support that plan. • Without a well considered, disciplined and coordinated communication plan, accountability narratives are likely to take over. CO SGP Growth Calculations in 2021

Questions CO SGP Growth Calculations in 2021

About this copy

Colorado SGP Growth Calculations in 2021 — Colorado Department of Education, May 24, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.