AzM2 REPORTING AND ANALYSES DURING THE COVID-19 PANDEMIC AN OVERVIEW Damian Betebenner, Center for Assessment ATAC Discussion on Growth May 3rd, 2021 This work is licensed under a Creative Commons Attribution 4.0 International License.

Start at the End: 2021 Data Reporting and Use • Recent federal guidance indicates that states are responsible to report results from 2021 testing (however not used for accountability) • State reporting is, in general, highly aligned with accountability. • Growth & Status analyses can be used for other purposes besides accountability. • Even though analyses outlined herein deal with status/growth, they don’t necessarily align with accountability narratives. • It is critical for leadership (at the highest level) to map intended uses of assessment data and think through communication plan in support of that. • Analytics and reporting can be built specified after the vision of uses and communication is established. . • Without a well considered, disciplined and coordinated communication plan, accountability narratives are likely to take over (which may be what AZ is OK with. just be aware). AzM2 Reporting and Analysis During the COVID-19 Pandemic

2021 Data Reporting and Use • Arizona will report status and growth based upon AzM2 results. • Arizona, like other states, is considering situations that may impact results (e.g., low participation). • The pandemic will complicate interpretation and many states are concerned with misinterpretation. • In addition to standard reporting of results, many states are considering supplemental reporting or special reports to help stakeholders interpret the data and, for example, help stakeholders understand the impact of the pandemic on student achievement as measured by state summative assessments. AzM2 Reporting and Analysis During the COVID-19 Pandemic

2021 Calculation of Growth • Calculation of growth will be straightforward in 2021 • Because no assessment was given in Spring 2020, Arizona will utilize 2019 and 2018 as prior scores for the state assessment. • Both cohort and baseline referenced SGPs will be calculated. • Cohort referenced SGPs utilize current year growth norms derived from 2021 and earlier data. • Baseline referenced SGPs will utilize pre-Covid growth norms. • Baseline referenced SGPs will allow for an examination of the academic impact of the pandemic on student learning in Arizona. • Dr. Xiaoyuan Tan has extensive experience conducting these types of growth analyses AzM2 Reporting and Analysis During the COVID-19 Pandemic

2021 Data Reporting and Use Beyond reporting the score (e.g., percent proficient, median SGP, …) as is, there are four basic options for reporting: • Suppress: The score is not reported. • Flag: The score is provided with cautions. • Adjust. The score is supplemented or replaced with an “adjusted” score. • Supplement: Additional information is provided alongside the score, potentially including an adjusted score. • Again, the options are not mutually exclusive. For example, a score might be flagged and reported alongside an adjusted score and contextual information explaining the adjusted score. AzM2 Reporting and Analysis During the COVID-19 Pandemic

2021 Data Reporting and Use • Arizona is likely committed to publicly reporting status and growth scores in 2021 (perhaps in multiple ways). • The dilemma is what do 2021 status or growth scores mean. • If the psychometricians can keep the scale stable, individual scores will mean the same thing as they always do. • Summary scores (e.g., at the school level) might be more problematic. People will (and should) ask what were these scores before the pandemic (i.e., 2019) • Wanting to compare 2021 scores to 2019 scores IS the right impulse. People want to gauge the impact of the pandemic. • Is it possible to support these comparisons? How can we support these comparisons? To not support them will likely leave people with lots of questions (“Why did you make us test if we can understand how we’re doing compared to before?”) • Comparisons are potentially problematic due to low participation in 2021. AzM2 Reporting and Analysis During the COVID-19 Pandemic

2021 Data Reporting and Use • The investigative analyses discussed hereafter assume that the assessment data validly describes student attainment in the assessed content area. • Operational analyses (not discussed here) are necessary to insure the validity of these data and the defensibility of subsequent investigative analyses. 1 Table taken from Making Sense of 2021 Assessment Results by Dadey, Keng, Boyer, & Marion (2021) AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Academic Impact • The academic impact of the pandemic is variously referred to as learning loss, unfinished learning, COVID-19 slide. • The terms used are highly contentious. Be aware and use your terms carefully. • As states formulate policies to help with student academic recovery, there is a need for data/information/evidence on: • Who: Who is experiencing or has experienced learning loss? • What: What have students experienced learning loss in? • How Much: How much learning loss have students experienced? • Answers to these questions will be critical in helping to understand how to help? For example, if learning loss is substantial for some groups, recovery will not be immediate and will require many months if not years to ameliorate. AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Data • Data are necessary to answer Who?, What? and How much? • Arizona has extensive academic data available to determine learning loss for students: • AzM2 state summative assessment data • Possibly English Language Proficiency data (ELL students) • Interim assessment data from assessment vendors. • Multiple sources of data allow for: • Investigation of student groups not contained in some data sets (e.g., ELL students) • Triangulation of results across multiple data sets. AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Conceptualizing Academic Impact • Think of the pandemic as a head-wind that slowed the progress of students. • As such, there are two ways to use data to detect the impact of the pandemic. • Status change: Compare similar grade/content area groups from 2019 to 2021. Need to take special care the groups are comparable. • Growth change: Compare growth rates for students from 2019 to 2021 using historical norms. • NOTE: Status and growth change are the 2019 to 2021 comparisons I mentioned earlier. • Both status and growth comparisons get at the same thing --- the impact of the pandemic on student attainment. • NOTE: Any analysis of pandemic impact (status or growth) requires comparison to a counterfactual baseline! • NOTE: If you provide 2021 results (status or growth), stakeholder will (rightly) want to compare that to pre-COVID (e.g., 2019) results. AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Understanding Impact AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Academic Recovery • With some understanding of the academic impact of the pandemic the next step is plan for and monitor recovery. • The magnitude of the impact (how much?) will indicate the magnitude of the recovery. • Early evidence from interim assessments suggest that in elementary school mathematics, students in some states have lost more than 12 weeks of learning. • Learning loss in ELA is more modest for these interim assessments. • Learning recovery will take on many forms (V-, U-, L-shaped recoveries) AzM2 Reporting and Analysis During the COVID-19 Pandemic

COVID-19 Pandemic: Monitoring Recovery AzM2 Reporting and Analysis During the COVID-19 Pandemic

Summary • Arizona is well positioned to with derive useful information from the AzM2 assessment results. • Issues potentially impacting the potential uses of the test scores (e.g., low participation, remote testing, shortened assessment forms) are minor to non-existent in Arizona • Calculation of skip-year growth with AzM2 assessments is straightforward and something Arizona and other states have done in the past. • The greatest challenge confronting states including Arizona is effectively communicating results from the state assessment to stakeholders. AzM2 Reporting and Analysis During the COVID-19 Pandemic

Questions AzM2 Reporting and Analysis During the COVID-19 Pandemic

About this copy

AzM2 Reporting and Analyses During the COVID-19 Pandemic — Arizona Department of Education, April 23, 2021.

This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.