Leveraging Assessment Derived Data to Inform COVID-19 School Support Strategies

CCSSO SCASS · February 25, 2021

Leveraging Assessment Derived Data to Inform COVID-19 School Support Strategies — CCSSO SCASS, February 25, 2021.
Published

February 25, 2021

Event
Conference
Slides
18
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Leveraging Assessment Derived Data to Inform COVID-19 School Support Strategies Nathan Dadey & Damian Betebenner The National Center for the Improvement of Educational Assessment February 25, 2021 ASR SCASS Draft Paper Available This work is licensed under a Creative Commons Attribution 4.0 International License.
  2. Background: Providing a Shared Understanding • The pandemic has complicated state assessment in unforeseen ways. • As states move forward with testing, being clear about the purposes of analyses will be more important than ever ▪ Time is short and the context is fraught ▪ Assessment is often confounded with accountability ▪ Planning of school supports should go hand in hand with plans for analysis Planning should start now, if it hasn’t already! www.nciea.org
  3. Background, Continued We can’t wait until the assessment data comes in to plan. • Plans for school support should be ongoing • Analyses of assessment should be targeted to provide needed information for those plans, while working within a number of constraints, including: ▪ Organizing and providing school support Timeline for suppor Is it Fall 2021? ▪ Turn around time for score reporting and analyses www.nciea.org
  4. Strands of Analysis Strand Description Focus Responsible Party Operational Typical yearly procedures and analyses Validity of test scores Assessment vendor(s) meant to support the technical quality • Comparability We hope this of test scores and intended • Interpretation doesn’t blow up interpretation(s) and uses of the assessment results. Investigative Prioritized analysis aimed at Effect on learning and Assessment vendor(s) and understanding the effects of the performance SEAs (with negotiated, clear pandemic on learning and • Overall effects lines of responsibility) performance. • Differential effects Note the interactive nature between the strands of analysis: • Findings from preliminary discovery analyses can inform operational analyses. • Valid test scores from operational analyses can support the findings of in-depth discovery analyses. www.nciea.org
  5. Key Questions of Interest • Who participated in the test in 2021? • What do the test scores mean in 2021? • Are the individual and aggregate scores in 2021 comparable to those in previous years? • What are appropriate (or inappropriate) uses of test scores in 2021? • Which schools and districts have the largest decreases in performance? ▪ What characteristics explain, or at least associate with, these decreases? • What subgroups display the largest decreases? • Do students in remote learning conditions perform differently than those in in-person learning conditions? • Etc. www.nciea.org
  6. Operational Analyses • Sound investigative analyses are predicated upon assessment data of high technical quality. ▪ Operational analyses tend to proceed investigative analyses ▪ Sound investigative analyses can occur with compromised assessment data, but such cases are rare (e.g., getting to the right conclusion with the wrong data) • Concerns about validity of the data derived from assessments abound, e.g., ▪ Comparability of remote versus in-person testing ▪ Impact of low participation rates (e.g., potential impacts on item parameters under post equating, possible impacts on normative statistics) www.nciea.org
  7. Investigative analyses • Assuming assessment data of good technical quality, what investigative analyses should be pursued? • USED’s recent relaxation of accountability provisions makes investigative analyses even more critical. • Typical analyses that support accountability are not the only, or even the major, stories we want to investigate this year. www.nciea.org
  8. Investigative analyses • Support planning/intervention is rooted in our ideas, or hypothesis, about what we think is happening to students ▪ Based on our current, incomplete picture of student experiences • These hypotheses should be tested and the results used to inform and revise our support plans ▪ Support plans should be ready to run with the results of planned analyses (and in worse case scenarios, run without the data anyway) ▪ Support plans should be amenable to change as new data/ evidence becomes available. www.nciea.org
  9. Support Planning Planning & Developing: Operational • Key questions Implement- Analyses ation • Benchmarks based on Simulations using Historical Data Investigative • Code based on Analyses simulated data Feb March April May June July August Sep www.nciea.org
  10. Support Implement- Planning ation Planning & Developing: Operational • Key questions Implement- Analyses ation • Benchmarks based on Simulations using Historical Data Implement- Investigative • Code based on ation Analyses simulated data Feb March April May June July August Sep www.nciea.org
  11. Some Overarching Questions • How did overall performance change relative to prior years? • Who has the lowest performance? • How does performance compare across key groups? www.nciea.org
  12. Using Investigative Analyses to Support Planning • Hypothesis: The academic impact (sometimes referred to as learning loss) of the pandemic is being unevenly felt by students. • If true, targeted intervention is more prudent than blanket intervention. • For states considering targeted assistance: ▪ Who needs help? Grades, schools, districts, demographic subgroups, COVID related subgroups, English Language Learners. ▪ What do they need help in? Mathematics, ELA, Science, non-tested subjects? ▪ How much help do they need? Is recovery needed large or small? www.nciea.org
  13. Many Factors to Consider • Unit. At what level will the analyses be focused? ▪ Individual students ▪ Classrooms, schools, districts, or the state overall ▪ Specific subgroups (e.g., race/ethnicity, economically disadvantaged, students with disabilities, English language learners) ▪ Specific grades (and grade-groupings, like early elementary) or content areas ▪ Learning conditions (e.g. remote, in-person, hybrid) • Measure. What outcome measure will be examined? ▪ Achievement (e.g., level scale scores or proficiency levels), often in terms of cross-sectional comparisons ▪ Growth (e.g., score differences on a vertical scale, student growth percentiles), often in terms of longitudinal comparisons ▪ Gaps between groups defined in terms of achievement or growth www.nciea.org
  14. Prioritizing Analyses • Internal vs. External ▪ Are the results are intended for use outside of department, and if so, what specific stakeholders are meant to be addressed? • Availability ▪ How easily accessible will the data be? ▪ Like the distinction of internal vs. external, level of access involves stakeholders and refers to the specific ways in which the results will be provided and communicated (e.g., posted on the department website, communicated through a press release, as part of a webinar). www.nciea.org
  15. Prioritizing Analyses • Reporting Format ▪ How will the results be provided (e.g., a simple spreadsheet, posted as part of the state’s report card, as an interactive web-based visualization)? • Timing ▪ Is there a specific window of time by which the results are needed? For example, is there a window of time an analysis needs to be completed by to support an intervention? ▪ Results will likely come in waves from different data sources, how will this impact timing. • Responsible Party ▪ Who will be conducting the analyses and what is their capacity to implement the analyses? Will the analysis be seen as credible to key stakeholders when coming from that responsible party? www.nciea.org
  16. Additional Considerations Contextual data. How can contextual data, such OTL and learning and testing environment data, be collected in 2021 to help contextualize findings? ▪ See, for example, Marion (2020) ▪ Consider asking experts, such as the TAC to review your survey or data collection instruments. Interaction effects. How can the various factors influencing 2021 test score be disentangled? www.nciea.org
  17. Example Investigative analyses for 2021 Participation • Participation rates on assessments are likely to be substantially less than in normal years. • Understanding who and who did not participate in state assessments will be critical in interpreting overall results. ▪ By demographic subgroups (e.g., ses, ethnicity) ▪ By mode of instruction (remote/hybrid/in person learner) ▪ By grade, school, district www.nciea.org
  18. Example Investigative analyses for 2021: Using Growth to Assesses Impact • Student academic growth analyses can be used to assess the academic impact of the pandemic. • Baseline skip-year growth norms (2017 to 2019) can be used with 2019 to 2021 data to assess how much “slower” students learned during the pandemic relative to pre- pandemic times. • Differential impact on groups can be investigated by aggregating up SGPs at the group level. • With accurate mode of instruction data (remote/hybrid/in person), impact of mode of instruction can be assessed. www.nciea.org