COVID-19 Academic Impact and Recovery

class: center, middle, inverse, title-slide

.title[
# COVID-19 Academic Impact and Recovery
]
.subtitle[
## Conceptualization and Implementation
]
.author[
### Damian Betebenner
]
.institute[
### Center for Assessment
]
.date[
### <p>June 30th, 2022<br>CCSSO ASR-TILSA 2022<br>Summer Meeting</p>
<a href="https://github.com/dbetebenner/TILSA_2022_Covid_19_Impact_Recovery/blob/main/docs/Covid_Academic_Impact_and_Recovery.Rmd" target="_blank" class="github-corner" aria-label="View source on Github"><svg viewBox="0 0 250 250" aria-hidden="true"><path d="M0,0 L115,115 L130,115 L142,142 L250,250 L250,0 Z"></path><path class="octo-arm" d="M128.3,109.0 C113.8,99.7 119.0,89.6 119.0,89.6 C122.0,82.7 120.5,78.6 120.5,78.6 C119.2,72.0 123.4,76.3 123.4,76.3 C127.3,80.9 125.5,87.3 125.5,87.3 C122.9,97.6 130.6,101.9 134.4,103.2" fill="currentColor" style="transform-origin: 130px 106px;"></path><path class="octo-body" d="M115.0,115.0 C114.9,115.1 118.7,116.5 119.8,115.4 L133.7,101.6 C136.9,99.2 139.9,98.4 142.2,98.6 C133.8,88.0 127.5,74.4 143.8,58.0 C148.5,53.4 154.0,51.2 159.7,51.0 C160.3,49.4 163.2,43.6 171.4,40.1 C171.4,40.1 176.1,42.5 178.8,56.2 C183.1,58.6 187.2,61.8 190.9,65.4 C194.5,69.0 197.7,73.2 200.1,77.6 C213.8,80.2 216.3,84.9 216.3,84.9 C212.7,93.1 206.9,96.0 205.4,96.6 C205.1,102.4 203.0,107.8 198.3,112.5 C181.9,128.9 168.3,122.5 157.7,114.1 C157.9,116.9 156.7,120.9 152.7,124.9 L141.0,136.5 C139.8,137.7 141.6,141.9 141.8,141.8 Z" fill="currentColor"></path></svg></a>
]

---

layout: true

<div class="footer" style="left: 5%;"><img src="https://mirrors.creativecommons.org/presskit/icons/cc.svg" style = "max-width: 40%; opacity: 0.5;"></img></div>
<div class="footer" style="left: 7.5%;"><img src="https://mirrors.creativecommons.org/presskit/icons/by.svg" style = "max-width: 40%; opacity: 0.5;"></img></div>
<div class="footer" style="left: 40%;"><a class="footer" href="https://dbetebenner.github.io/TILSA_2022_Covid_19_Impact_Recovery">COVID-19 Academic Impact and Recovery</a></div>

---

<div>
<style type="text/css">.xaringan-extra-logo {
width: 55px;
height: 64px;
z-index: 0;
background-image: url(https://centerforassessment.github.io/assets/favicon.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:0.5em;right:0.5em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://www.nciea.org'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---
class: inverse, center, middle

# Conceptualizing Impact & Recovery

---

# What is "academic impact"?

- The pandemic and all the ensuing disruptions functioned/functions as an “academic headwind”, impeding (in general) the academic progress of students.

- Headwinds impede progress in two ways:

1. They slow one’s rate of progress (speedometer)
    2. And by slowing one’s rate of progress they lead to less distance being travelled (odometer).

- In education these two impediments manifest as:

1. Decrease in student growth (i.e. decrease in velocity = deceleration).
    2. Decrease in student attainment.

---

# Learning Deceleration/Acceleration

- Summarization of assessment data (either for diagnostic or accountability purposes) emphasizes two types of data results: Status (i.e. student attainment) and Growth (i.e., student academic progress)

- With the pandemic, emaphasis has changed to look at _academic impact_

- We emphasize that Impact is synonymous with _decleration_ and Recovery is synonymous with _acceleration_

- Deceleration, by definition, is the change in velocity (i.e. growth)

- Improvement (increasing student attainment/status in a system over time) is indicative of acceleration (unless the population is changing leading to the increased attainment/status)

---
class: center, middle

### [Focusing State Educational Accountability Systems: Four Methods of Judging School Quality and Progress](https://www.nciea.org/sites/default/files/publications/Dale020402.pdf)

---

# COVID-19 Academic Impact

- Discussion of academic impact requires the consideration of a counterfactual:

#### <center><span style="color: red;">Where would student(s) have been had the pandemic not occurred?</span></center>

- The impact to student attainment and student growth frame the two primary types of analyses that will be discussed

1. Academic Impact as it relates to change (i.e. decrease) in student attainment
    2. Academic Impact as it relates to change (i.e., decrease) in student growth

---

# Data for Analyses

- States drew primarily upon their large scale assessment data to investigate academic impact

- State summative assessment data given in grade 3 to 8 in ELA and Mathematics
    - State English Language Proficiency (ELP) assessment data (e.g., WIDA-ACCESS) given in grades K to 12

---

# Academic Impact Validity Threats

- Student particition: Participation in states varied greatly. Several states reported normal participation (> 95%) while others had participation well below 50%.

- Pandemic impact on student test-taking motivation.

- Test adminstration modality

- No evidence of differential test-taking motivation pre-pandemic to pandemic.

---

# Status based analyses of academic impact

- Change in status (i.e. attainment): Compare the status of a pre-pandemic students with students in during/after the pandemic.

- States compare same grade/content area performance from 2019 to 2021 (e.g., Grade 5 math in 2019 with Grade 5 math in 2021)

- The most common quantity used to communicate change in status is change in percent proficient

- Though not "wrong", change in percent proficient (or percent above/below any threshold) is not usually a good measure of central tendency.

- One can also calculate change in mean scale score (standardized, to create an effect size)

---

# Status based analysis of academic impact

- To refine status based measures of change, we calculate the equi-percentile distributional shift associated with pre-pandemic and pandemic score.

#### `$$\mbox{Status Impact} \equiv (\mbox{Scale Score}_{2019} - \mbox{Scale Score}_{2021} | \mbox{Percentile Rank})$$`

- We created a custom graphics class in R to help illustrate this

---
class: center, middle

---
class: center, middle

---

# Growth based analysis of academic impact

- States analyzed utilize Student Growth Percentiles (SGPs) as its growth model

- SGPs used by states are "norm-referenced",  utilizing fresh growth norms each year

- To investigate academic impact, baseline growth norms (i.e., coefficient matrices) were calculated:

- Baseline coefficient matrices utilized 2016, 2017 and 2019 to accomodate skip-year growth
    - WIDA-ACCESS coefficient matrices utilized 2018 and 2019.

- The extent to which median/mean SGPs in 2021 deviate below 50 is indicative of deceleration (i.e., slow down in student learning)

---
class: center, middle

---
class: center, middle

---

# Categorization of Impact

- __Severe:__ `$\mbox{Growth Impact} \leq -25$` or `$\mbox{Status Impact} \leq -0.4$`.
- __Large:__ `$-25 < \mbox{Growth Impact} \leq -15$` or `$-0.4 < \mbox{Status Impact} \leq -0.25$`.
- __Moderate:__ `$-15 < \mbox{Growth Impact} \leq -5$` or `$-0.25 < \mbox{Status Impact} \leq -0.1$`.
- __Modest to None:__ `$-5 < \mbox{Growth Impact} \leq 5$` or `$-0.1 < \mbox{Status Impact} \leq 0.05$`.
- __Improvement:__ `$\mbox{Growth Impact} > 5$` or `$\mbox{Status Impact} > 0.05$`.

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
# Summarization of Academic Impact Across Content Area and Grade

- The vast amounts of data makes it difficult for users to grasp who and to what extent academic impact is occurring

- We created another graphic class in R to help summarize this vast amount of data across content area and subgropup

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---
class: center, middle

---

# Across state views of academic impact

- The Center for Assessment worked with about a dozen states on understand pandemic related academic impact

- Applying the same set of analyses to all states allows us to help individual states see their impact and impact on student subgroups against other states.

---
class: center, middle

---
class: center, middle

---

# Summary

- Data from 2021 State assessments was successfully marshaled to investigate COVID-19 related academic impact

- No evidence of test invaldity was detected to undermine the analyses

- Academic impact results were summarized across grade, content area, demographic, academic subgroups and districts to help stakeholders in Indiana identify who and to what extent students were academically impact by the pandemic

- Data from 2022 state assessments will be used in the coming weeks to investigate whether and the extent to which students are recovering academically from the pandemic related losses.