New Hampshire Pandemic Related Academic Impact

New Hampshire Department of Education · November 7, 2021

New Hampshire Pandemic Related Academic Impact — New Hampshire Department of Education, November 7, 2021.
Published

November 7, 2021

Event
State agency
Slides
39
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. New Hampshire Pandemic Related Academic Impact Scott Marion, Damian Betebenner & Adam Van Iwaarden Center for Assessment New Hampshire Department of Education Technical Advisory Meeting November 8th, 2021
  2. New Hampshire Testing in 2021 • As you are all aware, the pandemic made for a complicated year of testing. • Early in 2021, accountability waivers were granted for states but testing waivers were not universally granted. • New Hampshire administered both the SAS and the WIDA- ACCESS assessments. • SAS is administered in grades 3 to 8 in mathematics and English language arts. (recall, no SAS testing in 2020) • WIDA-ACCESS is administered in grades K to 12. (recall, there was WIDA-ACCESS testing in 2020). New Hampshire Academic Impact Analyses
  3. Pandemic Related Academic Impact • The pandemic and all the various, ensuing disruptions functions as a kind of “academic headwind”, impeding (in general) the academic progress of students. • Headwinds impede progress in two ways: • They slow one’s rate of progress • The slower rate of progress leads to less distance being travelled. • In education these two impediments manifest themselves (roughly) with assessments as: • Decrease in student growth (decrease in learning velocity). • Decrease in student attainment (decrease in amount learned). • In some states, understanding this phenomenon requires special consideration of low participation/changing demographics. New Hampshire Academic Impact Analyses
  4. Pandemic Related Academic Impact • Understanding pandemic related impact necessarily requires looking back to pre-pandemic results. • The impulse of stakeholders to want to compare 2021 results with 2019 results is the correct impulse at both the individual, school, district, and state level. • Wanting to look at 2019 to 2021 change is an attempt to investigate impact. • The challenge is making 2019 to 2021 comparisons apples-to-apples comparisons. • Supporting these comparisons requires having a complete understanding of who tested and who didn’t. New Hampshire Academic Impact Analyses
  5. New Hampshire SGP Operational Considerations in 2021 • New Hampshire SAS growth analyses are skip-year/gap year growth from 2019 to 2021: • ELA – Grade 3/2019 to Grade/5 2021; Grade 4/2019 to Grade 6/2021; Grade 5/2019 to Grade 7/2021; Grade 6/2019 to Grade 8/2021. • Mathematics – Grade 3/2019 to Grade 5/2021; Grade 4/2019 to Grade 6/2021; Grade 5/2019 to Grade 7/2021; Grade 6/2019 to Grade 8/2021. • New Hampshire WIDA-ACCESS growth analyses are consecutive year growth from 2020 to • In 2021, two types of SGP analyses were conducted for New Hampshire data: • Cohort referenced analyses: These analyses create growth norms from the 2019 to 2021 students. These growth norms yield cohort means/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2019 and prior data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact. • Projections (lagged and straight) were also calculated for 2021 data using baseline growth norms in case they are needed. New Hampshire Academic Impact Analyses
  6. SGP Analyses and Academic Impact • 2019 to 2021 Status and Growth comparisons get at the same thing: deceleration in student learning due to the COVID-19 pandemic “headwind”. • Status Comparison: Compare same grade attainment results (e.g., percent proficient or mean scale score) in 2019 and 2021. • Growth Comparison: Compare baseline SGP results in 2019 and 2021. • One of the best ”unread” papers on accountability system design and indicators is by Dale Carlson where he discusses the relationship between changes in status (i.e., improvement) and changes in growth (acceleration). https://www.nciea.org/sites/default/files/publications/ Dale020402.pdf • We present (briefly) status results in addition to growth results. New Hampshire Academic Impact Analyses
  7. SGP Summary of Cohort versus Baseline Referenced • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Typical way that SGPs are calculated. • Mean/median will be 50 for all grade x content area analyses. • Cannot investigate overall impact of the pandemic on students. • Can investigate differential impact of the pandemic on students. • Does not require scale stability across years. • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (usually below) 50 is indicative of less learning/pandemic related academic impact. • A mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact and differential impact • Requires scale stability across years. New Hampshire Academic Impact Analyses
  8. SGP Calculations • SGP calculations were performed using the SGP package and R software environment based upon data supplied by CDE. • To facilitate the calculation of baseline SGPs across multiple states, we created an additional R package, SGPmatrices, that contains New Hampshire matrices, amongst others. • https://centerforassessment.github.io/SGPmatrices/ • Source code associated with calculations are available in the New Hampshire GitHub repository. New Hampshire Academic Impact Analyses
  9. SGP Assumptions • SGP analyses (cohort and baseline referenced) presume valid scale scores • Students forming the growth norm group are presumed to have taken the same (or equivalent) test at the same time. Scores across students in that instance should mean the same thing. • SGPs, however, are often used as a post-hoc check of scale score validity/comparability via Reductio ad absurdum • In 2021, check, for example, form comparability, mode-of- administration comparability, etc. • If scores are not comparable, then we either attempt to make them comparable or run separate analyses. New Hampshire Academic Impact Analyses
  10. Scale Score Validity • Prior to the arrival of 2021 test results (across many states), there were pervasive concerns about ”comparability” of results to previous (e.g., • The term comparability is often used in two different ways: • Comparability of individual scale scores: Does the scale mean the same thing that it used to. • Comparability of aggregate scores (e.g., percent proficient). Is percent proficient (e.g., for a school) comparable to prior years. • Both issues are of relevance. Note that without comparability of individual scale scores, you lose comparability at the aggregate level. • We focus initially on comparability of individual scale scores as that it critical for the calculation of baseline referenced SGPs. New Hampshire Academic Impact Analyses
  11. Scale Score Validity • Threats to comparability of individual level scores can occur due to numerous conditions including: • Different modes of administration (in person versus remote) • Differential motivation of test takers. • Equating anomalies (e.g., post-equating of results over the course of the pandemic) • Differential test taking conditions including altered/shortened test forms. • We note that lower participation rates in and of themselves are not sufficient to undermine scale score validity unless these conditions can be shown to somehow impact the meaning of student scores. New Hampshire Academic Impact Analyses
  12. Scale Score Validity • Given plans to calculated baseline referenced SGPs in numerous states, the Center put in place several checks to ensure validity of scale scores. • A priori Checks: • Reviewed results from states/organizations using baseline SGPs • Ran sensitivity analyses to examine to what extent perturbed data impact baseline SGP analyses. • Examined equating and psychometric reports (e.g., person fit statistics) for 2021 to detect anomalous data. • Post-hoc Checks: • Compared results across states to see whether results conform to what is showing up in other states. • Inspected results within state looking for anomalous results which may indicate issues related to scale stability. New Hampshire Academic Impact Analyses
  13. Baseline referenced SGPs • Baseline referenced SGPs have been calculated in several states as well as with interim assessments over the last decade. • The Georgia Department of Education calculated baseline SGPs and used them as part of their accountability system for 5 years in the early 2010s prior to the assessment transition in 2015. • Baseline SGPs demonstrated excellent stability for grade level assessments. • Goal of using baseline SGPs was to avoid zero-sum game associated with norm- referenced analyses each year. • Renaissance Learning uses baseline matrices to calculate SGPs in real time. • Quantile regression is equivariant to monotone transitions to scale so, in general, robust to minor perturbations of scale. New Hampshire Academic Impact Analyses
  14. Baseline referenced SGPs • We are not aware of any psychometric concerns regarding the validity of scale scores reported by NHDOE. • Baseline growth results for New Hampshire are, in general, consistent with what has been observed in the dozen states we have examined nationally. • In the following slides we report the baseline referenced SGPs. • We also show 2019 SGPs (baseline/skip-year) to illustrate academic impact. New Hampshire Academic Impact Analyses
  15. SAS Attainment: 2019 to 2021 Percent Proficient ELA Mathematics Grade % Proficient 2019 % Proficient Difference 2021 to % Proficient 2019 % Proficient 2021 Difference 2021 to (Count/Participation 2021 (Count/ 2019 (Count/Participation (Count/Participation 2019 Rate) Participation Rate) Rate) Rate) New Hampshire Academic Impact Analyses
  16. Points of Clarification for Status Results • Decrease in attainment (as a decrease in percent proficient) between 2019 and 2021 is the most often reported statistic related to state testing in 2021. • This analysis & statistic (percent proficient) are problematic. • Using 2019 as a counterfactual for where students would have been had the pandemic not occurred is problematic for several reasons including: • Changes in student cohort composition is not accounted for. • Improvement between 2019 and 2021 that would have occurred is not accounted for. • Missing data in 2021 not accounted for. New Hampshire Academic Impact Analyses
  17. Points of Clarification for Status Results • Drops in percent proficient are: • Modest in ELA when compared to what has been observed in other states (high single digit decreases) /mathematics low double-digit decreases). • Typical in mathematics when compared to what has been observed in other states (low to mid teen decreases) • Based upon multiple imputation analyses, decreases observed in percent proficient almost certainly underestimate the true decrease in percent proficient had all students tested. • Participation rates (derived from decreases in observed test scores between 2019 and 2021) vary by grade and state/organization but suggest participation between 75% and 85% (mostly toward the lower end) • In general, current results based upon incomplete data likely represent a best-case scenario compared to what would appear given fuller participation. New Hampshire Academic Impact Analyses
  18. SAS: 2019 to 2021 Median Baseline SGPs by Grade ELA Mathematics Grade Median SGP 2019 Median SGP Difference 2021 to Median SGP 2019 Median SGP 2021 Difference 2021 to 3 NA NA NA NA NA NA 4 NA NA NA NA NA NA New Hampshire Academic Impact Analyses
  19. Discussion • Impacts in mathematics exceeded those in ELA. • Decreases in median SGP in mathematics were, in general, in the lower twenties. • Decreases in median SGP in ELA were in the middle single digits. • The results in ELA are some of the smallest decreases we’ve seen nationally. • The results in mathematics were consistent with many of the larger impact states. • The fact that growth is uncorrelated with prior achievement implies missing data impacts are much more muted with growth than with status. • In other states we’ve classified academic impacts as: • > -5: Modest to None • -15 to -5: Moderate • < -15: Large New Hampshire Academic Impact Analyses
  20. WIDA-ACCESS Results for 2020/2021 Median Composite Achievement Level Median Composite SGP Baseline Growth GRADE 2020 2021 Difference 2020 2021 Difference New Hampshire Academic Impact Analyses
  21. Discussion • Impacts in lower grades much more severe than those in middle schools. • Impacts in high school grades are negligible to non-existent. • Same patterns seen in a dozen other WIDA states we have looked at. • Based upon a presentation by WIDA at NCSA/CCSSO in June, drops are largely attributable to decreases in scores in speaking. • Lack of impact in high school grades likely indicates that typical, pre-pandemic growth represented impoverished rates of learning. New Hampshire Academic Impact Analyses
  22. SAS: 2019 to 2021 Median SGPs by Ethnicity ELA Mathematics Ethnicity Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021 2019 2021 to 2019 2019 2021 to 2019 American Indian/ 42.0 (115) 47.5 (78) -8.0 45.5 (112) 24.0 (79) -21.5 Alaskan Native Asian 62.0 (1,394) 61.0 (1,065) -9.0 66.0 (1,419) 48.0 (1,068) -18.0 Black/African 44.0 (704) 39.0 (587) -4.0 45.0 (725) 25.0 (611) -20.0 American Hispanic/Latino 48.0 (2,770) 38.0 (2,094) -8.0 47.0 (2,837) 22.0 (2,155) -25.0 White 50.0 (37,376) 46.0 (30,440) -7.0 50.0 (37,410) 29.0 (30,501) -21.0 Two or More 55.0 (1,375) 48.0 (1,143) -7.0 54.0 (1,387) 28.0 (1,148) -26.0 Races New Hampshire Academic Impact Analyses
  23. Discussion of Ethnicity Results • Broken out by ethnicity, 2021 baseline SGP results show similar rank ordering (i.e., growth gaps) as seen in normal years. • Student academic growth is lower in mathematics than in ELA for all ethnic subgroups. • Growth impacts by ethnicity show some variability. For example, Hispanic students more impacted than other groups --- though all groups are impacted, especially in mathematics. New Hampshire Academic Impact Analyses
  24. SAS Growth: 2019 to 2021 Median SGPs by Disadvantaged Status ELA Mathematics Free/Reduced Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021 Lunch Status 2019 2021 to 2019 2019 2021 to 2019 No 52.0 (32,030) 49.0 (27,494) -3.0 53.0 (31,998) 31.0 (27,567) -22.0 Yes 45.0 (11,750) 35.0 (7,940) -10.0 43.0 (11,939( 23.0 (8,023) -20.0 New Hampshire Academic Impact Analyses
  25. Discussion of disadvantaged status • Baseline SGP results show similar growth gaps in 2021 (students indicated as receiving free/reduced lunch have lower SGPs than students indicated as not receiving free/reduced lunch. • Academic impacts are for students receiving free/reduced lunch in 2021 were larger than for those students not receiving. • The result differs from other states where free/non-free reduced lunch groups generally show (roughly) the same impact. New Hampshire Academic Impact Analyses
  26. SAS Growth: 2019 to 2021 Median SGPs by Gender ELA Mathematics Gender Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021 2019 2021 to 2019 2019 2021 to 2019 Female 53.0 (21,400) 49.0 (17,255) -4.0 51.0 (21,459) 30.0 (17,310) -21.0 Male 46.0 (22,370) 42.0 (18,179) -4.0 48.0 (22,468) 29.0 (18,280) -19.0 New Hampshire Academic Impact Analyses
  27. Discussion of Gender Results • Growth gender gaps historically exist (favoring females) in both ELA and mathematics. • Growth gender gaps are historically larger in ELA than mathematics. • Gaps are similarly large in math and ELA for males and females. New Hampshire Academic Impact Analyses
  28. SAS Growth: 2019 to 2021 Median SGPs by IEP Status ELA Mathematics IEP Status Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021 2019 2021 to 2019 2019 2021 to 2019 IEP No 52.0 (36,582) 49.0 (29,440) -3.0 52.0 (36,685) 30.0 (29,534) -22.0 IEP Yes 38.0 (7,198) 32.0 (5,994) -6.0 39.0 (7,252) 27.0 (6,056) -12.0 New Hampshire Academic Impact Analyses
  29. Discussion of IEP Status Results • IEP status gaps historically exist and favor non-IEP students in both ELA and mathematics. • Gaps in 2021 were larger for IEP students in ELA than non-IEP students. The opposite was true in Mathematics. • Potential explanation include: • Concerted effort in mathematics to educate IEP students during the pandemic thus minimizing the academic impact to that group of students. • Impoverished growth rates pre-pandemic such that pandemic impacts didn’t substantially change rates of learning. New Hampshire Academic Impact Analyses
  30. Participation Rates and Interpretation • An issue impacting the interpretation of growth (and status) results in 2021 is participation in the state assessment. • Low participation doesn’t impact calculation of cohort referenced growth norms nor use of baseline referenced growth norms. • Low participation does impact comparisons of group level results with historical (e.g., 2019) results based upon higher participation rates. • For example, how should one compare growth and status results for school in 2019 and 2021 with very different rates of participation. New Hampshire Academic Impact Analyses
  31. Multiple Imputation • Given the rich historical data available for students missing in 2021, and the observed performance of comparable students from 2019 to 2021, it is possible to simulate what we think their performance might have been. • Combining multiple simulations can inform us about potential bias in what we observed at state/district/school levels in 2021, as well as inform us about levels of uncertainty around them. • Multiple imputation (MI) of missing data is widely adopted and well-developed statistical analysis technique used in many fields to deal with missing data problems. New Hampshire Academic Impact Analyses
  32. Multiple Imputation • MI uses information from the observed data to generate parameter estimates in three steps: • Imputation: A prediction model generates a set of plausible values for the missing observations, resulting in M imputed data sets. • Analysis: The analysis (e.g., baseline student growth percentiles) is conducted on each of the M data sets. • Pooling: Parameter and (more importantly) uncertainty estimates are constructed by pooling across the M analyses. New Hampshire Academic Impact Analyses
  33. Multiple Imputation • In the context of COVID impact analyses, we use the multiple (plausible) parameter estimates to inform us about the uncertainty around what was observed. That is, we are more interested in using MI to determine whether we trust the statistics calculated from the observed (with missing) data than to produce “corrected” parameter estimates. • Statistics of particular interest: • Mean/median scale scores • Mean/median baseline SGPs • Effect size estimates • Percent Proficient New Hampshire Academic Impact Analyses
  34. Multiple Imputation • Before performing a MI analysis with New Hampshire data, missing data patterns were investigated. • Patterns suggest some relationship to factors such economic status and ethnicity. • The relationship with prior achievement is relatively small. • These relationships suggest that missingness in New Hampshire is non-ignorable, but does not preclude the use of MI. • Data can be considered “Missing-at-random” – relationships exist between missingness and the observed data (and so can not be considered “missing completely-at-random”) but the relationships are not systematic (e.g. all students below proficiency in 2019 were missing in 2021). New Hampshire Academic Impact Analyses
  35. Multiple Imputation • Given the 2021 missingness patterns and the good coverage of observed scores for students from all demographic groups and prior achievement levels, MI is a reasonable approach to dealing with SAS missing data. • Imputation models included: • Student prior achievement • Student demographics • FRL Status • Disability status • English language proficiency status • Gender • Ethnicity dummy variables for White, African American, Hispanic and Asian students • School level clustering • 30 imputations (M = 30 with a maximum of 10 iterations per imputation) were conducted. New Hampshire Academic Impact Analyses
  36. Multiple Imputation • Imputation models were restricted to include only students who were enrolled, but not tested. • Knowing school/district affiliation is critical for assessing the impact of missing data for subsequent analyses at these levels. • Excluding students who “disappeared” from the system may not give us a full account of the impact of missing data at the state level. However, a comparison of the enrolled vs. non-enrolled missing students does not lead us to believe their inclusion would meaningfully change inferences at the state level. New Hampshire Academic Impact Analyses
  37. Communication of Results in 2021 and beyond • Technical challenges to the calculation of growth in 2021 are likely minor compared to communication challenges. • State reporting is, in general, highly aligned with accountability. • Even though analyses outlined herein deal with status/growth, they don’t necessarily align with accountability narratives. • It is critical for leadership (at the highest level) to map out a communication plan and then build analytics and reporting to support that plan. • Without a well considered, disciplined and coordinated communication plan, accountability narratives are likely to take over. New Hampshire Academic Impact Analyses
  38. SGP Analysis Plans post 2021 (2022 and beyond) • Spring 2021 represented an ideal time to get a snapshot of student attainment as we are presumably/hopefully at or near the point of maximal impact on student attainment. • That is, recovery has not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Analytics in 2022 will be keenly interested about whether recovery is occurring: Who, what, how much. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery! New Hampshire Academic Impact Analyses
  39. Summary of Findings • SGP analyses were successfully conducted for New Hampshire SAS/WIDA- ACCESS testing in spring 2021 • For the SAS, results of the analyses suggest modest academic impacts in ELA and large academic impacts in mathematics due to the pandemic. • For WIDA-ACCESS, academic impact was large in elementary grades, moderate in middle school grades, and no academic impact was indicated in the high school grades (likely due to the fact that pre-pandemic growth was already impoverished for high school students). • Results suggest uniform impact by content area across elementary, middle and high school. • Recovery from impacts this large will likely take multiple years and require additional supports beyond that provided by normal schooling. New Hampshire Academic Impact Analyses