New Hampshire Pandemic Related
Academic Impact
Scott Marion, Damian Betebenner & Adam Van Iwaarden
Center for Assessment
New Hampshire Department of Education
Technical Advisory Meeting
November 8th, 2021
New Hampshire Testing in 2021
• As you are all aware, the pandemic made for a complicated year
of testing.
• Early in 2021, accountability waivers were granted for states but
testing waivers were not universally granted.
• New Hampshire administered both the SAS and the WIDA-
ACCESS assessments.
• SAS is administered in grades 3 to 8 in mathematics and English language arts.
(recall, no SAS testing in 2020)
• WIDA-ACCESS is administered in grades K to 12. (recall, there was WIDA-ACCESS
testing in 2020).
New Hampshire Academic Impact Analyses
Pandemic Related Academic Impact
• The pandemic and all the various, ensuing disruptions functions as a kind of
“academic headwind”, impeding (in general) the academic progress of
students.
• Headwinds impede progress in two ways:
• They slow one’s rate of progress
• The slower rate of progress leads to less distance being travelled.
• In education these two impediments manifest themselves (roughly) with
assessments as:
• Decrease in student growth (decrease in learning velocity).
• Decrease in student attainment (decrease in amount learned).
• In some states, understanding this phenomenon requires special
consideration of low participation/changing demographics.
New Hampshire Academic Impact Analyses
Pandemic Related Academic Impact
• Understanding pandemic related impact necessarily requires looking
back to pre-pandemic results.
• The impulse of stakeholders to want to compare 2021 results with 2019
results is the correct impulse at both the individual, school, district, and
state level.
• Wanting to look at 2019 to 2021 change is an attempt to investigate
impact.
• The challenge is making 2019 to 2021 comparisons apples-to-apples
comparisons.
• Supporting these comparisons requires having a complete
understanding of who tested and who didn’t.
New Hampshire Academic Impact Analyses
New Hampshire SGP Operational Considerations in 2021
• New Hampshire SAS growth analyses are skip-year/gap year growth from 2019 to 2021:
• ELA – Grade 3/2019 to Grade/5 2021; Grade 4/2019 to Grade 6/2021; Grade 5/2019 to Grade 7/2021;
Grade 6/2019 to Grade 8/2021.
• Mathematics – Grade 3/2019 to Grade 5/2021; Grade 4/2019 to Grade 6/2021; Grade 5/2019 to Grade
7/2021; Grade 6/2019 to Grade 8/2021.
• New Hampshire WIDA-ACCESS growth analyses are consecutive year growth from 2020 to
• In 2021, two types of SGP analyses were conducted for New Hampshire data:
• Cohort referenced analyses: These analyses create growth norms from the 2019 to 2021 students.
These growth norms yield cohort means/median SGPs of 50 and obscure COVID-19 academic impact.
• Baseline referenced analyses: Historical growth norms were created using 2019 and prior data. These
growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall
COVID-19 academic impact.
• Projections (lagged and straight) were also calculated for 2021 data using baseline growth
norms in case they are needed.
New Hampshire Academic Impact Analyses
SGP Analyses and Academic Impact
• 2019 to 2021 Status and Growth comparisons get at the same thing:
deceleration in student learning due to the COVID-19 pandemic
“headwind”.
• Status Comparison: Compare same grade attainment results (e.g., percent
proficient or mean scale score) in 2019 and 2021.
• Growth Comparison: Compare baseline SGP results in 2019 and 2021.
• One of the best ”unread” papers on accountability system design and
indicators is by Dale Carlson where he discusses the relationship
between changes in status (i.e., improvement) and changes in growth
(acceleration). https://www.nciea.org/sites/default/files/publications/
Dale020402.pdf
• We present (briefly) status results in addition to growth results.
New Hampshire Academic Impact Analyses
SGP Summary of Cohort versus Baseline Referenced
• Cohort referenced SGPs
• Growth norms created from the current cohort of students.
• Typical way that SGPs are calculated.
• Mean/median will be 50 for all grade x content area analyses.
• Cannot investigate overall impact of the pandemic on students.
• Can investigate differential impact of the pandemic on students.
• Does not require scale stability across years.
• Baseline referenced SGPs.
• Growth norms created from historical (i.e., pre-COVID) cohorts of students.
• Mean/median 50 for historical cohort, not necessarily 50 for current cohort.
• Extent of deviation from (usually below) 50 is indicative of less learning/pandemic related
academic impact.
• A mean SGP deviation of 5 corresponds to an effect size of 0.18.
• Can investigate BOTH overall impact and differential impact
• Requires scale stability across years.
New Hampshire Academic Impact Analyses
SGP Calculations
• SGP calculations were performed using the SGP package and R
software environment based upon data supplied by CDE.
• To facilitate the calculation of baseline SGPs across multiple
states, we created an additional R package, SGPmatrices, that
contains New Hampshire matrices, amongst others.
• https://centerforassessment.github.io/SGPmatrices/
• Source code associated with calculations are available in the
New Hampshire GitHub repository.
New Hampshire Academic Impact Analyses
SGP Assumptions
• SGP analyses (cohort and baseline referenced) presume valid
scale scores
• Students forming the growth norm group are presumed to have taken
the same (or equivalent) test at the same time. Scores across students
in that instance should mean the same thing.
• SGPs, however, are often used as a post-hoc check of scale score
validity/comparability via Reductio ad absurdum
• In 2021, check, for example, form comparability, mode-of-
administration comparability, etc.
• If scores are not comparable, then we either attempt to make them
comparable or run separate analyses.
New Hampshire Academic Impact Analyses
Scale Score Validity
• Prior to the arrival of 2021 test results (across many states), there were
pervasive concerns about ”comparability” of results to previous (e.g.,
• The term comparability is often used in two different ways:
• Comparability of individual scale scores: Does the scale mean the same thing
that it used to.
• Comparability of aggregate scores (e.g., percent proficient). Is percent
proficient (e.g., for a school) comparable to prior years.
• Both issues are of relevance. Note that without comparability of
individual scale scores, you lose comparability at the aggregate level.
• We focus initially on comparability of individual scale scores as that it
critical for the calculation of baseline referenced SGPs.
New Hampshire Academic Impact Analyses
Scale Score Validity
• Threats to comparability of individual level scores can occur due to
numerous conditions including:
• Different modes of administration (in person versus remote)
• Differential motivation of test takers.
• Equating anomalies (e.g., post-equating of results over the course of the
pandemic)
• Differential test taking conditions including altered/shortened test forms.
• We note that lower participation rates in and of themselves are not
sufficient to undermine scale score validity unless these conditions can
be shown to somehow impact the meaning of student scores.
New Hampshire Academic Impact Analyses
Scale Score Validity
• Given plans to calculated baseline referenced SGPs in numerous states,
the Center put in place several checks to ensure validity of scale scores.
• A priori Checks:
• Reviewed results from states/organizations using baseline SGPs
• Ran sensitivity analyses to examine to what extent perturbed data impact
baseline SGP analyses.
• Examined equating and psychometric reports (e.g., person fit statistics) for
2021 to detect anomalous data.
• Post-hoc Checks:
• Compared results across states to see whether results conform to what is
showing up in other states.
• Inspected results within state looking for anomalous results which may
indicate issues related to scale stability.
New Hampshire Academic Impact Analyses
Baseline referenced SGPs
• Baseline referenced SGPs have been calculated in several states
as well as with interim assessments over the last decade.
• The Georgia Department of Education calculated baseline SGPs
and used them as part of their accountability system for 5 years
in the early 2010s prior to the assessment transition in 2015.
• Baseline SGPs demonstrated excellent stability for grade level assessments.
• Goal of using baseline SGPs was to avoid zero-sum game associated with norm-
referenced analyses each year.
• Renaissance Learning uses baseline matrices to calculate SGPs in real time.
• Quantile regression is equivariant to monotone transitions to scale so, in
general, robust to minor perturbations of scale.
New Hampshire Academic Impact Analyses
Baseline referenced SGPs
• We are not aware of any psychometric concerns regarding the
validity of scale scores reported by NHDOE.
• Baseline growth results for New Hampshire are, in general,
consistent with what has been observed in the dozen states we
have examined nationally.
• In the following slides we report the baseline referenced SGPs.
• We also show 2019 SGPs (baseline/skip-year) to illustrate academic
impact.
New Hampshire Academic Impact Analyses
SAS Attainment: 2019 to 2021 Percent Proficient
ELA Mathematics
Grade
% Proficient 2019 % Proficient Difference 2021 to % Proficient 2019 % Proficient 2021 Difference 2021 to
(Count/Participation 2021 (Count/ 2019 (Count/Participation (Count/Participation 2019
Rate) Participation Rate) Rate) Rate)
New Hampshire Academic Impact Analyses
Points of Clarification for Status Results
• Decrease in attainment (as a decrease in percent proficient) between
2019 and 2021 is the most often reported statistic related to state
testing in 2021.
• This analysis & statistic (percent proficient) are problematic.
• Using 2019 as a counterfactual for where students would have been
had the pandemic not occurred is problematic for several reasons
including:
• Changes in student cohort composition is not accounted for.
• Improvement between 2019 and 2021 that would have occurred is not
accounted for.
• Missing data in 2021 not accounted for.
New Hampshire Academic Impact Analyses
Points of Clarification for Status Results
• Drops in percent proficient are:
• Modest in ELA when compared to what has been observed in other states (high single digit
decreases) /mathematics low double-digit decreases).
• Typical in mathematics when compared to what has been observed in other states (low to
mid teen decreases)
• Based upon multiple imputation analyses, decreases observed in percent proficient
almost certainly underestimate the true decrease in percent proficient had all
students tested.
• Participation rates (derived from decreases in observed test scores between 2019
and 2021) vary by grade and state/organization but suggest participation between
75% and 85% (mostly toward the lower end)
• In general, current results based upon incomplete data likely represent a best-case
scenario compared to what would appear given fuller participation.
New Hampshire Academic Impact Analyses
SAS: 2019 to 2021 Median Baseline SGPs by Grade
ELA Mathematics
Grade
Median SGP 2019 Median SGP Difference 2021 to Median SGP 2019 Median SGP 2021 Difference 2021 to
3 NA NA NA NA NA NA
4 NA NA NA NA NA NA
New Hampshire Academic Impact Analyses
Discussion
• Impacts in mathematics exceeded those in ELA.
• Decreases in median SGP in mathematics were, in general, in the lower twenties.
• Decreases in median SGP in ELA were in the middle single digits.
• The results in ELA are some of the smallest decreases we’ve seen nationally.
• The results in mathematics were consistent with many of the larger impact
states.
• The fact that growth is uncorrelated with prior achievement implies missing
data impacts are much more muted with growth than with status.
• In other states we’ve classified academic impacts as:
• > -5: Modest to None
• -15 to -5: Moderate
• < -15: Large
New Hampshire Academic Impact Analyses
WIDA-ACCESS Results for 2020/2021
Median Composite Achievement Level Median Composite SGP Baseline Growth
GRADE
2020 2021 Difference 2020 2021 Difference
New Hampshire Academic Impact Analyses
Discussion
• Impacts in lower grades much more severe than those in middle
schools.
• Impacts in high school grades are negligible to non-existent.
• Same patterns seen in a dozen other WIDA states we have looked
at.
• Based upon a presentation by WIDA at NCSA/CCSSO in June,
drops are largely attributable to decreases in scores in speaking.
• Lack of impact in high school grades likely indicates that typical,
pre-pandemic growth represented impoverished rates of
learning.
New Hampshire Academic Impact Analyses
SAS: 2019 to 2021 Median SGPs by Ethnicity
ELA Mathematics
Ethnicity
Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021
2019 2021 to 2019 2019 2021 to 2019
American Indian/ 42.0 (115) 47.5 (78) -8.0 45.5 (112) 24.0 (79) -21.5
Alaskan Native
Asian 62.0 (1,394) 61.0 (1,065) -9.0 66.0 (1,419) 48.0 (1,068) -18.0
Black/African 44.0 (704) 39.0 (587) -4.0 45.0 (725) 25.0 (611) -20.0
American
Hispanic/Latino 48.0 (2,770) 38.0 (2,094) -8.0 47.0 (2,837) 22.0 (2,155) -25.0
White 50.0 (37,376) 46.0 (30,440) -7.0 50.0 (37,410) 29.0 (30,501) -21.0
Two or More 55.0 (1,375) 48.0 (1,143) -7.0 54.0 (1,387) 28.0 (1,148) -26.0
Races
New Hampshire Academic Impact Analyses
Discussion of Ethnicity Results
• Broken out by ethnicity, 2021 baseline SGP results show similar
rank ordering (i.e., growth gaps) as seen in normal years.
• Student academic growth is lower in mathematics than in ELA
for all ethnic subgroups.
• Growth impacts by ethnicity show some variability. For example,
Hispanic students more impacted than other groups --- though
all groups are impacted, especially in mathematics.
New Hampshire Academic Impact Analyses
SAS Growth: 2019 to 2021 Median SGPs by Disadvantaged
Status
ELA Mathematics
Free/Reduced
Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021
Lunch Status
2019 2021 to 2019 2019 2021 to 2019
No 52.0 (32,030) 49.0 (27,494) -3.0 53.0 (31,998) 31.0 (27,567) -22.0
Yes 45.0 (11,750) 35.0 (7,940) -10.0 43.0 (11,939( 23.0 (8,023) -20.0
New Hampshire Academic Impact Analyses
Discussion of disadvantaged status
• Baseline SGP results show similar growth gaps in 2021 (students
indicated as receiving free/reduced lunch have lower SGPs than
students indicated as not receiving free/reduced lunch.
• Academic impacts are for students receiving free/reduced lunch
in 2021 were larger than for those students not receiving.
• The result differs from other states where free/non-free reduced lunch
groups generally show (roughly) the same impact.
New Hampshire Academic Impact Analyses
SAS Growth: 2019 to 2021 Median SGPs by Gender
ELA Mathematics
Gender
Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021
2019 2021 to 2019 2019 2021 to 2019
Female 53.0 (21,400) 49.0 (17,255) -4.0 51.0 (21,459) 30.0 (17,310) -21.0
Male 46.0 (22,370) 42.0 (18,179) -4.0 48.0 (22,468) 29.0 (18,280) -19.0
New Hampshire Academic Impact Analyses
Discussion of Gender Results
• Growth gender gaps historically exist (favoring females) in both
ELA and mathematics.
• Growth gender gaps are historically larger in ELA than
mathematics.
• Gaps are similarly large in math and ELA for males and females.
New Hampshire Academic Impact Analyses
SAS Growth: 2019 to 2021 Median SGPs by IEP Status
ELA Mathematics
IEP Status
Median SGP Median SGP Difference 2021 Median SGP Median SGP Difference 2021
2019 2021 to 2019 2019 2021 to 2019
IEP No 52.0 (36,582) 49.0 (29,440) -3.0 52.0 (36,685) 30.0 (29,534) -22.0
IEP Yes 38.0 (7,198) 32.0 (5,994) -6.0 39.0 (7,252) 27.0 (6,056) -12.0
New Hampshire Academic Impact Analyses
Discussion of IEP Status Results
• IEP status gaps historically exist and favor non-IEP students in
both ELA and mathematics.
• Gaps in 2021 were larger for IEP students in ELA than non-IEP
students. The opposite was true in Mathematics.
• Potential explanation include:
• Concerted effort in mathematics to educate IEP students during the
pandemic thus minimizing the academic impact to that group of
students.
• Impoverished growth rates pre-pandemic such that pandemic impacts
didn’t substantially change rates of learning.
New Hampshire Academic Impact Analyses
Participation Rates and Interpretation
• An issue impacting the interpretation of growth (and status) results in
2021 is participation in the state assessment.
• Low participation doesn’t impact calculation of cohort referenced
growth norms nor use of baseline referenced growth norms.
• Low participation does impact comparisons of group level results with
historical (e.g., 2019) results based upon higher participation rates.
• For example, how should one compare growth and status results for
school in 2019 and 2021 with very different rates of participation.
New Hampshire Academic Impact Analyses
Multiple Imputation
• Given the rich historical data available for students missing in 2021,
and the observed performance of comparable students from 2019
to 2021, it is possible to simulate what we think their performance
might have been.
• Combining multiple simulations can inform us about potential bias
in what we observed at state/district/school levels in 2021, as well
as inform us about levels of uncertainty around them.
• Multiple imputation (MI) of missing data is widely adopted and
well-developed statistical analysis technique used in many fields to
deal with missing data problems.
New Hampshire Academic Impact Analyses
Multiple Imputation
• MI uses information from the observed data to generate
parameter estimates in three steps:
• Imputation: A prediction model generates a set of plausible values
for the missing observations, resulting in M imputed data sets.
• Analysis: The analysis (e.g., baseline student growth percentiles) is
conducted on each of the M data sets.
• Pooling: Parameter and (more importantly) uncertainty estimates
are constructed by pooling across the M analyses.
New Hampshire Academic Impact Analyses
Multiple Imputation
• In the context of COVID impact analyses, we use the multiple
(plausible) parameter estimates to inform us about the uncertainty
around what was observed. That is, we are more interested in using
MI to determine whether we trust the statistics calculated from the
observed (with missing) data than to produce “corrected” parameter
estimates.
• Statistics of particular interest:
• Mean/median scale scores
• Mean/median baseline SGPs
• Effect size estimates
• Percent Proficient
New Hampshire Academic Impact Analyses
Multiple Imputation
• Before performing a MI analysis with New Hampshire data, missing
data patterns were investigated.
• Patterns suggest some relationship to factors such economic status
and ethnicity.
• The relationship with prior achievement is relatively small.
• These relationships suggest that missingness in New Hampshire is
non-ignorable, but does not preclude the use of MI.
• Data can be considered “Missing-at-random” – relationships exist between
missingness and the observed data (and so can not be considered “missing
completely-at-random”) but the relationships are not systematic (e.g. all
students below proficiency in 2019 were missing in 2021).
New Hampshire Academic Impact Analyses
Multiple Imputation
• Given the 2021 missingness patterns and the good coverage of observed scores for
students from all demographic groups and prior achievement levels, MI is a
reasonable approach to dealing with SAS missing data.
• Imputation models included:
• Student prior achievement
• Student demographics
• FRL Status
• Disability status
• English language proficiency status
• Gender
• Ethnicity dummy variables for White, African American, Hispanic and Asian students
• School level clustering
• 30 imputations (M = 30 with a maximum of 10 iterations per imputation) were
conducted.
New Hampshire Academic Impact Analyses
Multiple Imputation
• Imputation models were restricted to include only students
who were enrolled, but not tested.
• Knowing school/district affiliation is critical for assessing the impact
of missing data for subsequent analyses at these levels.
• Excluding students who “disappeared” from the system may not
give us a full account of the impact of missing data at the state
level. However, a comparison of the enrolled vs. non-enrolled
missing students does not lead us to believe their inclusion would
meaningfully change inferences at the state level.
New Hampshire Academic Impact Analyses
Communication of Results in 2021 and beyond
• Technical challenges to the calculation of growth in 2021 are likely
minor compared to communication challenges.
• State reporting is, in general, highly aligned with accountability.
• Even though analyses outlined herein deal with status/growth, they
don’t necessarily align with accountability narratives.
• It is critical for leadership (at the highest level) to map out a
communication plan and then build analytics and reporting to support
that plan.
• Without a well considered, disciplined and coordinated communication
plan, accountability narratives are likely to take over.
New Hampshire Academic Impact Analyses
SGP Analysis Plans post 2021 (2022 and beyond)
• Spring 2021 represented an ideal time to get a snapshot of student attainment as
we are presumably/hopefully at or near the point of maximal impact on student
attainment.
• That is, recovery has not really begun for most students in earnest.
• It’s impossible to talk about recovery sensibly without understanding how big the
impact is.
• Analytics in 2022 will be keenly interested about whether recovery is occurring:
Who, what, how much.
• Monitoring recovery will be at least as critical as monitoring the impact of the
pandemic on student achievement.
• Beware the K-shaped recovery!
New Hampshire Academic Impact Analyses
Summary of Findings
• SGP analyses were successfully conducted for New Hampshire SAS/WIDA-
ACCESS testing in spring 2021
• For the SAS, results of the analyses suggest modest academic impacts in ELA
and large academic impacts in mathematics due to the pandemic.
• For WIDA-ACCESS, academic impact was large in elementary grades, moderate
in middle school grades, and no academic impact was indicated in the high
school grades (likely due to the fact that pre-pandemic growth was already
impoverished for high school students).
• Results suggest uniform impact by content area across elementary, middle and
high school.
• Recovery from impacts this large will likely take multiple years and require
additional supports beyond that provided by normal schooling.
New Hampshire Academic Impact Analyses
About this copy
New Hampshire Pandemic Related Academic Impact — New Hampshire Department of Education, November 7, 2021.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.