Hawaii SGP Calculations in 2021
Background and Results
Damian Betebenner & Adam VanIwaarden
Center for Assessment
Hawaii Department of Education
September 2nd, 2021
Hawaii Testing in 2021
• Hawaii administered both SBA and WIDA-ACCESS in spring 2021.
• Both assessments were full form, in person administrations.
• SBA administered in Grades 3 to 8 and 11 in ELA and Mathematics. NOT
administered in spring 2020.
• WIDA-ACCESS administered in K to 12. It was administered in spring
Hawaii SGP Calculations 2021
SBA SGP Operational Considerations in 2021
• Due to no SBA testing in Spring 2020, SBA growth analyses will be skip-
year/gap year growth from 2019 to 2021.
• Two types of SGP analyses were conducted for SBA data:
• Cohort referenced analyses: These analyses create growth norms from the
2019 to 2021 students. These growth norms yield cohort means/median
SGPs of 50 and obscure COVID-19 academic impact.
• Baseline referenced analyses: Historical growth norms were created using
2019 and prior data. These growth growth analyses don’t have mean/median
SGPs of 50 and can be used to investigate overall COVID-19 academic impact
• Projections (lagged and straight) were calculated for 2021 data using
baseline growth norms in case they were needed.
Hawaii SGP Calculations 2021
WIDA-ACCESS SGP Operational Considerations in 2021
• Two types of SGP analyses were conducted for WIDA-ACCESS data:
• Cohort referenced analyses: These analyses create growth norms from the
2020 to 2021 students. These growth norms yield cohort mean/median SGPs
of 50 and obscure COVID-19 academic impact.
• Baseline referenced analyses: Historical growth norms were created using
2020 and prior data. These growth growth analyses don’t have mean/median
SGPs of 50 and can be used to investigate overall COVID-19 academic impact.
• Use composite score from WIDA-ACCESS data. No analyses performed
using sub-scores.
• Projections (lagged and straight) were calculated for 2021 data using
both cohort and baseline growth norms in case they were needed.
Hawaii SGP Calculations 2021
SGP Summary of Cohort versus Baseline Referenced
• Cohort referenced SGPs
• Growth norms created from the current cohort of students.
• Mean/median will by 50 for all grade x content area analyses
• Cannot investigate overall impact of the pandemic on students.
• Can investigate differential impact of the pandemic on students.
• Does not require scale stability across years.
• Baseline referenced SGPs.
• Growth norms created from historical (i.e., pre-COVID) cohorts of students.
• Mean/median 50 for historical cohort, not necessarily 50 for current cohort.
• Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related
academic impact.
• Mean SGP deviation of 5 corresponds to an effect size of 0.18.
• Can investigate BOTH overall impact and differential impact
• Requires scale stability across years.
Hawaii SGP Calculations 2021
SGP Assumptions
• SGP analyses presume valid scale scores
• Students forming the growth norm group are presumed to have taken the same (or
equivalent) test at the same time. Scores have to mean the same thing.
• SGPs, however, are often used as a post-hoc check of scale score validity/
comparability via Reductio ad absurdum
• In 2021, check form comparability and mode-of-administration comparability.
• RIDE has extensive experience (e.g., PARCC paper/pencil versus online) investigating
and accommodating mode-effects.
• If scores are not comparable then we either attempt to make them comparable or
run separate analyses.
• We assume something close to population invariance with respect to IRT
scaling.
• The population of students tested in 2021 and differences to 2019 is a concern
so we begin with data on the population of students tested.
Hawaii SGP Calculations 2021
SBA Participation
ELA Mathematics
Grade
Hawaii SGP Calculations 2021
WIDA-ACCESS Participation (Not sure I have all data for
this)
Overall
Overall Grade
Grade 2021
Hawaii SGP Calculations 2021
SGP Calculations
• SGP calculations were performed using the SGP package and R
software environment based upon data supplied by the Hawaii
Department of Education.
• To facilitate the calculation of baseline SGPs across multiple
states, we created an additional R package, SGPmatrices, that
contains Hawaii matrices, amongst many others.
• https://centerforassessment.github.io/SGPmatrices/
• Source code associated with calculations are available on GitHub:
• SBA: https://github.com/CenterForAssessment/Hawaii
• WIDA-ACCESS: https://github.com/CenterForAssessment/WIDA_HI
Hawaii SGP Calculations 2021
SGP Results
• The following slides show SGP results for 2021.
• We report only baseline referenced SGPs in these slides
• Differential impact shown in baseline SGPs will mirror that found in
cohort referenced SGPs.
• Due to the highly skewed nature of the SGP results, we recommend
median SGP over mean SGP in this instance.
• We also provide 2019 results to understand pandemic related
academic impact.
Hawaii SGP Calculations 2021
SGP Analyses and Academic Impact
• Understanding pandemic related impact necessarily requires looking
back to pre-pandemic results.
• The impulse of stakeholders to want to compare 2021 results with 2019
results is the correct impulse at both the individual, school, district, and
state level.
• Wanting to look at 2019 to 2021 change is an attempt to investigate
impact.
• The challenge is making 2019 to 2021 comparisons apples-to-apples
comparisons.
• Supporting these comparisons requires having a complete
understanding is who constitutes the missing data. More on that after
some initial comparisons.
Hawaii SGP Calculations 2021
SGP Analyses for 2021
• With comparable results we can quantify pandemic related
academic impact:
• At the state level
• Demographic subgroups including ethnicity, poverty
• Academic subgroups (special education, low/high achievers)
• By region/district/school
• By mode of education (remote/hybrid/in-person).
• Attendance
• English Language Learner impact can be evaluated using WIDA-ACCESS
results.
Hawaii SGP Calculations 2021
SBA Percent Proficient by Grade
ELA Mathematics
Grade
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Grade
ELA Mathematics
Grade
3 NA NA NA NA
4 NA NA NA NA
Hawaii SGP Calculations 2021
Discussion
• Median baseline SGPs in ELA were in the mid 30s/lower 40s and
median baseline SGPs in mathematics were in the 20s (grade 11
median SGP of 38).
• Deviation from normal growth (i.e., 50) represents the extent to
which academic growth slowed down from 2019 to 2021, that is,
academic impact.
• Larger academic impact in ELA than mathematics.
• Academic impact is fairly uniform across grades.
• Academic impact is very large likely requiring multiple years and
additional educational supports/interventions to recover.
Hawaii SGP Calculations 2021
WIDA-ACCESS Median SGPs by Grade
Overall Overall
Grade Grade
K NA NA 6 52 (1,268) 38 (1,008)
Hawaii SGP Calculations 2021
Discussion of Grade Level Results
• Median baseline SGPs are low in elementary school grades and
are close to 50 in the high school grades.
• Deviation from normal growth (i.e., 50) represents the extent to
which academic growth slowed down from 2020 to 2021, that is,
academic impact.
• Academic impact is very large in elementary and middle school
and close to non-existent in high school.
• Lack of impact is consistent with:
• Great academic support for the group during the pandemic.
• Impoverished learning conditions prior to the pandemic such that the pandemic
didn’t alter the status quo.
Hawaii SGP Calculations 2021
SBA Median/Mean Skip-Year SGPs by Ethnicity
ELA Mathematics
Ethnicity
Am. In./AL Native
Asian
African American
Hispanic or Lat.
Multiple Ethnicities
Native HI/PI
No Race reported
White
Hawaii SGP Calculations 2021
Discussion of Ethnicity Results
Hawaii SGP Calculations 2021
SBA Median/Mean Skip-Year SGPs by Free/Reduced Lunch
Status
ELA Mathematics
Free/Reduced Lunch
Status
FRL Yes
FRL No
Hawaii SGP Calculations 2021
Discussion of Free/Reduced Lunch Results
Hawaii SGP Calculations 2021
SBA Median/Mean Skip-Year SGPs by IEP Status
ELA Mathematics
IEP Status
Non-IEP
IEP
Students with 504
Hawaii SGP Calculations 2021
Discussion of IEP Results
Hawaii SGP Calculations 2021
SBA Median/Mean Skip-Year SGPs by ELL Status
ELA Mathematics
ELL Status
ELL No
ELL Yes
Hawaii SGP Calculations 2021
Discussion of ELL Results
Hawaii SGP Calculations 2021
SBA Median/Mean Skip-Year SGPs by Gender
ELA Mathematics
Gender
Female
Male
Hawaii SGP Calculations 2021
Discussion of Gender Results
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade 5
& 6 Mathematics
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade 7
& 8 Mathematics
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade
11 Mathematics
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade 5
& 6 ELA
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade 7
& 8 ELA
Hawaii SGP Calculations 2021
SBA Median Skip-Year SGPs by Prior Achievement: Grade
11 ELA
Hawaii SGP Calculations 2021
Discussion of Growth by Prior Achievement Results
• Except in Grades 5 and 6 ELA, a fairly consistent pattern is that
higher achieving students demonstrate lower growth (have
greater academic impact) than lower achieving students.
• We seen this results in several states, though there are some
where the impact is fairly uniform across prior achievement
level.
Hawaii SGP Calculations 2021
Participation Rates and Interpretation
• An issue impacting the interpretation of growth (and status) results in
2021 is participation in the state assessment.
• Low participation doesn’t impact calculation of cohort referenced
growth norms nor use of baseline referenced growth norms.
• Low participation does impact comparisons of group level results with
historical (e.g., 2019) results based upon higher participation rates.
• For example, how should one compare growth and status results for
school in 2019 and 2021 with very different rates of participation.
Hawaii SGP Calculations 2021
Multiple Imputation
• Given the rich historical data available for students missing in 2021,
and the observed performance of comparable students from 2019
to 2021, it is possible to simulate what we think their performance
might have been.
• Combining multiple simulations can inform us about potential bias
in what we observed at state/district/school levels in 2021, as well
as inform us about levels of uncertainty around them.
• Multiple imputation (MI) of missing data is widely adopted and
well-developed statistical analysis technique used in many fields to
deal with missing data problems.
Hawaii SGP Calculations 2021
Multiple Imputation
• MI uses information from the observed data to generate
parameter estimates in three steps:
• Imputation: A prediction model generates a set of plausible values
for the missing observations, resulting in M imputed data sets.
• Analysis: The analysis (e.g., baseline student growth percentiles) is
conducted on each of the M data sets.
• Pooling: Parameter and (more importantly) uncertainty estimates
are constructed by pooling across the M analyses.
Hawaii SGP Calculations 2021
Multiple Imputation
• In the context of COVID impact analyses, we use the multiple
(plausible) parameter estimates to inform us about the
uncertainty around what was observed. That is, we do not use
MI to produce “corrected” parameter estimates, but rather to
help determine whether we trust the statistics calculated from
the observed (with missing) data.
• Statistics of particular interest:
• Mean/median scale scores
• Mean/median baseline SGPs
• Effect size estimates
Hawaii SGP Calculations 2021
Multiple Imputation
• Before performing a MI analysis with SBA data, missing data
patterns were investigated.
• Patterns suggest some relationship to factors such as prior
achievement and economic status.
• These relationships suggest that missingness in SBA is non-
ignorable, but does not preclude the use of MI.
• Data can be considered “Missing-at-random” – relationships exist between
missingness and the observed data (and so can not be considered “missing
completely-at-random”) but the relationships are not systematic (e.g. all
students below proficiency in 2019 were missing in 2021).
Hawaii SGP Calculations 2021
SGP Analysis Plans post 2021 (2022 and beyond)
• Spring 2021 represents an ideal time to get a snapshot of student
attainment as we are presumably at or near the point of maximal
impact on student attainment.
• That is, recovery has not really begun for most students in
earnest.
• It’s impossible to talk about recovery sensibly without
understanding how big the impact is.
• Monitoring recovery will be at least as critical as monitoring the impact of
the pandemic on student achievement.
• Beware the K-shaped recovery!
Hawaii SGP Calculations 2021
Communication of Results in 2021
• Technical challenges to the calculation of growth in 2021 are likely
minor compared to communication challenges.
• State reporting is, in general, highly aligned with accountability.
• Even though analyses outlined herein deal with status/growth, they
don’t necessarily align with accountability narratives.
• It is critical for leadership (at the highest level) to map out a
communication plan and then build analytics and reporting to support
that plan.
• Without a well considered, disciplined and coordinated communication
plan, accountability narratives are likely to take over.
Hawaii SGP Calculations 2021
About this copy
Hawaii SGP Calculations in 2021 — Hawaii Department of Education, September 2, 2021.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.