Hawaii SGP Growth Calculations in 2023 Operational and Investigative Considerations Damian Betebenner & Adam VanIwaarden — Center for Assessment Hawaii DOE Kickoff Meeting May 16th, 2023

SGP Analysis To Be Performed in 2023 • Following what was performed in 2022, in 2023, the same two types of SGP analyses will be conducted with SBA and WIDA- ACCESS data: • Cohort referenced analyses: These analyses create growth norms from the 2021, 2022, and 2023 students. We are back to two prior data point “post”-COVID. These growth norms yield cohort means/median SGPs of 50 and obscure COVID-19 academic impact. • Baseline referenced analyses: Historical growth norms were created using 2017, 2018 and 2019 data. These growth growth analyses don’t have mean/median SGPs of 50 and can be used to investigate overall COVID-19 academic impact and recovery.

SGP Analysis Plans for 2023 • Cohort referenced SGPs • Growth norms created from the current cohort of students. • Mean/median will by 50 for all grade x content area analyses • Cannot investigate overall impact & recover of the pandemic on students. • Can investigate differential impact & recovery of the pandemic on students. • Does not require scale stability across years. • Projections possible • Baseline referenced SGPs. • Growth norms created from historical (i.e., pre-COVID) cohorts of students. • Mean/median 50 for historical cohort, not necessarily 50 for current cohort. • Extent of deviation from (i.e. below) 50 is indicative of less learning/pandemic related academic impact. • Mean SGP deviation of 5 corresponds to an effect size of 0.18. • Can investigate BOTH overall impact & recovery as well as differential impact & recovery. • Requires scale stability across years. • Projections possible

Growth for Accountability in 2023

SGP for Accountability • Growth results for 2022 (baseline referenced) were promising indicating rates of learning in excess of what was observed in the state prior to the pandemic. • That is, students, on average, are recovering. • The hope is that growth (baseline referenced) in 2023 will be at least a good as 2022. • Hawaii will received both cohort and baseline reference SGPs. • These can be summarized in a number of ways: • Mean or Median SGP • Percent of students above a threshold or percent making or exceeding a year’s worth of learning.

Growth for Supplemental Reporting in 2023

SGP Analysis Plans post 2021 (2023 and beyond) • Spring 2021 represented (probably) an ideal time to get a snapshot of student attainment as we were presumably at or near the point of maximal impact on student attainment. • That is, recovery had not really begun for most students in earnest. • It’s impossible to talk about recovery sensibly without understanding how big the impact is. • Monitoring recovery will be at least as critical as monitoring the impact of the pandemic on student achievement. • Beware the K-shaped recovery!

SGP Analysis Plans for 2023 • Conduct analyses of recover in line with what was done in 2022. • Examine the extent to which recovery is continuing including how schools are performing.

Pandemic Related Academic Impact • As stated previously, any discussion of recovery implies some understanding of what the impact is/was. • As a basis for investigating academic impact/recovery, we undertook a comprehensive look at academic impact in Hawaii • SBA: Determined academic impact from 2019 to 2021 using grades 3, 4, 5, 6, 7, and 8 in ELA and mathematics. • WIDA-ACCESS: Determined academic impact from 2020 to 2021 using grades K to 12 ELP

How do we quantify academic impact using test data • The pandemic was like a “headwind” that (presumably) slowed down the learning of students due to a variety of factors. • We can use test data in two ways to quantify the impact of this headwind: • Best way: Quantify how much the rate of learning (i.e., growth) slowed: • SBA: Use historical skip-year growth norms to quantify growth and compare to pre-COVID growth. • ACCESS: Use historical consecutive-year growth norms and compare to pre-COVID growth • Next best way: Look at how much 2021 status changed: • SBA: Use 2019 status (or some adjustment thereof) as a proxy. • ACCESS: Use 2020 status (or some adjustment thereof) as a proxy. • Adjustments include (potentially) imputation for missing data

Status Based Recovery: Grade Mathematics

Status Based Impact: Grade 8 Mathematics

Growth Based Impact: Grade 8 Mathematics

Growth Based Impact

SBA Mathematics Grade 8 Hawaii 2021 to 2022 COVID⌧19 Academic Recovery All Students Growth: Grade 7 2021 to Grade 8 2022 Status: Grade 8 2021 to Grade 8 2022 Summary 2021 Grade 7 Scaled Score Decile Cuts 2021 Grade 8 Scaled Score Observed Academic Impact: Overall and by Quintile Grade 8 Scaled/Standardized Score and Achievement Level Cuts Overall 2022 Grade 8 Scaled Score Decile Cuts 2018 to 2019 Mean Conditional Status 90 1.5 2021 Percentiles (Observed) 2021 to 2022 Mean Conditional Status (Observed) 2022 Percentiles (Observed) 2021 to 2022 Growth Impact 2021 to 2022 Status Impact Stabilization 2021 and 2022 Mean Scaled Scores 70 Recovery Stabilization Stabilization Stabilization Stabilization 2022 Grade 8 Scaled Score Participation −0.5 Total Status Growth 2300 2300 2021: 24.7% Proficient 2022: 30.8% Proficient −2.5 Growth/Status Change Legend 2021 Grade 7 Scaled Score with Achievement Level Cuts Grade 8 Overall Percentile Ranks Deterioration Stabilization Recovery 2019 & 2022 Grade 8 Median SGPs by Quintile 2021 & 2022 Grade 8 Mean Scaled (Standardized) Scores by Quintile Q1 Q2 Q3 Q4 Q5 Overall Q1 Q2 Q3 Q4 Q5 Overall 2019 Grade 8 Students 50.0 51.0 50.0 50.0 49.0 50.0 2021 Grade 8 Students 2351 (−1.56) 2453 (−0.73) 2509 (−0.28) 2569 ( 0.21) 2674 ( 1.06) 2511 (0.00) 2022 Grade 8 Students 63.0 57.0 49.0 49.0 51.0 53.0 2022 Grade 8 Students 2353 (−1.54) 2460 (−0.67) 2524 (−0.16) 2589 ( 0.37) 2697 ( 1.25) 2524 (0.12) Calculations suppressed Academic Impact (Observed) 13.0 6.0 −1.0 −1.0 2.0 3.0 Change (Observed) 2 (0.02) 7 (0.06) 15 (0.12) 20 (0.16) 23 (0.19) 13 (0.12) Count Observed 2019 2,355 2,370 2,389 2,421 2,389 11,924 Count Observed 2021 2,131 2,153 2,151 2,155 2,127 10,717 due to small group N (< 50) Count Observed 2022 1,974 1,976 1,954 2,000 1,983 9,887 Count Observed 2022 2,496 2,489 2,484 2,478 2,495 12,442 2022 to 2021 Scaled/Standardized Score Difference 2022 to 2021 SGP Difference (Observed) 2022 to 2021 Grade 8 Status Change (Observed) 2022 to 2021 Conditional Status Difference (Observed) −0.87: Correlation SGP and Status Percentile Change Conditional Status/Growth Change 25 1.00: Correlation SGP and Conditional Status Difference 25 2021 to 2022 2021 Grade 7 Scaled Score with Decile & Achievement Level Cuts Grade 8 Percentile Ranks and Scaled Scores (2021 Norm)

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.55 −0.07 0.1 0.2 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Economically Disadvantaged Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Non−Economically Disadvantaged Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Special Education Stabilization Stabilization Stabilization Stabilization Stabilization Deterioration Recovery Recovery Stabilization Stabilization Stabilization Stabilization Asian Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Recovery Recovery Stabilization Hispanic Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Native Hawaiian Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Recovery Recovery Stabilization ELL Stabilization Recovery Stabilization Stabilization Stabilization Deterioration Recovery Recovery Recovery Stabilization Stabilization Deterioration Female Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization Male Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Recovery Recovery Stabilization

ELA And Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade, Content Area & Achievement Quintile Deterioration Recovery −30 −7 10 30 Growth Change NOTE: Quintiles for each student −0.55 −0.07 0.1 0.2 Status Change subgroup are based upon quintile cuts defined for All Students. Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) Box width indicates group proportion. ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students Economically Disadvantaged Non−Economically Disadvantaged Special Education Asian White Hispanic Native Hawaiian ELL Female Male

Questions

About this copy

Hawaii SGP Growth Calculations in 2023 — Hawaii Department of Education, May 16, 2023.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.