Growth Models: A Unified Theory Connecting student growth, pandemic recovery and long term policy goals Damian Betebenner Center for Assessment HIDOE Leadership Meeting March 21st, 2023

Academic Change & Academic Growth Concepts and Terminology

Academic Change Growth models in education are concerned with understanding the change in what a student knows and can do between two points in time. Learning Development Maturation Time 1 Time 2 Attainment Attainment Change in attainment between Time 1 and Time 2

Academic Change There are numerous ways in which academic change for an individual student can be characterized. • Content referenced change: Two lists indicating what a student can do at Time 1 and Time 2. • Scale referenced change: Given an assessment with a suitable vertical scale, the difference between attainment at Time 1 and Time 2 in scale score points. • Achievement level referenced change: Given an assessment with suitable achievement levels, the change in achievement level between Time 1 and Time 2

Academic Change & Academic Growth An academic growth model allows us to understand raw academic change. Growth Model A Growth Model is a “contextual wrapper” that helps us understand Academic the raw academic change Change that we observe.

Attainment & Growth Education isn’t the filling of a pail, but the lighting of a fire William Butler Yeats Attainment is indicative of how full the pail is. Growth is indicative of how bright the fire is.

Academic Change Growth models support different types of understanding • Description: Student Growth Percentiles are a descriptive growth model allowing users to describe individual and group level student growth in both a norm- and criterion- referenced fashion. • Explanation: Value-added models are explanatory models which (arguably) isolate the causes of the observed changed in student attainment. For example, a “teacher effect” or a “school effect”

Student Growth Percentiles Concepts and Terminology

Student Growth Percentiles (SGPs) • The most commonly used student academic growth measure in the United States. Used by approximately 25 states. • Used since 2010 in Hawaii • SGPs range from 1 to 99 where 1 is extremely low growth and 99 is extremely high growth. • An SGP of 50 is often used to indicate “a year’s worth of growth” or “a year’s worth of learning”.

Individual Student Growth Chart

Summarizing Growth for Groups of Students • Like attainment, individual student growth is often summarized for schools and demographic subgroups for accountability purposes. • Median SGP: Take the median of the SGPs • Mean SGP: Take the mean of the SGPs • Percent making a year or more worth of learning. • For each grade and content area, the median SGP is 50. • Groups with median SGPs above 50 are learning (on average) at rates exceeding the state median. • These summaries are often used for accountability purposes (school identification)

Hawaii School Performance SBA Growth & Prior Achievement 2022 SBA Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2021 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.15 Lower Achievement Lower Achievement Correlation (N ≥ 13): 0.15 Growth 2022 Median Student Growth Percentile

Hawaii School Performance SBA Growth & Prior Achievement 2022 SBA Reading Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2021 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Summary Statistics Correlation: 0.34 Lower Achievement Lower Achievement Correlation (N ≥ 13): 0.34 Growth 2022 Median Student Growth Percentile

COVID-19 Academic Impact & Recovery Academic Growth and Impact

COVID-19 Pandemic • The COVID-19 Pandemic led to the largest disruption of education in the history of the United States. • Three years into the pandemic the impacts are still being felt by students: academically, emotionally, physically, and financially and educators. • Student growth can help us understand both the academic impact and academic recovery due to the pandemic.

What is Academic Impact? • Academic Impact: Also referred to as learning loss or unfinished learning. • The pandemic and all its ensuing disruptions functioned as an “academic headwind”, impeding the academic progress (i.e., learning) of students. • Headwinds impede progress in two ways: ▪ They slow one’s rate of progress (speedometer) ▪ And by slowing one’s rate of progress they lead to less distance being travelled (odometer). • Academic impact decreases the rate of student learning (deceleration) which, in turn, leads to lower attainment.

What is Academic Recovery? • Academic Recovery: Also referred to as learning recovery • Academic recovery refers to increases in student learning sufficient to ameliorate previous academic impacts. • Academic recovery is analogous to a tailwind, increasing the rate of student learning (acceleration) which, in turn leads to higher attainment. • And to understand whether academic recovery is occurring, one must know the extent of academic impact.

Impact & Recovery: Educational Assessment • Hawaii’s Smarter Balanced and WIDA-ACCESS are ideally suited to help investigate academic impact and recovery. • With educational assessments, impact/recovery manifest as: • Decrease/increase in student growth • Decrease/increase in velocity = deceleration/acceleration • Decreases/increases in student growth lead to decreases/ increases in student attainment • Growth is to speedometer as attainment is to odometer.

Impact From Spring 2020 to Summer 2021 Smarter Balanced: Spring 2019 and Spring 2021 Assessments ACCESS: Winter 2020 and Winter 2021 Assessments

Impact Phase • During spring 2019-2020 and the 2020-2021 academic year, school was extremely disrupted due to pandemic related mitigation efforts. • Due to all the disruptions the expectation was that student learning would be impacted (i.e., would slow down) leading to lower student growth and lower student attainment. • Data from two sources was utilized to examine the extent to which the pandemic impacted student learning: • Smarter Balanced data from 2019 and 2021 • WIDA-ACCESS data from 2020 and 2021

Academic Impact •Using student growth, academic impact is categorized along a continuum •Academic impact is examined both overall and by academic quintile. •Academic impact was examined by each grade and content area tested for all students as well as for several demographic subgroups. •Non-participation in 2021 assessments was larger than normal. We utilized multiple imputation techniques to understand the extent to which missing data impacted observed results.

SBA ELA and Mathematics Hawaii 2019 to 2021 COVID⌧19 Academic Impact Grade & Content Area Deterioration Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large Economically Disadvantaged Moderate Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large Non−Economically Disadvantaged Moderate Modest to None Large Moderate Moderate Moderate Large Large Severe Severe Large Large Special Education Moderate Modest to None Moderate Modest to None Modest to None Modest to None Large Moderate Moderate Moderate Modest to None Modest to None Asian Moderate Modest to None Moderate Large Moderate Moderate Large Large Severe Severe Severe Severe White Moderate Moderate Large Large Moderate Moderate Large Moderate Severe Large Large Large Hispanic Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe Native Hawaiian Moderate Moderate Moderate Moderate Moderate Modest to None Severe Large Severe Large Moderate Large ELL Moderate Improvement Moderate Moderate Moderate Moderate Large Moderate Large Large Moderate Large Female Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Severe Large Large Male Large Moderate Moderate Moderate Moderate Moderate Severe Large Severe Large Moderate Large

WIDA ACCESS WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade & Content Area Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 All Students Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None WIDA⌧ACCESS Composite WIDA HI 2020 to 2021 COVID⌧19 Academic Impact Grade, Content Area & Achievement Quintile Impact Recovery −40 −25 −15 −5 5 20 Growth Change −0.55 −0.4 −0.25 −0.1 0.05 0.2 Status Change Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) WIDA−ACCESS Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile Quintile All Students

All Students Multi⌧State 2019 to 2021 COVID⌧19 Academic Impact ELA And Mathematics by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Hawaii Moderate Moderate Large Moderate Moderate Moderate Severe Large Severe Large Large Large State A Moderate Moderate Large Moderate Moderate Moderate Large Large Large Severe Large Severe State B Moderate Moderate Moderate Moderate Large Moderate State C Moderate Moderate Modest to None Moderate Modest to None Moderate Large Large Large Large Large Large State D Moderate Moderate Large Large Large Large Large Moderate Severe Severe Large Severe State E Large Moderate Large Moderate Large Moderate Severe Large Large Severe Large Large State F Moderate Moderate Large Large Large Large Severe Severe Severe Severe Large Severe State G Large Large Large Moderate Large Moderate Large Severe Large Large Moderate Moderate State H Moderate Moderate Moderate Moderate Moderate Moderate Moderate Large Large Large Large Large State I Moderate Large Moderate Moderate Modest to None Moderate Moderate Moderate Moderate Moderate Moderate Moderate

All Students Multi⌧State 2020 to 2021 COVID⌧19 Academic Impact WIDA⌧ACCESS Composite by Grade Growth/Status Change Legend Modest Calculations suppressed Severe Large Moderate to Improvement None due to small group N (< 50) Composite Grade 0 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 State A Modest to None Modest to None Large Large Large Severe Large Moderate Moderate Moderate Moderate Modest to None Modest to None State B Large Large Large Large Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None State C Modest to None Moderate Moderate Large Large Large Large Moderate Moderate Moderate Moderate Modest to None Modest to None State D Moderate Large Large Large Large Large Moderate Modest to None Modest to None Moderate Modest to None Modest to None Modest to None State E Large Large Moderate Large Large Large Large Moderate Moderate Modest to None Modest to None Modest to None Modest to None State F Moderate Moderate Moderate Moderate Moderate Moderate Moderate Modest to None Improvement Modest to None Improvement Improvement Modest to None State G Moderate Large Moderate Large Large Moderate Moderate Moderate Moderate Moderate Modest to None Modest to None Improvement State H Moderate Moderate Moderate Large Large Large Large Moderate Large Moderate Moderate Modest to None Modest to None

Recovery From Fall 2021 to Summer 2022 Smarter Balanced: Spring 2021 and Spring 2022 Assessments ACCESS: Winter 2021 and Winter 2022 Assessments

Academic Recovery •Using student growth, academic recovery is categorized along a continuum •Academic impact is examined both overall and by academic quintile. •Academic impact was examined by each grade and content area tested for all students as well as for several demographic subgroups.

SBA ELA and Mathematics Hawaii 2021 to 2022 COVID⌧19 Academic Recovery Grade & Content Area Deterioration Recovery −30 −7 10 30 Growth Change −0.38 −0.09 0.12 0.38 Status Change Calculations suppressed Deterioration Stabilization Recovery due to small group N (< 50) ELA Mathematics Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 All Students Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Economically Disadvantaged Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Non−Economically Disadvantaged Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Special Education Stabilization Stabilization Deterioration Deterioration Stabilization Deterioration Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Asian Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Recovery Stabilization White Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization Hispanic Stabilization Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Native Hawaiian Stabilization Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Stabilization Stabilization Stabilization Stabilization ELL Stabilization Stabilization Deterioration Stabilization Stabilization Deterioration Recovery Recovery Stabilization Stabilization Stabilization Deterioration Female Recovery Recovery Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization Male Recovery Stabilization Stabilization Stabilization Stabilization Stabilization Recovery Recovery Recovery Stabilization Stabilization Stabilization

Overall Recovery • Analysis of 2022 SBA and ACCESS data indicates a lot of trends in Hawaii to be proud of. • In both ELA and mathematics, learning acceleration was at least as much to be considered stabilization and in many cases could be considered as genuine recovery. • Compared to a dozen other states that the Center conducts these analyses, Hawaii comes out as the best in terms of pandemic recovery. • Results from 2023 data will be critical to assess whether these exemplary rates of learning (beyond what the state experience pre-COVID) are being maintained.

Now What Leveraging Assessment Data to Help Students Overcome COVID-19 Academic Impacts

Exemplar Schools: Impact & Recovery • Original discussions about examining ROI were refined to examine Return on Activities: • Identify outlier schools in terms of impact/recovery. • Investigate whether activities (or lack thereof) could potentially be responsible for the school’s outlier status. • For schools with plausible activities leading to exemplary recovery, investigate links to relevant investment/funding. • Step 1 is to find these outlier schools. • We investigated schools along two dimensions: • Academic Impact (2019 to 2021 decrease in student learning) • Academic Recovery (2021 to 2022 increase in student learning relative to pre- pandemic 2018-2019 learning)

Exemplar Schools: Impact & Recovery • We searched for exemplar schools to place into three categories • Large Impact/Large Recovery (9 schools) • Minimal Impact/Large Recovery (4 schools) • Large Impact/Poor Recovery (3 schools) • 16 schools were sampled across a diverse range of socio-economic and demographic characteristics with the goal of finding replicable strategies that can be used by Hawaii schools to help students recover.

Exemplar Schools: Impact & Recovery • Some initial findings: • Distributive leadership works well in “normal” situations, authoritative leadership is necessary during a crisis. • Everyone was in crisis mode from March 2020-August 2020 • Strong leadership adapted existing / implemented new programs quickly and effectively • Funding was not a benefit or limitation for recovery • Strong leadership went into the community, school funds, foundations – found the money when needed or paid out of their own pocket for food distribution, providing masks, providing other health services • Strong leadership focused on the whole school community including parents and staff • Data use for identifying struggling students was flexible

Questions

About this copy

Growth Models: A Unified Theory — Hawaii Department of Education, March 22, 2023.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.