Hawaii Student Growth Analyses: History and Conceptual Overview

Hawaii Department of Education · August 22, 2024

Hawaii Student Growth Analyses: History and Conceptual Overview — Hawaii Department of Education, August 22, 2024.
Published

August 22, 2024

Event
State agency
Slides
30
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Hawaii Student Growth Analyses History and Conceptual Overview Damian W Betebenner & Adam VanIwaarden National Center for the Improvement of Educational Assessment Honolulu, Hawaii August 22nd, 2024
  2. Topics for Discussion • About Me (Us) • What is student growth? A quick overview. • A brief history of student growth analyses in Hawaii. • SGP analyses associated with the 2023-2024 academic year.
  3. About us • Damian Betebenner • Joined Center for Assessment in 2007 • Co-Developed SGPs and the SGP package with Adam VanIwaarden • Began working with Hawai’i Department of Education on growth in 2012 • Currently working with about 20 states/organizations on student growth. • Lives in Needham, MA (just outside of Boston) • Adam VanIwaarden • Intern at Center for Assessment in 2010 • Working full time at Center as part of SGP development and implementation • Lives in Pueblo, CO
  4. Student Growth: Introductory Concepts • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: • The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. • Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.
  5. Student Growth Introductory Concepts A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.
  6. Student Growth: Introductory Concepts • There are 3 general approaches of calculating student growth: • Growth as an amount of change (i.e., gain). • Growth as a norm/peer referenced quantity. • Growth as a criterion referenced quantity.
  7. Student Growth: Introductory Concepts • Growth as an amount of change: • What could be more simple? • Like height or weight just calculate the amount of change. • Requires a vertical scale (which Hawaii has with SBA) • Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: • Comparisons between students often leads to wrong conclusions • Interpreting change requires context.
  8. Student Growth: Introductory Concepts Back to the story problem!
  9. Student Growth: Introductory Concepts • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.
  10. Student Growth: Introductory Concepts • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2023 to 2024 Top 10% ELA Grade 4 2024 +8.76 Bottom 10% ELA Grade 4 2024 +31.5 Top 10% Mathematic Grade 4 2024 +16.7 Bottom 10% Mathematics Grade 4 2024 +44.5 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point. • The point here is that many people who know something about psychometrics will say that “uneven gains” is due to education measurement scales not having strict measurement properties. • That’s a red herring • The example story problem shows that even with an interval scale, gains can be uneven. • In fact, why would we expect gains to be even irregardless of initial starting point.
  11. Student Growth: Introductory Concepts • This is analogous to height/weight measures for infants. • Growth (i.e., change) amounts are tricky to interpret and require context. • Peer based comparisons used by growth analyses are a way to acquire that context. • Regression is the statistical technique to mathematically model these peer based comparisons. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm-referenced quantity locating that 4 inch increase alongside the height increases of similar children.
  12. Student Growth: Introductory Concepts • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • The Hawaii SGP Model creates growth norms for all students using regression (i.e., peer-based) analyses. • These norms allow stakeholders to understand what an ambitious yet reasonable expectation is for each student.
  13. Student Growth Percentiles • Research on the SGP model began in 2007. • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. • Goal or the model is to create an individual level growth metric that is informative to different stakeholders ranging from parents to policy makers.
  14. History of student growth in Hawaii • Begun in 2012 as part of Hawaii’s successful RttT grant. • Originally focused on teacher evaluation. • Included use of interactive visualizations to help communicate SGP results. • Included the calculation of student growth percentiles (SGPs) and student growth targets (SGTs) (not sure whether SGTs were used). • Used (and not used) in different ways over the last decade. • Used to investigate COVID-19 related academic impact and recovery.
  15. Calculation of Student growth percentiles • Student Growth Percentiles and Projections are calculated using the SGP package (2019) which utilized the R statistical software environment. • Both the SGP package and R are open source (free!) software. • All source code for the SGP package as well as the Hawaii SGP analyses are open source and posted on the Center for Assessment Github site: • https://github.com/CenterForAssessment • https://github.com/CenterForAssessment/SGP • https://github.com/CenterForAssessment/Hawaii • Details about the R software environment and SGP package are available in an accompanying presentation tomorrow.
  16. Calculation of Student growth percentiles • Calculation of SGPs is conducted in two broad steps: • Data Preparation: • Raw student level assessment data and raw student-teacher roster data are provided by Hawaii DOE. These data are modestly cleaned and prepared for SGP analysis. • Data Analysis: • One data is prepared, the SGP package and its functions are used to conduct all SGP analyses
  17. Calculation of Student growth percentiles • Data preparation: • Student level data preparation: • The R script used to create the final student level data for SGP analysis is available on the Hawaii SGP Github website: https://github.com/CenterForAssessment/ Hawaii/blob/master/Hawaii_Data_LONG_2024.R
  18. Calculation of Student growth percentiles • Data analysis: • The R script used to run the entire SGP is available on the Hawaii SGP Github website: https://github.com/CenterForAssessment/Hawaii/blob/master/ Hawaii_SGP_2024.R • The SGP R script includes a number of internal steps creating SGPs, SGTs (student growth targets, sometimes referred to as adequate growth percentiles), school level summaries, and outputting the results. • Previously, individual student reports were generated for the Hawaii DOE as well using the SGP package (see accompanying sample reports).
  19. Calculation of Student growth percentiles • Data validation and transfer • SGP analyses produce numerous goodness-of-fit plots used to investigate whether SGP analyses produce valid results. • Output data are shared with Hawaii DOE via the Center for Assessment’s secure ShareFile website.
  20. SGP Output • SGP analyses provide a plethora of data that can help the HIDOE better understand how the education system is functioning. • Often, the data derived is fed primarily into the state accountability system. • In the following slides we review the 2024 SGP results as well as investigate other output derived from the SGP package.
  21. SGP Output • An SGP is a student level result that can be aggregated across any groups of interest. • The goal was to help everyone see that the “performance of the system” is based upon the performance of individual students. And if we want to make the system perform better then we need to help students perform better and this is what better performance means (higher rates of learning) • As I mentioned, early on the interest was looking at classroom (i.e., instructor) level growth. • This summary aggregation is akin to value-added metrics (I would argue that value-added in an inference as opposed to a statistic itself). • The goals of SGP were always to be first and foremost a descriptive measurement student growth learning. • As part of that communication effort, individual student reports were produced for all students in the state 5 to 10 years ago.
  23. SGP Output • Those of you who have heard me talk about COVID-19 impact analyses have heard me discuss at length the difference between examining status (e.g., percent proficient changes over time) versus looking at growth (e.g., median baseline SGP) over time. • The pandemic’s impact on student achievement taught me some invaluable lessons regarding how results are understood and communicated, especially to non-technical stakeholders. • In particular: • Status/Attainment is akin to a point in time measurement of what students know and can do. • Growth/Learning is taking to velocity - the change in status/attainment over time. • The pandemic disrupted the conditions of learning which in turn slowed learning down which led to lower attainment. • When we examine growth results think of this as the (average) velocity of learning.
  24. 2024 SGP Summary Results (Median SGP) ELA Mathematics SGP SGP Baseline SGP SGP Baseline Grade 4 50 (11,733) 52 (11,733) 50 (11,784) 55 (11,784) Grade 5 50 (12,065) 48 (12,065) 50 (12,096) 47 (12,096) Grade 6 50 (11,409) 50 (11,409) 51 (11,464) 51 (11,464) Grade 7 50 (11,125) 48 (11,125) 50 (11,208) 51 (11,208) Grade 8 50 (11,187) 47 (11,187) 50 (11,272) 47 (11,272) Grade 11 50 (7,701) 46 (7,701) 50 (7,741) 57 (7,741)
  25. 2023 SGP Summary Results • Mean/Median/Count for baseline referenced SGPs in ELA and mathematics are as follows: ELA Mathematics Grade
  26. SGP & Policy • Over the last couple of years as states reset long-term trends due to the impact of the COVID-19 pandemic I’ve seen too many bad interpretations of trend data. • The implicit goal behind reporting percent proficient trend data is to hopefully seeing increasing rates of proficiency over time (we want more kids to be proficient in ELA and math). • Increases, if they exist, are often modest. • Though percent proficient is critically important, I would argue that it is an impoverished and incoherent way to motivate the changes that we actually believe we have some control over. • Achievement/attainment are an outcome of growth/learning. It is the latter that we have some control over, yet we focus the attention of educators on the former.
  27. Hawaii: 2024 Reading Hawaii: 2024 Reading Norm & Criterion Referenced Growth & Achievement Norm & Criterion Referenced Growth & Achievement No Grade 10 Assessment No Grade 10 Assessment No Grade 9 Assessment No Grade 9 Assessment Percentile Growth Trajectory Percentile Growth Trajectory Scale Score Scale Score 2024 Achievement Percentile 2024 Achievement Percentile SGP SGP trajectories for aa 3rd trajectories for from the starting from student starting grade student 3rd grade the Level Level 1/Level 1/Level 22 cut. cut. SGP trajectories SGP student starting grade student 3rd grade for aa 3rd trajectories for the Level from the starting from Level 2/Level 2/Level 33 cut. cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11
  28. Hawaii: 2024 Mathematics Hawaii: 2024 Mathematics Norm & Criterion Referenced Growth & Achievement Norm & Criterion Referenced Growth & Achievement No Grade 10 Assessment No Grade 9 Assessment No Grade 9 Assessment No Grade 10 Assessment Percentile Growth Trajectory Percentile Growth Trajectory Scale Score Scale Score 2024 Achievement Percentile 2024 Achievement Percentile SGP trajectories for SGP trajectories for aa 3rd grade student 3rd grade student starting from the starting from the Level Level 1/Level 1/Level 22 cut. cut. SGP trajectories SGP trajectories for for aa 3rd 3rd grade grade student student starting starting from from the the Level Level 2/Level 2/Level 33 cut. cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11
  29. SGP & Policy “Success is a journey, not a destination. The doing is often more important than the outcome.” Arthur Ashe • Journey versus Destination, Process versus Product, Input versus Outcome, … • Accountability often focuses on the latter whereas it is the former that we have control over.