Hawaii Growth Model: Changing Conversations about Student Performance

Hawaii REL · September 2016

Hawaii Growth Model: Changing Conversations about Student Performance — Hawaii REL, September 2016.
Published

September 26, 2016

Event
Invited talk
Slides
28
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Hawaii Growth Model: Changing Conversations about Student Performance Damian W Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire September 15th, 2016
  2. Student Growth: Introduction • The existence of annual student testing together with unique student identifiers forms the foundation for the calculation of student growth. • Over the last decade, most states have begun calculation of longitudinal student growth using large scale state assessments. • The analysis of student growth is controversial because of the uses to which it is put: Teacher & School Evaluation. • Development of the Hawaii Growth Model involved the creation of a diagnostically useful quantity that could be used (or not used) at a later time for many purposes.
  3. Why Student Growth? If you care about student achievement, then you care about student growth. • Where you are is a function of where you started and how quickly you’ve travelled since. • Student growth and student attainment/achievement are related in this way. • Distance=Rate x Time • Those who start behind tend to stay behind and those that start ahead tend to stay ahead.
  4. Student Growth: Introductory Concepts • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.
  5. Student Growth Introductory Concepts A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.
  6. Student Growth: Introductory Concepts • There are 3 general approaches of calculating student growth: – Growth as an amount of change (i.e., gain). – Growth as a norm/peer referenced quantity. – Growth as a criterion referenced quantity.
  7. Student Growth: Introductory Concepts • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which Hawaii has since 2015/SBAC) – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions – Interpreting change requires context.
  8. Student Growth: Introductory Concepts Back to the story problem!
  9. Student Growth: Introductory Concepts • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.
  10. Student Growth: Introductory Concepts • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2015 to 2016 Top 10% Reading Grade 4 2015 +26.5 Bottom 10% Reading Grade 4 2015 +65.5 Top 10% Mathematic Grade 4 2015 +28.9 Bottom 10% Mathematics Grade 4 2015 +69.0 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point.
  11. Student Growth: Introductory Concepts •This is analogous to height/weight measures for infants. •Growth amounts are tricky to interpret and require context. •Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm- referenced quantity locating that 4 inch increase alongside the height increases of similar children.
  12. Student Growth: Introductory Concepts • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/ student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • The Hawaii SGP Model (i.e., the Colorado Growth Model) creates growth norms for all students using regression (i.e., peer-based) analyses. • These norms allow stakeholders to understand what an ambitious yet reasonable expectation is for each student.
  13. Hawaii Growth Model (Student Growth Percentiles) • Research on the SGP model began in 2007. • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. • Goal or the model is to create an individual level growth metric that is informative to different stakeholders ranging from parents to policy makers.
  14. Hawaii Growth Model (Student Growth Percentiles) • Different growth models answer different questions. • Different stakeholder groups ask different questions. • Parents are most concerned about how their child is doing. • To effectively engage parents and communities in a productive conversation about student learning (i.e. growth), a growth model should describe: • What is? How much are students learning/growing. • What should be? How much would we like students to grow/learn. • What is reasonable? How much is it reasonable to expect students to grow/learn. • Most other growth models do not provide a student level metric. They provide a school summary used for accountability purposes. 14
  15. Hawaii Growth Model • Growth is achievement over time Exceeded Exceeds Met High Meets ● ● Typical Nearly Met Approaches Low Not Met Well Below HSA HSA SBA SBA SBA Math Math Math Math Math Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Scale Score 311 338 2553 2549 Achievement Level Meets Meets Met Nearly Met Achievement Growth Percentile 87 40 28 Growth Level High Typical Low Growth Source: Hawaii Department of Education/NCIEA
  16. Hawaii Growth Model • Growth is achievement over time Exceeded Exceeds Met ● High Meets ● Nearly Met Typical Approaches Low Not Met Well Below HSA HSA SBA SBA SBA ELA ELA ELA ELA ELA Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Scale Score 313 325 2499 2442 Achievement Level Meets Meets Nearly Met Not Met Achievement Growth Percentile 91 33 6 Growth Level High Low Low Growth Source: Hawaii Department of Education/NCIEA
  17. Growth Model: Other Options • Almost all current growth model use large amounts of data to create “norms” against which student growth is judged. • Absent large amounts of data (>1,000 grade/content area), professional judgment replaces large-scale data-based norms. • Value-tables can be used to monitor progress of students between achievement bands established for year-to-year assessments. • Some states (e.g., Alaska) use value-tables, particularly with smaller scale assessments for evaluating student growth for accountability purposes.
  18. Accuracy & Precision • There was a recent report released questioning whether SGPs should be reported at all. • The report questioned whether SGPs had enough precision to be useful for stakeholders. • Similar to validity, precision (and accuracy) should always be considered in the context of use: • “Precise enough for what purpose”. • “Accurate enough for what purpose.”
  19. Accuracy & Precision
  20. Accuracy & Precision • Infrared thermometers are precise enough for the given purpose. • Educational assessment is like taking a child’s temperature: • You wouldn’t make a high stakes medical decision based solely on a child’s temperature. • Doctors use multiple pieces of evidence to diagnose and ultimately treat. • A child’s temperature is only useful up to a point. You could measure a child’s temperature to the .0001 degree. But what’s the point???? • High reliability/precision in educational assessment is often used (incorrectly) as a justification for a high stakes decision.
  21. Conclusions & Recommendations • Subtraction isn’t a growth model. A growth model requires one consider where a student begins. • Norms require large numbers of students. Smaller numbers of students requires professional judgment to decide what is ambitious yet reasonable progress for a student. • Precision and accuracy of results must be balanced with utility: Precise and accurate enough for what purpose. • Technical characteristics of tests and growth measures derived from them a often misunderstood (even by testing experts). • Practical and technical issues become even more apparent when (high) stakes are associated with outcomes.
  23. SGP: Academic Peers 250 0 0 co 15 0 2 5 S re 260 0 le S ca le 0 ca core 0 230 20 Source: Hawaii Department of Education/NCIEA
  24. SGP: Academic Peers 250 0 0 co 15 0 2 5 S re 260 0 le S ca le 0 ca core 0 230 20 Source: Hawaii Department of Education/NCIEA
  25. SGP: Academic Peers 250 0 0 co 15 0 2 5 S re 260 0 le S ca le 0 ca core 0 230 20 Source: Hawaii Department of Education/NCIEA
  26. SGP: Academic Peers th 0 Per ce nt i 240 leG th 0 row 250 0 0 co 15 0 2 5 S re S le ca le 0 Sca S re 20 co Source: Hawaii Department of Education/NCIEA
  27. SGP: Academic Peers th Per 230 ce nt i le rowth 2 250 0 0 co 15 0 2 5 S re S le ca le 0 S ca S re 20 co Source: Hawaii Department of Education/NCIEA
  28. SGP: Academic Peers Pe 2 de m cor Ac cale 75% 75 th Per 230 ce nt i le rowth 2 250 0 0 co 15 0 2 5 S re S le ca le 0 S ca S re 20 co Source: Hawaii Department of Education/NCIEA