Talks archive, 2015–2019 – Hawaii Growth Model: Changing Conversations about Student Performance

Hawaii Growth Model: Changing Conversations about Student Performance Damian W Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire September 15th, 2016

Student Growth: Introduction • The existence of annual student testing together with unique student identifiers forms the foundation for the calculation of student growth. • Over the last decade, most states have begun calculation of longitudinal student growth using large scale state assessments. • The analysis of student growth is controversial because of the uses to which it is put: Teacher & School Evaluation. • Development of the Hawaii Growth Model involved the creation of a diagnostically useful quantity that could be used (or not used) at a later time for many purposes.

Why Student Growth? If you care about student achievement, then you care about student growth. • Where you are is a function of where you started and how quickly you’ve travelled since. • Student growth and student attainment/achievement are related in this way. • Distance=Rate x Time • Those who start behind tend to stay behind and those that start ahead tend to stay ahead.

Student Growth: Introductory Concepts • The concept of student growth isn’t as difficult to understand and popular perception suggests. • Calculations may be complicated but knowing how to calculate student growth is not synonymous with understanding student growth: – The number π (pi=3.14159…) is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.

Student Growth Introductory Concepts A Math Story Problem • Whose high jump changed more? Explain your answer. • Whose high jump change is more remarkable? Explain your answer. • Which high jumper improved more? Explain your answer.

Student Growth: Introductory Concepts • There are 3 general approaches of calculating student growth: – Growth as an amount of change (i.e., gain). – Growth as a norm/peer referenced quantity. – Growth as a criterion referenced quantity.

Student Growth: Introductory Concepts • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which Hawaii has since 2015/SBAC) – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions – Interpreting change requires context.

Student Growth: Introductory Concepts Back to the story problem!

Student Growth: Introductory Concepts • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.

Student Growth: Introductory Concepts • The same holds true for the vertically scaled tests (e.g., SBAC). Average Scale Score Change 2015 to 2016 Top 10% Reading Grade 4 2015 +26.5 Bottom 10% Reading Grade 4 2015 +65.5 Top 10% Mathematic Grade 4 2015 +28.9 Bottom 10% Mathematics Grade 4 2015 +69.0 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point.

Student Growth: Introductory Concepts •This is analogous to height/weight measures for infants. •Growth amounts are tricky to interpret and require context. •Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm- referenced quantity locating that 4 inch increase alongside the height increases of similar children.

Student Growth: Introductory Concepts • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/ student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth. • The Hawaii SGP Model (i.e., the Colorado Growth Model) creates growth norms for all students using regression (i.e., peer-based) analyses. • These norms allow stakeholders to understand what an ambitious yet reasonable expectation is for each student.

Hawaii Growth Model (Student Growth Percentiles) • Research on the SGP model began in 2007. • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. • Goal or the model is to create an individual level growth metric that is informative to different stakeholders ranging from parents to policy makers.

Hawaii Growth Model (Student Growth Percentiles) • Different growth models answer different questions. • Different stakeholder groups ask different questions. • Parents are most concerned about how their child is doing. • To effectively engage parents and communities in a productive conversation about student learning (i.e. growth), a growth model should describe: • What is? How much are students learning/growing. • What should be? How much would we like students to grow/learn. • What is reasonable? How much is it reasonable to expect students to grow/learn. • Most other growth models do not provide a student level metric. They provide a school summary used for accountability purposes. 14

Hawaii Growth Model • Growth is achievement over time Exceeded Exceeds Met High Meets ● ● Typical Nearly Met Approaches Low Not Met Well Below HSA HSA SBA SBA SBA Math Math Math Math Math Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Scale Score 311 338 2553 2549 Achievement Level Meets Meets Met Nearly Met Achievement Growth Percentile 87 40 28 Growth Level High Typical Low Growth Source: Hawaii Department of Education/NCIEA

Hawaii Growth Model • Growth is achievement over time Exceeded Exceeds Met ● High Meets ● Nearly Met Typical Approaches Low Not Met Well Below HSA HSA SBA SBA SBA ELA ELA ELA ELA ELA Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Scale Score 313 325 2499 2442 Achievement Level Meets Meets Nearly Met Not Met Achievement Growth Percentile 91 33 6 Growth Level High Low Low Growth Source: Hawaii Department of Education/NCIEA

Growth Model: Other Options • Almost all current growth model use large amounts of data to create “norms” against which student growth is judged. • Absent large amounts of data (>1,000 grade/content area), professional judgment replaces large-scale data-based norms. • Value-tables can be used to monitor progress of students between achievement bands established for year-to-year assessments. • Some states (e.g., Alaska) use value-tables, particularly with smaller scale assessments for evaluating student growth for accountability purposes.

Accuracy & Precision • There was a recent report released questioning whether SGPs should be reported at all. • The report questioned whether SGPs had enough precision to be useful for stakeholders. • Similar to validity, precision (and accuracy) should always be considered in the context of use: • “Precise enough for what purpose”. • “Accurate enough for what purpose.”

Accuracy & Precision

Accuracy & Precision • Infrared thermometers are precise enough for the given purpose. • Educational assessment is like taking a child’s temperature: • You wouldn’t make a high stakes medical decision based solely on a child’s temperature. • Doctors use multiple pieces of evidence to diagnose and ultimately treat. • A child’s temperature is only useful up to a point. You could measure a child’s temperature to the .0001 degree. But what’s the point???? • High reliability/precision in educational assessment is often used (incorrectly) as a justification for a high stakes decision.

Conclusions & Recommendations • Subtraction isn’t a growth model. A growth model requires one consider where a student begins. • Norms require large numbers of students. Smaller numbers of students requires professional judgment to decide what is ambitious yet reasonable progress for a student. • Precision and accuracy of results must be balanced with utility: Precise and accurate enough for what purpose. • Technical characteristics of tests and growth measures derived from them a often misunderstood (even by testing experts). • Practical and technical issues become even more apparent when (high) stakes are associated with outcomes.

SGP: Academic Peers 250 0 0 co 15 0 2 5 S re 260 0 le S ca le 0 ca core 0 230 20 Source: Hawaii Department of Education/NCIEA

SGP: Academic Peers th 0 Per ce nt i 240 leG th 0 row 250 0 0 co 15 0 2 5 S re S le ca le 0 Sca S re 20 co Source: Hawaii Department of Education/NCIEA

SGP: Academic Peers th Per 230 ce nt i le rowth 2 250 0 0 co 15 0 2 5 S re S le ca le 0 S ca S re 20 co Source: Hawaii Department of Education/NCIEA

SGP: Academic Peers Pe 2 de m cor Ac cale 75% 75 th Per 230 ce nt i le rowth 2 250 0 0 co 15 0 2 5 S re S le ca le 0 S ca S re 20 co Source: Hawaii Department of Education/NCIEA

About this copy

Hawaii Growth Model: Changing Conversations about Student Performance — Hawaii REL, September 2016.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.