Understanding Student Growth

Center for Assessment RLI · May 2016

Understanding Student Growth — Center for Assessment RLI, May 2016.
Published

May 12, 2016

Event
Colloquium
Slides
38
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. UNDERSTANDING STUDENT GROWTH Damian W Betebenner National Center for the Improvement of Educational Assessment Bedford MA May 10th, 2016
  2. Why Student Growth? If you care about student achievement, then you care about student growth. ๏ Where you are is a function of where you started and how quickly you’ve travelled since. ๏ Student growth and student attainment/ achievement are related in this way. ๏ Distance=Rate x Time ๏ Those who start behind tend to stay behind and those that start ahead tend to stay ahead.
  3. “If you can’t solve a problem, then there is an easier problem you can solve: find it.” George Pólya Mathematical Discovery on Understanding, Learning, and Teaching Problem Solving, Volume I
  5. A 5th Grade Story Problem ๏ Whose high jump changed more? Explain your answer. ๏ Whose high jump change is more remarkable? Explain your answer. ๏ Which high jumper improved more? Explain your answer.
  6. A 5th Grade Story Problem ๏ Judy improved 2 inches. ๏ Anna Improved 6 inches. ๏ 6 inches > 2 inches. ๏ An inch is an inch!!! ๏ But is an inch really an inch???
  7. A 5th Grade Story Problem ๏ Are we comparing apples to apples? ๏ Anna’s “beginner’s gains” happen in many sports. ๏ 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.
  8. A 5th Grade Story Problem Properties that don’t hold in simple situations will not hold in complicated situations. This simple example illustrates how minimally helpful a scale is to understanding change. ๏ A vertical scale allows one to answer a very narrow question: Who changed more? ๏ Remarkable improvement need not be synonymous with a large change. ๏ This example illustrates that a vertical scale is of little help in answering substantive questions.
  9. Another Story ๏ Suppose your child showed no change in height between the ages of 3 and 5. ๏ Upon discovering this your pediatrician responds: “Don’t worry, my height hasn’t changed for the last 10 years and I’m perfectly healthy.” ๏ What do you think of this Doctor’s diagnosis? ๏ Again, 0 inches = 0 inches. But is an inch an inch??? ๏ It’s remarkable (and extremely concerning!) for a toddler not to grow. It’s not remarkable for an adult to show no change in height. ๏ Context matters!
  10. Student Academic Growth ๏ The same holds true for the vertically scaled tests (e.g., STAR). Average Student Scale Score Change 2014 to 2015 Top 10% Reading Grade 4 2014 -7.5 Bottom 10% Reading Grade 4 2014 +24.5 Top 10% Mathematic Grade 4 2014 -12.6 Bottom 10% Mathematics Grade 4 2014 +18.4 ๏ Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point. ๏ Low achieving students tend to show much larger scale score gains than high achieving students! ๏ This isn’t a defect of test scales! It’s just the way it is (like high jumping).
  11. Moral of the Story Subtraction isn’t a growth model. Subtraction only gets partial credit. Must go beyond simple subtraction to get full credit and really understand student academic growth.
  12. So how can one understand student growth?
  13. Context is Important ๏ Analogous to height/weight measures for infants. ๏ Growth amounts are tricky to interpret and require context. ๏ Peer based comparisons used by growth analyses are a way to acquire that context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches—is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm- referenced quantity locating that 4 inch increase alongside the height increases of similar children.
  14. Context is Important ๏ Peer-based analyses are sometimes called norm-referenced analyses. ๏ Like with infant height/weight charts, peers are infants/toddlers with similar characteristics (e.g., sex and age). ๏ For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. ๏ Multiple priors give more accurate starting point (more on that later). ๏ Demographics not used but could be. ๏ For interim assessments like STAR, time elapsed between tests is a peer characteristic. ๏ Peer based norms are invaluable in providing context for stakeholders to understand student growth. ๏ Student growth percentiles (SGP) were created to provide a peer- based context to understand student academic growth.
  15. Student Growth Percentiles ๏ Research on the SGP model began in 2007 (Betebenner, 2008). ๏ Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. (Betebenner, 2009). ๏ Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. (Betebenner, 2012). ๏ Used in approximately 30 states for various diagnostic and accountability purposes. ๏ Extended for use with interim assessments in collaboration with Renaissance Learning (2012).
  16. 6 0 co T S re im 600 0 le e 1 4 0 ca ca 2 le 800 0 e S 2 0 im core
  17. 6 0 co T S re im 600 0 le e 1 4 0 ca ca 2 le 800 0 e S 2 0 im core
  18. 6 0 co T S re im 600 0 le e 1 4 0 ca ca 2 le 800 0 e S 2 0 im core
  19. 8 co re 0 0 le 6 ca th S ro nt i le er ce 4 im th re co le ca im
  20. 8 co re th 0 0 le ro w 6 ca nt i le S er ce 2 th 75 4 im re co le ca im
  21. e m Sc o a a le A Sc 7 5 75th er ce nt i le G rowth 8 0 6 0 co T S re im 600 0 le e1 4 0 S ca ca 2 le 800 0 e S 2 0 T im core
  22. Student Growth Percentiles (SGPs) ๏ Similar to height and weight percentiles for infants/toddlers, SGPs range from 1 to ๏ 1st percentile growth is impoverished and 99th percentile growth is exemplary. ๏ Peer-based growth calculations (SGPs) are only as good as the assessments on which they are based. ๏ SGPs are uncorrelated with students beginning level of achievement and allow for all students to demonstrate growth regardless of whether they are low or high achieving. ๏ SGPs can be aggregated for any group of interest to see whether the group is demonstrating, on average, high or low growth. ๏ School/District ๏ Demographic subgroup ๏ Classroom/Teacher ๏ Curriculum/Program
  23. Why more than one prior score? ๏ STAR SGPs use up to two prior scores in their calculation. ๏ For example, a student’s 2015-2016 Fall to Spring SGP would use (if available) both the student’s Spring 2014-2015 & Fall 2015-2016 prior scores. ๏ The additional prior score gives a more accurate starting level for the student. ๏ There is concern that including 2nd prior contaminates the SGP with what happened at an earlier time. ๏ Including 2nd prior: SGP doesn’t depend on what happened earlier. ๏ Not including 2nd prior: SGP does depend on what happened earlier. ๏ Consider the following tale of two students.
  24. Why more than one prior score? ๏ Fall to Spring 2015-2016: 469 to 529 ๏ Only one prior: SGP 76, high growth Reading Advanced Very High High Achievement Typical ● Low DEMO Reading Scale Score Very Low ● Growth Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th −− −− Grade 6 Grade 6 Grade 7 2014−2015 Fall 2014−2015 Spring 2015−2016 Fall 2015−2016 Spring 2016−2017 Fall Scale Score −− −− 469 529 Achievement Level −− −− Part Proficient Proficient Achievement Growth Percentile −− −− 76 Growth Level −− −− High Growth
  25. Why more than one prior score? ๏ Same scores with 2nd prior from Spring 2014-2015: 396 ๏ SGP 96: Very high growth. Fall to Spring growth is more impressive given lower previous achievement. Reading Advanced Very High Achievement High Typical ● Low DEMO Reading Scale Score Very Low ● Growth Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Part Proficient ● Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th −− Grade 5 Grade Grade 6 Grade 7 Scale Score −− 369 469 529 Achievement Level −− Unsatisfactory Part Proficient Proficient Achievement Growth Percentile −− 90 96 Growth Level −− Very High Very High Growth
  26. Why more than one prior score? ๏ Same scores with 2nd prior from Spring 2014-2015: 550 ๏ SGP 51: Typical growth. Fall to Spring growth is less impressive given high previous achievement. Reading Advanced Very High High Achievement ● Typical ● Low DEMO Reading Scale Score Very Low ● Growth Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th −− Grade 5 Grade 6 Grade 6 Grade 7 2014−2015 Fall 2014−2015 Spring 2015−2016 Fall 2015−2016 Spring 2016−2017 Fall Scale Score −− 550 469 529 Achievement Level −− Advanced Part Proficient Proficient Achievement Growth Percentile −− 1 51 Growth Level −− Very Low Typical Growth
  27. Why more than one prior score? ๏ SGPs indicate how impressive growth is given prior achievement. ๏ Including second prior gives a more accurate view of growth than without. ๏ Omitting 2nd prior would under-value growth for students who have continued upward and over-value growth for students who dropped and bounced back. ๏ That is, previously (2nd prior) high achieving students would be favored for higher growth. ๏ Including 2nd prior makes growth (SGP) independent of 2nd prior.
  28. Student Growth Percentiles ๏ Peer based comparisons (norms) are helpful but limited by their very nature. ๏ Norms tell you where you are in a distribution but not whether the ranking is good or bad. ๏ One needs criteria as well as norms to have a complete understanding. ๏ A criterion-referenced interpretation examines the quantity against a set of standards. ๏ A criterion-referenced interpretation tries to answer: ๏ What is “good” growth? ๏ What is “bad” growth? ๏ What is “good enough” growth?
  29. Student Growth Percentiles ๏ Criterion-referenced growth is also referred to as growth-to-standard. ๏ The standards used to examine growth are often the state’s performance standards. ๏ Peer based analyses (e.g., SGP) are used in many states as part of growth-to- standard analyses. ๏ Growth-to-standard analyses produce growth targets determining whether a student’s growth is sufficient to reach/maintain state standards. ๏ STAR interim assessments also provide growth to standard information where the standards are state standards linked to the STAR scale. ๏ Absent norms, criterion-referenced growth produces results similar to achievement indicating: ๏ Students that start behind tend to stay behind. ๏ Students that start ahead tend to stay ahead.
  30. Growth & Accountability ๏ Most of the controversy surrounding student growth analyses centers on their (high stakes) uses: ๏ Teacher evaluation ๏ School evaluation/accreditation ๏ Principal evaluation ๏ There is broad disagreement on whether results derived from large scale standardized assessments (state summative, interim) are appropriate for personnel or institutional evaluation. ๏ Indicators/measures and uses often get conflated leading to push back against the measures (assessments) themselves. ๏ There are many great uses for student growth results (SGPs) besides personnel/institutional evaluation.
  31. Growth & Accountability Non-Accountability Uses of Growth ๏ Progress monitoring of individual students: Is a student on track to college readiness? ๏ Curricula evaluation: Do students receiving different curricula demonstrate different growth? ๏ Ed-System evaluation: Do students at different grade levels in the district grow at the same rate? ๏ Test monitoring: Is their a mode effect in the test administration? ๏ Monitoring progress during standards transition: Were some schools more successful at transition to new standards than other schools in terms of the growth shown?
  32. Growth & Teacher Evaluation ๏ Likely the most controversial use of tests scores. ๏ Began with Tennessee Value-Added Assessment System (TVAAS now EVAAS) in early 90s. ๏ With the availability of longitudinal student assessment data from NCLB, data was available in every state for the purpose. ๏ Increased dramatically in late 2000s/early 2010s under Obama/ Duncan Accountability Policy. ๏ Numerous states also adopted laws requiring teacher evaluation based in some part on test scores. ๏ Might have been the straw (along with Common Core) that broke the “Ed Reform” back.
  33. Growth & Teacher Evaluation ๏ Student growth is usually only a minority part of the overall teacher evaluation. ๏ Weights applied to student growth in teacher evaluation vary widely (e.g., not at all to 50%). ๏ ESSA has decreased pressure for new teacher evaluation systems in states. ๏ Evaluation in non-tested grades/subjects has yielded some questionable practices. ๏ Unintended consequences (e.g., Atlanta cheating scandal) provide cautionary tales. ๏ What might Accountability 2.0 look like?
  34. Accountability 2.0 ๏ Based upon general dissatisfaction with accountability systems emerging from the NCLB/Waiver era and new flexibility under ESSA, states are asking what an accountability system might look like. ๏ Design principles: ๏ Accountability done With stakeholders instead of To stakeholders. ๏ Accountability as a flashlight instead of a hammer. ๏ Data used for improvement instead of punishment. ๏ The change from Accountability 1.0 to 2.0 is more cultural than technical.
  35. References ๏ Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155-170. Taylor & Francis, New York. ๏ Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4): 42-51. ๏ Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439-450. Routledge, New York. ๏ Betebenner, D. W., VanIwaarden, A., Domingue, B., and Shang, Y. (2016). SGP: Student Growth Percentiles & Percentile Growth Trajectories. (R package version 1.5-0.0)
  37. Growth & Teacher Evaluation Argument #1 for Teacher Evaluation ๏ Value-added methods (VAMs) measure the teacher’s contribution to student learning. ๏ Because student learning is important and we can measure the teacher’s contribution to it, teachers should be held accountable for student learning. ๏ This argument is often proffered by economists. ๏ Arguments against it often focus on the technical characteristics of the VAMs: ๏ Unreliability: Values can change a lot from year-to-year ๏ Invalidity: Examples abound of great teachers receiving low VAM scores. ๏ Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
  38. Growth & Teacher Evaluation Argument #2 for Teacher Evaluation ๏ Student learning is important. ๏ Because student learning is important, providers of education should be held accountable, in part, for student learning. ๏ Indicators of student learning can be derived from assessment scores (e.g., SGP). ๏ Similar to holding a sports coach responsible, in part, for the win/loss percentage of his/her team. ๏ Arguments against it often focus on the technical characteristics of the VAMs: ๏ Unreliability: Values can change a lot from year-to-year. ๏ Invalidity: Examples abound of great teachers receiving low growth scores. ๏ Campbell’s Law (what often happens when winning is valued too much?)