Sufficient Growth Discussion

Utah Department of Education · January 10, 2016

Sufficient Growth Discussion — Utah Department of Education, January 10, 2016.
Published

January 10, 2016

Event
State agency
Slides
28
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Utah State Office of Education Technical Advisory Committee Salt Lake City, Utah January 13-14, 2016
  2. “Sufficient Growth” (2) (a) For the purpose of determining whether a student demonstrates sufficient growth in the 2014-15 school year, or a succeeding school year, in language arts, mathematics, or science as provided in Section 53A-1-1107, the board shall establish a year 2 growth target for a student for each statewide assessment the student takes. (b) A student demonstrates sufficient growth if the student's scale score on a statewide assessment in year 2 is equal to or exceeds the year 2 growth target established pursuant to Subsections (2)(c) and (2)(d). (c) The board shall establish a year 2 growth target for each cohort of students with the same scale score on a particular statewide assessment in year 1. (d) (i) The board shall establish a year 2 growth target based on actual student growth in the 2014-15 school year as measured by statewide assessments administered at the end of the 2013-14 and 2014-15 school years. (ii) Among a cohort of students with the same scale score on a particular statewide assessment in the 2013-14 school year, the scale score of the student who scores on a similar statewide assessment in the 2014-15 school year, at a percentile determined by the board in rule, becomes the year 2 growth target for statewide assessments administered in the 2014-15 school year and succeeding years for any student with a year 1 scale score that is the same as the cohort's scale score in the 2013-14 school year. UT Technical Advisory Committee_Jan. 13-14, 2016 2
  3. Options Discussed by TAC, PAC, and State Board 1. Tie the target a student growth percentile (SGP) of 30 2. Set the next-year target scale score to be the cut score one level up from the current-year performance level (or the cut score for the highest performance level for students already achieving at this level) 3. Set next-year target scale scores separately for each current-year scale score deciles by calculating the average gain score for that decile based on prior years’ data, and adding that to the current-year scale score for every student. 4. Set the next-year target scale score to the Adequate Growth Percentile (AGP) necessary to move to the next performance level over a specified number of years or remain in the top performance level for a specified number of years 5. Set sufficient growth to an SGP of 40 for each student UT Technical Advisory Committee_Jan. 13-14, 2016 3
  4. Preferred Option: SGP = 40 • Set the target at the 40th growth percentile for each student • Determine the percentage of students in each school meeting the growth target • Follow the Grading Schools aggregation rules UT Technical Advisory Committee_Jan. 13-14, 2016 4
  5. But… • Utah’s key legislative leaders insisted that the law requires the use of a specific scale score instead of framing the target in the SGP metric • This is not unique to Utah. Previously, Arizona and Indiana have requested conversion of an SGP target to a scale score target. • The software used to calculates SGPs (SGP package) was extended several years ago to support that use case. UT Technical Advisory Committee_Jan. 13-14, 2016 5
  6. • Unfortunately, there are many misconceptions regarding SGP/scale score targets • On the following slides, we present some key properties associated with such targets and then offer two options for fulfilling the legislative request UT Technical Advisory Committee_Jan. 13-14, 2016 6
  7. Zero-sum game • A 40th percentile SGP/scale score target does not result in a zero-sum game where 40 percent of students “lose” and 60 percent of the students “win” • Targets are set a year in advance so (ideally) teachers know each student’s scale score goal at the beginning of the school year. • If the system improves in the ensuing year, then a higher percentage of students will reach their target than in the base year. – That is, more than 60% of students (even, in theory, 100% of students) can meet or exceed their target. UT Technical Advisory Committee_Jan. 13-14, 2016 7
  8. Comparable Targets • Basing targets on SGP=40 ensures that all student have the same relative difficulty in reaching their target, even though scale score increases/gains differ by student • There is a misconception that scale score increases/gain scores are uniform regardless of where a student starts – This simply is not true! • Establishing a single SGP target of 40 creates a uniform level of difficulty for all students even though the scale score changes for each student will differ UT Technical Advisory Committee_Jan. 13-14, 2016 8
  9. Comparable Targets Mean Scale Score Gains by Prior Score Decile: ELA Grade 4 UT Technical Advisory Committee_Jan. 13-14, 2016 9
  10. Comparable Targets Mean Scale Score Gains by Prior Score Decile: ELA Grade 5 UT Technical Advisory Committee_Jan. 13-14, 2016 10
  11. Comparable Targets Mean Scale Score Gains by Prior Score Decile: ELA Grade 6 UT Technical Advisory Committee_Jan. 13-14, 2016 11
  12. Comparable Targets Mean Scale Score Gains by Prior Score Decile: ELA Grade 7 UT Technical Advisory Committee_Jan. 13-14, 2016 12
  13. Comparable Targets Mean Scale Score Gains by Prior Score Decile: ELA Grade 8 UT Technical Advisory Committee_Jan. 13-14, 2016 13
  14. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Math Grade 4 UT Technical Advisory Committee_Jan. 13-14, 2016 14
  15. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Math Grade 5 UT Technical Advisory Committee_Jan. 13-14, 2016 15
  16. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Math Grade 6 UT Technical Advisory Committee_Jan. 13-14, 2016 16
  17. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Math Grade 7 UT Technical Advisory Committee_Jan. 13-14, 2016 17
  18. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Math Grade 8 UT Technical Advisory Committee_Jan. 13-14, 2016 18
  19. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Science Grade 5 UT Technical Advisory Committee_Jan. 13-14, 2016 19
  20. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Science Grade 6 UT Technical Advisory Committee_Jan. 13-14, 2016 20
  21. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Science Grade 7 UT Technical Advisory Committee_Jan. 13-14, 2016 21
  22. Comparable Targets Mean Scale Score Gains by Prior Score Decile: Science Grade 8 UT Technical Advisory Committee_Jan. 13-14, 2016 22
  23. Two Approaches • Single Prior • Multiple Priors • Student growth projections, which are the basis of the SGP/scale score targets utilized by Utah and other states can be calculated using either a single or multiple scale score priors (if available). • The use of single or multiple scale score priors is usually done to be consistent with the way in which SGPs are calculated. UT Technical Advisory Committee_Jan. 13-14, 2016 23
  24. Single Prior • Ensures that every student with the same prior (Year 1) score will have the same scale score target (Year 2) • An advantage for using a single prior stems from the belief that it is more straightforward to communicate a target based upon a single rather than multiple priors • Disadvantage is that the single prior is the least accurate (and perhaps least reliable) case of SGPs UT Technical Advisory Committee_Jan. 13-14, 2016 24
  25. Multiple Priors • SGPs are almost always calculated with multiple scale score priors (when available). Thus, projections for students also utilize the same multiple scale score priors. • Multiple priors lead to more accurate individual targets. The disadvantage to using multiple priors often stems from the belief that it is more difficult to communicate a target based upon multiple priors than a single prior – For example, two students with the same immediately prior score may (likely will) have different targets based on the full score history UT Technical Advisory Committee_Jan. 13-14, 2016 25
  26. Single versus multiple prior based targets • Low ELA achieving student in 2014 (below prof/360) who had high growth in 2015 (SGP=92) raising their status (proficient/450). • Top Figure: Using only 2015 Score to set target for 2016 gives a scale score target of 465 (SGP 40 with 1 prior). • Bottom Figure: Using both 2014 and 2015 Scores to set student target for 2016 gives a scale score target of 445 (SGP 40 with 2 priors) . • Ignoring the 2014 scale score sets too high a target for the student. UT Technical Advisory Committee_Jan. 13-14, 2016 26
  27. Single versus multiple prior based targets • High ELA achieving student in 2014 (advanced/508) who had low growth in 2015 (SGP=3) dropping their status (proficient/450). • Top Figure: Using only 2015 Score to set target for 2016 gives a scale score target of 465 (SGP 40 with 1 prior). • Bottom Figure: Using both 2014 and 2015 Scores to set student target for 2016 gives a scale score target of 492 (SGP 40 with 2 priors) . • Ignoring the 2014 scale score sets too low a target for the student. UT Technical Advisory Committee_Jan. 13-14, 2016 27
  28. Discussion and Recommendations • Which option seems most promising and why? • Does the TAC think the increased accuracy associated with using multiple priors is worth the potential explanatory challenges? • If so, can the TAC suggest some good strategies for helping USOE construct compelling explanations? UT Technical Advisory Committee_Jan. 13-14, 2016 28