Talks archive, 2010–2014 – On Student Growth Percentiles & Accountability in Hawaii

References On Student Growth Percentiles & Accountability in Hawaii Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Teacher Leader Work Group March 10, 2014 Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012, Betebenner et al., 2014]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References Student Growth Percentiles 2007 Began development of the Colorado Growth Model (student growth percentiles). 2008 Began work with Massachusetts, Indiana and other states on implementation of growth model. 2008 Successfully application of Colorado Growth Model (criterion-referenced component) as part of growth model pilot program. 2009-2010 Began/continued work with approximately two-dozen other states on developing state growth models. 2010-2014 Worked with states on incorporating growth into RttT applications and with successful states on their use (Hawaii, Massachusetts, Georgia, Rhode Island). 2011-2014 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2014- Working with states transitioning to PARCC, SBAC assessment systems. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

Student Growth Percentile Adoption Adopted Model Considering Adoption Investigating Model

References About the growth model The Student Growth Percentiles Methodology is a norm- and criterion-referenced growth analysis method. The norm-referenced component of the model is the student growth percentile which indicates the relative growth a student makes. The criterion-referenced component of the model is the student growth percentile target which indicates the amount of growth required by the student to reach/maintain desirable achievement goals in a specified time frame. For example, catching-up to proficiency within 3 years. The norm- and criterion-referenced components together allow stakeholders to understand what is, what should be, and what is reasonable simultaneously. The goal is to create a durable metric to assist in moving beyond debates over what thermometer is best to discussions of how best to use the thermometer. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References Describing Student Growth Measuring (quantifying) student growth is not a simple task. Even with an interval (or ratio) scale (e.g., a yard stick), growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a growth norm to help understand student growth. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

Advanced Reading Achievement Proficient ● DEMO Reading Scale Score ● Catch Up Target Growth Part Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Typical 40th − 60th Unsatisfactory Low 20th − 39th Very Low 1st − 19th Grade 3 Grade 4 Scale Score 449 Achievement Level Achievement Target Unsatisfactory Proficient (529) Achievement Growth Percentile Growth Level Growth Target Catch Up (64) Growth

Advanced Reading Achievement DEMO Reading ● Scale Score Proficient ● Catch Up Growth Target Level Percentiles Very High 81st − 99th Part Proficient High 61st − 80th ● Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Scale Score 449 533 Achievement Level Achievement Target Unsatisfactory Part Proficient Proficient (562) Achievement Growth Percentile 69 Growth Level Growth Target High Catch Up (76) Growth

Advanced Reading Achievement Proficient ● ● ● Catch Up Target DEMO Reading Scale Score Growth Part Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Typical 40th − 60th Unsatisfactory Low 20th − 39th Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Grade 6 Scale Score 449 533 579 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient (586) Achievement Growth Percentile 69 85 Growth Level Growth Target High Very High Catch Up (71) Growth

Advanced Reading ● Move Up Achievement Target DEMO Reading ● Scale Score Keep Up ● Target Proficient Growth Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Keep Up (76)/Move Up (99) Growth

References What Norm Group? States generally re-norm growth every year such that each grade by content area set of students forms an SGP growth norm group. As such, the median SGP associated with each grade by content area group of students will be 50 each year. Some states (e.g., Massachusetts and Georgia) have desired to fix a baseline set of students against which to measure growth. Baseline referenced SGPs (because they are anchored to student growth from previous years) could have medians above/below 50. Calculating baseline SGPs requires at least four years of data in each grade and content area built on a stable scale. That is, one can’t have scale drift. Transitioning to a consortium assessment (SBAC/PARCC) will not permit the calculation of baseline referenced SGPs until at least 4 years of data are available on the new scale. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References Going from Students to Classrooms/Schools It’s of interest to examine schools/classrooms where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school/classroom (or other grouping) Colorado calculates the median of the student growth percentiles. If students were randomly assigned to classrooms/schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school/classroom performance. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

Hawaii School Performance HSA Growth & Prior Achievement 2013 HSA Mathematics Higher Achievement Higher Achievement School Size 2012 Prior Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2013 Median Student Growth Percentile

References Going from Students to Classrooms/Schools Refinements to the bubble plot can help investigate trends over time and see whether programs have led to a desirable outcome. For example, Hawaii created the Zones for School Innovation to help schools serving low achieving students. The following sequence of bubble plots highlight these schools (against the back drop of all schools) and examining their performance from 2009 to 2013. The trends observed in the bubble plots are consistent with a positive impact in both reading and mathematics. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

2009 School Performance Reading Growth & Achievement 2009 Reading Higher Achievement Higher Achievement School Size 2008 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students XX XX X X XX X Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2009 Median Student Growth Percentile

2010 School Performance Reading Growth & Achievement 2010 Reading Higher Achievement Higher Achievement School Size 2009 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2010 Median Student Growth Percentile

2011 School Performance Reading Growth & Achievement 2011 Reading Higher Achievement Higher Achievement School Size 2010 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

2012 School Performance Reading Growth & Achievement 2012 Reading Higher Achievement Higher Achievement School Size 2011 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012 Median Student Growth Percentile

2013 School Performance Reading Growth & Achievement 2013 Reading Higher Achievement Higher Achievement School Size 2012 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students 50 XX Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2013 Median Student Growth Percentile

2009 School Performance Mathematics Growth & Achievement 2009 Mathematics Higher Achievement Higher Achievement School Size 2008 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement XX Lower Growth X Higher Growth Lower Achievement Lower Achievement Growth 2009 Median Student Growth Percentile

2010 School Performance Mathematics Growth & Achievement 2010 Mathematics Higher Achievement Higher Achievement School Size 2009 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement XX X XX X Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2010 Median Student Growth Percentile

2011 School Performance Mathematics Growth & Achievement 2011 Mathematics Higher Achievement Higher Achievement School Size 2010 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

2012 School Performance Mathematics Growth & Achievement 2012 Mathematics Higher Achievement Higher Achievement School Size 2011 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students XX Achievement Lower Growth X Higher Growth Lower Achievement Lower Achievement Growth 2012 Median Student Growth Percentile

2013 School Performance Mathematics Growth & Achievement 2013 Mathematics Higher Achievement Higher Achievement School Size 2012 Percent at/above Proficient (Prior) Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students XX Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2013 Median Student Growth Percentile

References Going from Students to Classrooms/Schools Hawaii is currently discussing how to characterize instructor median SGPs as part of educator educator evaluation There are numerous ways states are doing this Fixed (i.e., SGP based) cuts. Adding confidence intervals around the median SGP or building them into the cuts. Variable SGP based cuts. Hawaii is currently investigating approach number 3. Hawaii’s approach will always yield the same percentage of teacher in the same categories (i.e., like grading on a curve). The cuts for the categories will likely change every year leading to a communication issue that will likely need to be addressed. This approach would not be consistent with baseline referenced SGPs, where the desire is to see higher and higher percentages of teachers in the effective and highly effective categories. Caterpillar plots can help illustrate some design choices. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

Median Teacher Student Growth Percentile +/− 1 Standard Error Median SGP Median SGP Teacher Rank

References The Transition from HSA to SBAC Some types of growth will still be calculable while transitioning from HSA to SBAC. In general, norm-referenced quantities will be calculable. Criterion-referenced quantities will require some concordance between old and new test to be calculated. Hawaii is not currently using criterion-referenced growth as part of reporting or accountability. Baseline referenced SGPs won’t be calculable until multiple years of data on the new scale is available. States already considering growth during assessment transition consider communication across the assessment transition to be their most significant difficulty (not analysis). Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References Questions??? Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Van Iwaarden, A., Domingue, B., and Shang, Y. SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.2-2.0. Damian W. Betebenner On Student Growth Percentiles & Accountability in Hawaii

Hawaii: 2013 Mathematics Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment 2013 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

Hawaii: 2013 Reading Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment 2013 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10

About this copy

On Student Growth Percentiles & Accountability in Hawaii — Hawaii Department of Education, March 10, 2014.

This is an image-based copy of the original beamer slides; builds and animations are flattened.