Talks archive, 2010–2014 – Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability

Overview Different Models/Different Questions Growth Models References Appendix Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Indianapolis, Indiana November 8th, 2013 Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix Student Growth Percentiles 2007 Began development of the Colorado Growth Model (student growth percentiles/percentile growth trajectories). 2008 Successful application of Colorado Growth Model (criterion-referenced/growth target component) as part of USED Growth Model Pilot Program. 2009 Began work with Indiana on the calculation and reporting of student growth percentiles and percentile growth trajectories. 2010 Worked with states on incorporating growth into RttT applications and with successful states on their use (Massachusetts, Georgia, New York, Rhode Island). 2011 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2012 Worked with 5 states on co-developing interactive visualizations to help better communicate student growth percentile results. 2013 Working to enhance collaboration between states on beneficial uses of growth data. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix About the growth model The Student Growth Percentiles Methodology is an open source norm- and criterion-referenced growth analysis method. The norm-referenced component of the model is the student growth percentile which indicates the growth a student makes. Student growth percentiles can be cohort (peer) referenced or baseline (non-peer) referenced. The criterion-referenced component of the model is the student growth percentile target which indicates the amount of growth required by the student to reach/maintain desirable achievement goals in a specified time frame. For example, catching-up to proficiency within 3 years. The norm- and criterion-referenced components together allow stakeholders to understand what is, what should be, and what is reasonable simultaneously. The goal of the metric was to assist states to move beyond debates over what “thermometer" is best to discussions of how best to use the “thermometer". Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix Built to maximize sharing A number of analytic tools (for doing calculations) and visualization tools have been developed with the goal to promote maximum sharing. Data isn’t valuable! It’s what one does with data that is potentially valuable! The analytic tools to calculate all the quantities associated with student growth percentiles are open source and freely available to all states [Betebenner et al., 2013]. https://github.com/CenterForAssessment The visualizations associated with student growth percentiles are open source for non-commercial use. Rhode Island is the latest: http://rigm.ride.ri.gov/growthmodel/index.htm Conduct training sessions at national conferences to help others calculate these quantities (like Indiana) on their own. Build enhanced visualization and reporting tools to make data come alive to stakeholders, allow for stakeholder collaboration and sharing, and lead to better stakeholder engagement with data. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide context for describing and ultimately understanding student growth. Damian W. Betebenner Growth Models and Accountability

Joshua Childress−Younger Sample School How to interpret this student growth & achievement report ELA Pass + Achievement ISTEP+ ● ISTEP+ ELA ● Scale Score Scale Score ● ● Growth Level Percentiles ISTEP+ ● High 65th − 99th Achievement Pass Levels ● Typical 35th − 64th Low 1st − 34th Did Not Pass Student Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Growth 2009 2010 2011 2012 2013 Percentile Scale Score 469 498 546 569 649 Achievement Level Achievement Target Pass Pass Pass Pass Pass Achievement Growth Percentile 57 75 90 81 Catch Up/Keep Up Growth Level Typical High High High Growth Move Up/Stay Up Growth Target Targets Suggested Uses Math Pass + Achievement ISTEP+ Math ● ● Scale Score ● Review past growth to assess student academic progress toward ● Growth ISTEP+ achievement goals. Level Percentiles ● Develop remediation or enrich− High 65th − 99th ment plans based on rate of growth needed to reach higher Pass Typical 35th − 64th ISTEP+ achievement levels. ● Identify the rate of progress ● Low 1st − 34th needed in order to reach or Did Not Pass maintain proficient status Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 on the ISTEP+ next year. 2009 2010 2011 2012 2013 Scale Score 435 518 554 580 649 Achievement Level Achievement Target Did Not Pass Pass Pass Pass Pass + Achievement Growth Percentile 85 92 72 94 Growth Level Growth Target High High High High Growth For more information please visit the Indiana Department of Education (IDOE) at www.doe.in.gov or contact webmaster@doe.in.gov Cooperatively developed by the IDOE & the Center for Assessment, Inc. Distributed by the IDOE.

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Two Flavors of Growth based Accountability Our work in the last few years investigating growth/value-added models has led us to believe that there are two general types of growth based accountability Norm-referenced growth based accountability. Criterion-referenced growth based accountability. These two approaches to growth impart different levels of stringency Norm-referenced Is the group of students growing better/worse (on average) than statistical expectation? Criterion-referenced Is each student in the group growing at a sufficient rate (i.e., targeted growth) to reach/maintain a desired level of achievement (e.g. Pass or Pass +)? Damian W. Betebenner Growth Models and Accountability

Indiana: 2013 ELA Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Norm-referenced student growth/accountability Using student growth percentiles, the median SGP for a group describes whether students in the group, on “average”, demonstrated growth above or below expectation. This is essentially, what value-added models quantify. Value-added models go an extra step an implicitly infer causality by labeling the aggregate measure as a school or teacher effect. Some value-added models include student level characteristics (e.g., demographics) and thus set differential expectations. Student growth percentiles do not include student level characteristics in the model. Any adjustments needed for accountability purposes can occur post-hoc. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Criterion-referenced student growth/accountability Former Secretary of Education Margaret Spellings initiated the Growth Model Pilot Program allowing states to include growth as a means of satisfying AYP. The “Bright Line Principle” of all students being proficient by 2014 remained. 15 states were accepted as part of the pilot. None saw a significant improvement in their AYP numbers because of the growth-to-proficiency goals. Indiana’s proposed growth-to-standard approach put forward in the Report of the Accountability System Review Panel is a trajectory gain score model utilized by many of the 15 states as part of the growth model pilot program. To my knowledge, none of the states adopting this approach outlined by Indiana continue to use that model. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Criterion-referenced student growth/accountability The reality in all state education systems I have worked with is that students who start behind tend to stay behind (Did Not Pass) and students who start ahead tend to stay ahead (Pass or Pass +). Targeted growth reflects this unfortunate reality: Schools serving the lower achieving students will likely have low percentages of students reaching targeted growth and schools serving high achieving students will likely have high percentages of students reaching targeted growth. Along, growth to proficient provides an impoverished view of student growth. This is not to say that targeted growth is unimportant. I must be balanced with a norm-referenced view of student progress. The SGP methodology’s norm and criterion-referenced components attempt to strike that balance allowing stakeholders to understand: What is? , What should be? and What is Reasonable? Damian W. Betebenner Growth Models and Accountability

Jaya Roberds Sample School How to interpret this student growth & achievement report ELA Achievement Pass + ISTEP+ ● ISTEP+ ELA ● ● Stay ●● Up Target Scale Score Scale Score Growth Level Percentiles ISTEP+ ● Keep Up Target High 65th − 99th Achievement Pass Levels ● Typical 35th − 64th Low 1st − 34th Did Not Pass Student Grade 3 Grade 4 Grade 5 Grade 6 Growth 2011 2012 2013 2014 Percentile Scale Score 451 493 554 Achievement Level Achievement Target Pass Pass Pass + Pass (502)/Pass + (579) Achievement Growth Percentile 79 74 Catch Up/Keep Up Growth Level High High Growth Move Up/Stay Up Growth Target Keep Up (5)/Stay Up (65) Targets Suggested Uses Math Pass + Achievement ● Move Up Target ISTEP+ Math ● ● Scale Score ● Review past growth to assess student academic progress toward ● Keep Up Growth Target ISTEP+ achievement goals. ● ● Level Percentiles ● Develop remediation or enrich− High 65th − 99th ment plans based on rate of growth needed to reach higher Pass Typical 35th − 64th ISTEP+ achievement levels. ● Identify the rate of progress Low 1st − 34th needed in order to reach or Did Not Pass maintain proficient status Grade 3 Grade 4 Grade 5 Grade 6 on the ISTEP+ next year. 2011 2012 2013 2014 Scale Score 506 512 554 Achievement Level Achievement Target Pass Pass Pass Pass (532)/Pass + (574) Achievement Growth Percentile 38 58 Growth Level Growth Target Typical Typical Keep Up (9)/Move Up (63) Growth For more information please visit the Indiana Department of Education (IDOE) at www.doe.in.gov or contact webmaster@doe.in.gov Cooperatively developed by the IDOE & the Center for Assessment, Inc. Distributed by the IDOE.

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Appendix Recommendations Consistent with the duties charged to the Accountability System Review Panel the state should perform thorough impact analyses on the outcomes of the results including: How does growth relate to achievement at both the individual and group level? Do schools serving lower achieving students demonstrate markedly different growth than schools serving high achieving students? Do growth results differ by school level (elementary, middle, high)? Do growth results differ by important demographic subgroups in the state? Are any differences observed consistent with expectations and acceptable? To what extent do the results of the growth model change the outcomes of the final A-F ratings based just upon status? Does growth answer questions of relevance to the various stakeholder groups and are those questions consistent with larger policy goals? Is the impact of growth on A-F outcomes coherent with policy? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Appendix References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Van Iwaarden, A., and Domingue, B. (2013). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.0-4.0. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Growth Models and Accountability

Indiana: 2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

About this copy

Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability — Indiana Department of Education, November 8, 2013.

This is an image-based copy of the original beamer slides; builds and animations are flattened.