Talks archive, 2010–2014 – Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability

Overview Different Models/Different Questions Growth Models References Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Mississippi Department of Education Jackson, Mississippi June 30th, 2014 Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [?, ?, ?]. - Currently refining and sharing these techniques with other states including Mississippi, Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Student Growth Percentiles 2007 Began development of the Colorado Growth Model (student growth percentiles/percentile growth trajectories). 2008 Successful application of Colorado Growth Model (criterion-referenced/growth target component) as part of USED Growth Model Pilot Program. 2010 Worked with states on incorporating growth into RttT applications and with successful states on their use (Massachusetts, Georgia, New York, Rhode Island). 2011 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2012 Worked with 5 states on co-developing interactive visualizations to help better communicate student growth percentile results. 2013 Working to enhance collaboration between states on beneficial uses of growth data. 2014 Working with numerous state partners and the SBAC and PARCC assessment consortia to understand the impact assessment transition will have on growth analyses. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References About the growth model The Student Growth Percentile (SGP) Methodology is an open source norm- and criterion-referenced growth analysis method. The norm-referenced component of the model is the student growth percentile which indicates the growth a student makes. Student growth percentiles can be cohort (peer) referenced or baseline (non-peer) referenced. The criterion-referenced component of the model is the student growth percentile target which indicates the amount of growth required by the student to reach/maintain desirable achievement goals in a specified time frame. For example, catching-up to proficiency within 3 years. The norm- and criterion-referenced components together allow stakeholders to understand what is, what should be, and what is reasonable simultaneously. The goal of the metric was to assist states to move beyond debates over what “thermometer" is best to discussions of how best to use the “thermometer". Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Built to maximize sharing A number of analytic tools (for doing calculations) and visualization tools have been developed with the goal to promote maximum sharing. Data isn’t valuable! It’s what one does with data that is potentially valuable! The analytic tools to calculate all the quantities associated with student growth percentiles are open source and freely available to all states [?]. https://sgp.io The visualizations associated with student growth percentiles are open source as well. Rhode Island is the latest: http://rigm.ride.ri.gov/growthmodel/index.htm Conduct training sessions at national conferences like AERA/NCME to help others calculate these quantities on their own. Build enhanced visualization and reporting tools to make data come alive to stakeholders, allow for stakeholder collaboration and sharing, and lead to better stakeholder engagement with data. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Why Student Growth? Where you are today is a result of where you started and how quickly you’ve travelled since. Distrance = Rate × Time. In education, where a student starts if often out of the schools control. However, once in the school, the general belief is that we have some control over how quickly a student "travels". Those students who start behind have to travel more quickly to catch-up given a fixed amount of time to reach the destination (e.g., 12 years). Those students who start ahead don’t have to travel as quickly to reach the desired destination on time. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for policy makers? Policy Maker Questions Can/how should student growth be used in judgments of educator quality? Can/how should student growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for student demographics need to be considered? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide context for describing and ultimately understanding student growth. Damian W. Betebenner Growth Models and Accountability

20 core ca ca le le 200 S re 2012 co

co re le ca ro th nt i le 12 ce er th 175 re co 150 le ca

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Growth Models and Accountability

Advanced Reading Achievement Very High DEMO Reading ● Scale Score High Typical Proficient Low Growth Level Percentiles Very Low ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth

Billy Gonzales North End Middle School How to interpret this student growth & achievement report Math Advanced Very High Achievement DEMO High ● DEMO Math ● Typical Scale Score Scale Score ● Low ● Growth Proficient Very Low Level Percentiles DEMO Very High 81st − 99th Achievement High 61st − 80th Levels Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Student Growth Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Percentile Scale Score 438 447 489 495 Achievement Level Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 18 64 44 Growth Level Very Low High Typical Growth Suggested Uses Review past growth to assess student academic progress toward Advanced Reading DEMO achievement goals. Achievement ● Develop remediation or enrich− Very High DEMO Reading ment plans based on rate of ● Scale Score High growth needed to reach higher ● Typical DEMO achievement levels. Proficient Low Growth Level Percentiles ● Identify the rate of progress Very Low needed in order to reach or ● Very High 81st − 99th maintain proficient status High 61st − 80th on the DEMO next year. Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth For more information please visit the Student Growth Percentiles (SGP) at centerforassessment.github.io/SGP/ or contact dbetebenner@nciea.org Cooperatively developed by the SGP & the Center for Assessment, Inc. Distributed by the SGP.

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Two Flavors of Growth based Accountability Our work in the last few years investigating growth/value-added models has led us to believe that there are two general types of growth based accountability Norm-referenced growth based accountability. Criterion-referenced growth based accountability. These two approaches to growth impart different levels of stringency Norm-referenced Is the group of students growing better/worse (on average) than statistical expectation? Criterion-referenced Is each student in the group growing at a sufficient rate (i.e., targeted growth) to reach/maintain a desired level of achievement (e.g. Pass or Pass +)? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Norm-referenced student growth/accountability Using student growth percentiles, the median SGP for a group describes whether students in the group, on “average”, demonstrated growth above or below expectation. This is essentially, what value-added models quantify. Value-added models go an extra step an implicitly infer causality by labeling the aggregate measure as a school or teacher effect. Some value-added models include student level characteristics (e.g., demographics) and thus set differential expectations. Student growth percentiles do not include student level characteristics in the model. Any adjustments needed for accountability purposes can occur post-hoc. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Criterion-referenced student growth/accountability Former Secretary of Education Margaret Spellings initiated the Growth Model Pilot Program allowing states to include growth as a means of satisfying AYP. The “Bright Line Principle” of all students being proficient by 2014 remained. 15 states were accepted as part of the pilot. None saw a significant improvement in their AYP numbers because of the growth-to-proficiency goals. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Criterion-referenced student growth/accountability The reality in all state education systems I have worked with is that students who start behind tend to stay behind (Minimal or Basic) and students who start ahead tend to stay ahead (Proficient or Advanced). Targeted growth (criterion-referenced growth) reflects this unfortunate reality: Schools serving the lower achieving students will likely have low percentages of students reaching their growth targets and schools serving high achieving students will likely have high percentages of students reaching their growth targets. Alone, criterion-referenced growth (i.e, growth to proficient) provides an impoverished view of student growth. This is not to say that criterion-referenced growth is unimportant. It must be balanced with a norm-referenced view of student progress. The SGP methodology’s norm and criterion-referenced components attempt to strike that balance allowing stakeholders to understand: What is? , What should be? and What is Reasonable? simultaneously. Damian W. Betebenner Growth Models and Accountability

Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Overview Different Models/Different Questions Growth Models References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Van Iwaarden, A., Domingue, B., and Shang, Y. SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.2-2.0. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Growth Models and Accountability

About this copy

Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability — Mississippi Department of Education, June 30, 2014.

This is an image-based copy of the original beamer slides; builds and animations are flattened.