Talks archive, 2010–2014 – Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability

Overview Different Models/Different Questions Growth Models References Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Albuquerque, New Mexico November 13th, 2013 Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Student Growth Percentiles 2007 Began development of the Colorado Growth Model (student growth percentiles/percentile growth trajectories). 2008 Successful application of Colorado Growth Model (criterion-referenced/growth target component) as part of USED Growth Model Pilot Program. 2010 Worked with states on incorporating growth into RttT applications and with successful states on their use (Massachusetts, Georgia, New York, Rhode Island). 2011 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2012 Worked with 5 states on co-developing interactive visualizations to help better communicate student growth percentile results. 2013 Working to enhance collaboration between states on beneficial uses of growth data. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References About the growth model The Student Growth Percentile (SGP) Methodology is an open source norm- and criterion-referenced growth analysis method. The norm-referenced component of the model is the student growth percentile which indicates the growth a student makes. Student growth percentiles can be cohort (peer) referenced or baseline (non-peer) referenced. The criterion-referenced component of the model is the student growth percentile target which indicates the amount of growth required by the student to reach/maintain desirable achievement goals in a specified time frame. For example, catching-up to proficiency within 3 years. The norm- and criterion-referenced components together allow stakeholders to understand what is, what should be, and what is reasonable simultaneously. The goal of the metric was to assist states to move beyond debates over what “thermometer" is best to discussions of how best to use the “thermometer". Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Built to maximize sharing A number of analytic tools (for doing calculations) and visualization tools have been developed with the goal to promote maximum sharing. Data isn’t valuable! It’s what one does with data that is potentially valuable! The analytic tools to calculate all the quantities associated with student growth percentiles are open source and freely available to all states [Betebenner et al., 2013]. https://github.com/CenterForAssessment The visualizations associated with student growth percentiles are open source for non-commercial use. Rhode Island is the latest: http://rigm.ride.ri.gov/growthmodel/index.htm Conduct training sessions at national conferences to help others calculate these quantities on their own. Build enhanced visualization and reporting tools to make data come alive to stakeholders, allow for stakeholder collaboration and sharing, and lead to better stakeholder engagement with data. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Growth Models References What are the relevant questions for policy makers? Policy Maker Questions Can/how should student growth be used in judgments of educator quality? Can/how should student growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for student demographics need to be considered? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide context for describing and ultimately understanding student growth. Damian W. Betebenner Growth Models and Accountability

60 co 12 60 S re S ca 40 ca le le S 80 13 S core 20 20

co re th ro le i le ca nt er ce S th re 60 co le ca

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Growth Models and Accountability

Advanced Reading Achievement Very High DEMO Reading ● Scale Score High Typical Proficient Low Growth Level Percentiles Very Low ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth

Billy Gonzales North End Middle School How to interpret this student growth & achievement report Math Advanced Very High Achievement DEMO High ● DEMO Math ● Typical Scale Score Scale Score ● Low ● Growth Proficient Very Low Level Percentiles DEMO Very High 81st − 99th Achievement High 61st − 80th Levels Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Student Growth Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Percentile Scale Score 438 447 489 495 Achievement Level Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 18 64 44 Growth Level Very Low High Typical Growth Suggested Uses Review past growth to assess student academic progress toward Advanced Reading DEMO achievement goals. Achievement ● Develop remediation or enrich− Very High DEMO Reading ment plans based on rate of ● Scale Score High growth needed to reach higher ● Typical DEMO achievement levels. Proficient Low Growth Level Percentiles ● Identify the rate of progress Very Low needed in order to reach or ● Very High 81st − 99th maintain proficient status High 61st − 80th on the DEMO next year. Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth For more information please visit the Student Growth Percentiles (SGP) at centerforassessment.github.io/SGP/ or contact dbetebenner@nciea.org Cooperatively developed by the SGP & the Center for Assessment, Inc. Distributed by the SGP.

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Two Flavors of Growth based Accountability Our work in the last few years investigating growth/value-added models has led us to believe that there are two general types of growth based accountability Norm-referenced growth based accountability. Criterion-referenced growth based accountability. These two approaches to growth impart different levels of stringency Norm-referenced Is the group of students growing better/worse (on average) than statistical expectation? Criterion-referenced Is each student in the group growing at a sufficient rate (i.e., targeted growth) to reach/maintain a desired level of achievement (e.g. Pass or Pass +)? Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Norm-referenced student growth/accountability Using student growth percentiles, the median SGP for a group describes whether students in the group, on “average”, demonstrated growth above or below expectation. This is essentially, what value-added models quantify. Value-added models go an extra step an implicitly infer causality by labeling the aggregate measure as a school or teacher effect. Some value-added models include student level characteristics (e.g., demographics) and thus set differential expectations. Student growth percentiles do not include student level characteristics in the model. Any adjustments needed for accountability purposes can occur post-hoc. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Criterion-referenced student growth/accountability Former Secretary of Education Margaret Spellings initiated the Growth Model Pilot Program allowing states to include growth as a means of satisfying AYP. The “Bright Line Principle” of all students being proficient by 2014 remained. 15 states were accepted as part of the pilot. None saw a significant improvement in their AYP numbers because of the growth-to-proficiency goals. Damian W. Betebenner Growth Models and Accountability

Overview Different Models/Different Questions Norm-referenced student growth/accountability Growth Models Criterion-referenced student growth/accountability References Criterion-referenced student growth/accountability The reality in all state education systems I have worked with is that students who start behind tend to stay behind (Did Not Pass) and students who start ahead tend to stay ahead (Pass or Pass +). Targeted growth reflects this unfortunate reality: Schools serving the lower achieving students will likely have low percentages of students reaching targeted growth and schools serving high achieving students will likely have high percentages of students reaching targeted growth. Along, growth to proficient provides an impoverished view of student growth. This is not to say that targeted growth is unimportant. I must be balanced with a norm-referenced view of student progress. The SGP methodology’s norm and criterion-referenced components attempt to strike that balance allowing stakeholders to understand: What is? , What should be? and What is Reasonable? Damian W. Betebenner Growth Models and Accountability

Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Overview Different Models/Different Questions Growth Models References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Van Iwaarden, A., Domingue, B., and Shang, Y. SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.2-0.0. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Growth Models and Accountability

About this copy

Growth Models, Student Growth Percentiles, Student Growth Projections/Targets and Accountability — Albuquerque, November 13, 2013.

This is an image-based copy of the original beamer slides; builds and animations are flattened.