Talks archive, 2015–2019 – The Georgia Model of Student Growth

Introduction The Big Picture Growth & Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References The Georgia Model of Student Growth Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Atlanta, Georgia July 2019 Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Growth & Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Since 2007 have been refining and sharing these techniques with Georgia, Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Growth & Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References Student Growth Percentiles 2007 Began development of the Colorado Growth Model (student growth percentiles/percentile growth trajectories). 2008 Successful application of Colorado Growth Model (criterion-referenced/growth target component) as part of USED Growth Model Pilot Program. 2010 Began working with Georgia Department of Education on the development of the Georgia Model of Student Growth. 2010 Worked with states on incorporating growth into RttT applications and with successful states on their use (Massachusetts, Georgia, New York, Rhode Island). 2011 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2012 Worked with 5 states on co-developing interactive visualizations to help better communicate student growth percentile results. 2013 Extended SGP methodology to work with time-dependent interim assessments. 2007-2019 Working to extend and refine the open-source methodology with the assistance of state partners. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Growth & Accountability The Data Revolution Growth & Accountability: The Colorado Growth Model Lessons Learned Student Growth Percentiles and Accountability References From a Paucity to a Plethora of Data Data Data Everywhere (The Economist, 2010) More and better organized data: mega, giga, tera, peta, exa, zetta, yotta! Historical records of student achievement. Linkage of student achievement to student demographics, teachers, schools, educational programs, . . . Stakeholder interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Growth & Accountability The Data Revolution Growth & Accountability: The Colorado Growth Model Lessons Learned Student Growth Percentiles and Accountability References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judgment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Why Growth? Growth & Accountability Different Models/Different Questions Growth & Accountability: The Colorado Growth Model Relevant Questions/Concepts Student Growth Percentiles and Accountability References What are the relevant questions for policy makers? Policy Maker Questions Can/how should student growth be used in judgments of educator quality? Can/how should student growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for student demographics need to be considered? Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Describing Student Growth Growth & Accountability Student Growth Percentiles Growth & Accountability: The Colorado Growth Model Criterion Referenced Growth Student Growth Percentiles and Accountability References Growth & Accountability: The Colorado Growth Model How much growth & is it enough? Colorado begins conversations about education/educator quality by answering two questions: How much: How much growth did a student make? Is is enough: Is it enough growth to reach or maintain proficiency? Answering these two questions in a credible and clear manner sets the stage for subsequent accountability discussions. Colorado purposely separated the description (what is) from the determination of responsibility (who/what is responsible). This separation promoted greater ownership of results and participation in finding root causes. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Describing Student Growth Growth & Accountability Student Growth Percentiles Growth & Accountability: The Colorado Growth Model Criterion Referenced Growth Student Growth Percentiles and Accountability References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner The Georegia Model of Student Growth

20 core ca ca le le 700 S re 2019 co

co re le ca ro th nt i le 19 ce er th 600 re co 500 le ca

co re th S ro le le ca nt er ce S th 600 re co 500 le ca

Peer 5 7 5 m i re ade Sco Ac cale 201 75% 75 th Per 300 nt i ce le G wth ro 20 core ca ca le le 700 S re 20 19 co

Th is d 75th s: dat erive perc eer 5 7 5 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 300 nt i ce le G wth ro 20 core ca ca le le 700 S re 20 19 co

eer 4 2 0 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 8S eer all g ro ups th Per 300 nt i ce le G wth ro 20 core ca ca le le 700 S re 2019 co

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 30 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 40 Sc

Introduction The Big Picture Describing Student Growth Growth & Accountability Student Growth Percentiles Growth & Accountability: The Colorado Growth Model Criterion Referenced Growth Student Growth Percentiles and Accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Describing Student Growth Growth & Accountability Student Growth Percentiles Growth & Accountability: The Colorado Growth Model Criterion Referenced Growth Student Growth Percentiles and Accountability References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 75th percentile growth and a high achieving student with 25th percentile growth. The low achieving student grew at a rate exceeding 75 percent of their academic peers. The high achieving student grew at a rate exceeding just 25 percent of their academic peers. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Describing Student Growth Growth & Accountability Student Growth Percentiles Growth & Accountability: The Colorado Growth Model Criterion Referenced Growth Student Growth Percentiles and Accountability References Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner The Georegia Model of Student Growth

Mathematics Advanced Achievement ● Catch Up Target DEMO Mathematics Catch Up ●● ● Target ●● Scale Score Growth Proficient ● Level Percentiles Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 438 447 489 495 Achievement Level Achievement Target Proficient Part Proficient Part Proficient Part Proficient Proficient (514) Proficient (547) Achievement Growth Percentile 18 64 44 Growth Level Growth Target Very Low High Typical Catch Up (70) Catch Up (82) Growth

Billy Gonzales North End Middle School How to interpret this student growth & achievement report Mathematics Advanced Achievement ● Catch Target Up DEMO ● DEMO Mathematics ● Catch Up ●● Target ●● Scale Score Scale Score ● ● Growth Proficient ● Level Percentiles DEMO Very High 81st − 99th Achievement High 61st − 80th Part Proficient Levels Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Student Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Growth 2015−2016 2016−2017 2017−2018 2018−2019 2019−2020 Percentile Scale Score 438 447 489 495 Achievement Level Achievement Target Proficient Part Proficient Part Proficient Part Proficient Proficient (514) Proficient (547) Achievement Growth Percentile 18 64 44 Catch Up/Keep Up Growth Level Very Low High Typical Growth Move Up/Stay Up Growth Target Catch Up (70) Catch Up (82) Targets Suggested Uses Advanced Reading Move Up Achievement ● Target DEMO Reading ● Scale Score ● Review past growth to assess ● Keep Up student academic progress toward ● Catch Up ●● Target Proficient Target ●● Growth DEMO achievement goals. ● Level Percentiles ● Develop remediation or enrich− ● Very High 81st − 99th ment plans based on rate of High 61st − 80th growth needed to reach higher DEMO achievement levels. Part Proficient Typical 40th − 60th Low 20th − 39th ● Identify the rate of progress ● Very Low 1st − 19th needed in order to reach or Unsatisfactory maintain proficient status Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 on the DEMO next year. 2015−2016 2016−2017 2017−2018 2018−2019 2019−2020 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (592) Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Catch Up (60) Keep Up (76)/Move Up (99) Growth For more information please visit the Student Growth Percentiles Package (the SGP Package) at www.sgp.io or contact dbetebenner@nciea.org Cooperatively developed by the SGP Package & the Center for Assessment, Inc. Distributed by the SGP Package.

Introduction The Big Picture Growth & Accountability Teacher & School Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner The Georegia Model of Student Growth

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

Introduction The Big Picture Growth & Accountability Teacher & School Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References Student Growth Percentile Model The Student Growth Percentile (SGP) methodology is an open source methodology [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012] The model is a norm and criterion-referenced growth model that allows states to answer How much? and Is it enough?. The software used to calculate and visualize SGPs is free (open source) [Betebenner et al., 2019]. Currently over two dozen states are investigating/utilizing the model. Damian W. Betebenner The Georegia Model of Student Growth

Growth Model Types by State Growth Norm Model Value Added Model Gain Based Model

Introduction The Big Picture Growth & Accountability Teacher & School Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteris- tics. Treating the results as descriptive information and for iden- tification of schools that require more intensive investigation of organizational and instructional process characteristics are po- tentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more in- tensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner The Georegia Model of Student Growth

Introduction The Big Picture Growth & Accountability Growth & Accountability: The Colorado Growth Model Student Growth Percentiles and Accountability References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Iwaarden, A. V., Domingue, B., and Shang, Y. SGP: Student growth percentiles & percentile growth trajectories. Available online complete with source code from https://sgp.io. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. The Economist and Cukier, K. (2010). Data data everywhere. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner The Georegia Model of Student Growth

About this copy

The Georgia Model of Student Growth — Georgia Department of Education, July 25, 2019.

This is an image-based copy of the original beamer slides; builds and animations are flattened.