Talks archive, 2010–2014 – Student Growth Percentiles: An Overview

Overview SGP Student Growth Percentiles and Accountability References Student Growth Percentiles: An Overview Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Jackson, Mississippi February 28, 2013 Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP About me Student Growth Percentiles and Accountability References About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Nevada, Hawaii and 25 other states in various stages of investigation/adoption. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP About me Student Growth Percentiles and Accountability References Student Growth Percentiles 2007 Began develop of the Colorado Growth Model (student growth percentiles). 2008 Successful application of Colorado Growth Model (criterion-referenced component) as part of growth model pilot program. 2009-2010 Began/continued work with approximately two-dozen other states on developing state growth models. 2010 Worked with states on incorporating growth into RttT applications and with successful states on their use (Massachusetts, Georgia, New York, Rhode Island). 2011 Worked with a number of states on incorporating student growth percentiles into the ESEA waiver applications and teacher evaluation systems. 2012 Worked with 5 states on co-developing interactive visualizations to help better communicate student growth percentile results. Current 18 states have signed an MOU to participate in visualization work and share resources they develop around the growth model. 2013 Working to enhance collaboration between states on beneficial uses of growth data. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP About me Student Growth Percentiles and Accountability References About the growth model The Student Growth Percentiles Methodology is a norm- and criterion-referenced growth analysis method. The norm-referenced component of the model is the student growth percentile which indicates the relative growth a student makes. The criterion-referenced component of the model is the student growth percentile target which indicates the amount of growth required by the student to reach/maintain desirable achievement goals in a specified time frame. For example, catching-up to proficiency within 3 years. The norm- and criterion-referenced components together allow stakeholders to understand what is, what should be, and what is reasonable simultaneously. The goal of the metric was to assist states to move beyond debates over what thermometer is best to discussions of how best to use the thermometer. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP About me Student Growth Percentiles and Accountability References Built to maximize sharing A number of analytic tools (for doing calculations) and visualization tools have been developed with the goal to promote maximum sharing. Data isn’t valuable! It’s what one does with data that is potentially valuable! The analytic tools to calculate all the quantities associated with student growth percentiles are open source and freely available to all states [Betebenner et al., 2013]. https://github.com/SchoolView The visualizations associated with student growth percentiles are open source for non-commercial use. Rhode Island is the latest: http://rigm.ride.ri.gov/growthmodel/index.htm Conduct training sessions at national conferences to help other calculate these quantities (like Mississippi) on their own. Build enhanced visualization and reporting systems to make data come alive to stakeholders, allow for stakeholder collaboration and sharing, and lead to better stakeholder engagement with data. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP About me Student Growth Percentiles and Accountability References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judg- ment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner Student Growth Percentiles: An Overview

20 core ca ca le le 200 S re 2012 co

co re le ca ro th nt i le 12 ce er th 175 re co 150 le ca

co re th S ro le le ca nt er ce S th 175 re co 150 le ca

Peer 1 6 5 m i re ade Sco Ac cale 201 75% 75 th Per 100 nt i ce le G wth ro 20 core ca ca le le 200 S re 20 12 co

Th is d 75th s: dat erive perc eer 1 6 5 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 100 nt i ce le G wth ro 20 core ca ca le le 200 S re 20 12 co

eer 1 3 0 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 1S eer all g ro ups th Per 100 nt i ce le G wth ro 20 core ca ca le le 200 S re 2012 co

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 10 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 5 5 al or 12 Sc

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 10 75 er ce 0 nt 0 il e Gr ow th Sc 0 or al Sc Sc 5 5 al or 12 Sc

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Student Growth Percentiles: An Overview

Different Models/Different Questions Overview It’s all about conversations starting with questions SGP Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner Student Growth Percentiles: An Overview

Matthew Jones Sample School How to interpret this student growth & achievement report Advanced Math Achievement MCT2 ● MCT2 Math ● Very High Scale Score Scale Score Proficient ● ● High Moderate Growth Low Moderate ● Level Percentiles MCT2 Basic Very Low Very High 75th − 99th Achievement ● High Moderate 50th − 74th Levels Low Moderate 25th − 49th Minimal Very Low 1st − 24th Student Growth Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Percentile Scale Score 151 137 148 151 Achievement Level Proficient Minimal Basic Proficient Achievement Growth Percentile 2 76 71 Growth Level Very Low Very High High Moderate Growth Suggested Uses Review past growth to assess student academic progress toward Advanced Reading MCT2 achievement goals. Achievement ● Develop remediation or enrich− MCT2 Reading ment plans based on rate of ● Scale Score growth needed to reach higher Proficient MCT2 achievement levels. Growth Very High Level Percentiles ● Identify the rate of progress ● ● Basic ● High Moderate needed in order to reach or Very High 75th − 99th Low Moderate maintain proficient status High Moderate 50th − 74th on the MCT2 next year. Very Low ● Low Moderate 25th − 49th Minimal Very Low 1st − 24th Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Scale Score 146 143 145 114 Achievement Level Basic Basic Basic Minimal Achievement Growth Percentile 36 47 1 Growth Level Low Moderate Low Moderate Very Low Growth For more information please visit the Mississippi Department of Education (MDE) at www.mde.k12.ms.us or contact 601−359−3513 Cooperatively developed by the MDE & the Center for Assessment, Inc. Distributed by the MDE.

Colorado: 2012 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Overview SGP School Accountability Student Growth Percentiles and Accountability References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP School Accountability Student Growth Percentiles and Accountability References subsectionModel & Software for Student Growth Percentiles Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP School Accountability Student Growth Percentiles and Accountability References Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP School Accountability Student Growth Percentiles and Accountability References Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP Student Growth Percentiles and Accountability References Descriptive Accountability “This is the difference between a retrospective question of identifying fault as opposed to a prospective strategy to engineer some corrective measure, almost independent of considering whether there was blame-worthiness. And to move away from the blame-worthiness paradigm toward something that is more regulatory in nature where one might seize upon disparities or circumstances that are for some reason deemed unacceptable and engineer the interventions needed to bring about the necessary change. . . . It’s the no-fault gap closing strategy in which the effort is to build a consensus about a vision of an improved society rather than figure out where’s the person we want to pillory.” C. Edley (2006) Damian W. Betebenner Student Growth Percentiles: An Overview

Overview SGP Student Growth Percentiles and Accountability References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Van Iwaarden, A., and Domingue, B. (2013). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.0-4.0. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Student Growth Percentiles: An Overview

About this copy

Student Growth Percentiles: An Overview — Mississippi Department of Education, February 28, 2013.

This is an image-based copy of the original beamer slides; builds and animations are flattened.