Talks archive, 2010–2014 – An Introduction to Student Growth Percentiles

Intro Student Growth Percentiles and Accountability References An Introduction to Student Growth Percentiles Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH University of Massachusetts, Amherst Amherst, Massachusetts March 9th, 2012 Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth About Me Damian Betebenner - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Nevada and approximately 25 other states in various stages of investigation/adoption. Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a norm-referenced description of student growth (i.e., a growth norm). Damian W. Betebenner An Introduction to Student Growth Percentiles

20 core ca ca le le 280 S re 2012 co

co th 260 re ro i le le nt ca er ce th 260 re co 240 le ca

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically equivalent to the high achiever’s. Equally rare does not imply equally good! Judgments about the adequacy of student growth require external criteria. That is, criterion referencing. Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2011] and the SGP package [Betebenner and VanIwaarden, 2012] In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Colorado, Massachusetts, Indiana, Nevada, Arizona, Wisconsin are states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner An Introduction to Student Growth Percentiles

Describing Student Growth Intro Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner An Introduction to Student Growth Percentiles

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

Example Student 12 (1234567812) Sample Middle School Advanced Math Achievement High CSAP Math ● Scale Score Typical ● Growth Proficient ● Level Percentiles Low High 66th − 99th ● Typical 35th − 65th Part Proficient Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 379 388 444 462 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 16 69 51 Growth Level Low High Typical Growth Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth Advanced Writing Achievement High CSAP Writing ● Scale Score Typical ● Growth Proficient Level Percentiles ● Low High 66th − 99th Part Proficient ● Typical 35th − 65th Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 425 449 469 504 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 55 53 79 Growth Level Typical Typical High Growth Copyright 2009, Colorado Department of Education/Center for Assessment, Inc.

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

West Virginia: 2010−2011 Mathematics Norm & Criterion Referenced Growth to College Readiness Percentile Growth Trajectory to College Readiness College Ready Scale Score Not College Ready 2010−2011 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11

Hawaii: 2011 Mathematics Norm & Criterion Referenced Growth & Achievement 2011 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Massachusetts: 2011 ELA Norm & Criterion Referenced Growth & Achievement Baseline Referenced Percentile Growth Trajectory 2011 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

Intro School Accountability Student Growth Percentiles and Accountability Concluding Remarks References Going from Students to Schools/Classrooms It’s of interest to examine schools or classrooms/teachers where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner An Introduction to Student Growth Percentiles

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

Intro School Accountability Student Growth Percentiles and Accountability Concluding Remarks References What’s Next? Many states have publicly released growth results. The response has been overwhelmingly positive. Calculating of the quantities is the easy part. Getting people to understand them and use them correctly is the hard part. Colorado, Massachusetts, Indiana, Virginia, and other states are investigating a multi-state platform for data visualization and reporting built from a common growth analysis. Damian W. Betebenner An Introduction to Student Growth Percentiles

Student Growth Percentile Adoption Adopted Model Considering Adoption Investigating Model

Intro Student Growth Percentiles and Accountability References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W. and VanIwaarden, A. (2012). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles. R package version 0.8-0.0. R Development Core Team (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner An Introduction to Student Growth Percentiles

About this copy

An Introduction to Student Growth Percentiles — UMASS Amherst, March 9, 2012.

This is an image-based copy of the original beamer slides; builds and animations are flattened.