Talks archive, 2015–2019 – An Introduction to Student Growth Percentiles: RLI UK

The Big Picture Student Growth Percentiles and Accountability Future Directions References An Introduction to Student Growth Percentiles Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH RLI UK January 15th, 2015 Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth From a Paucity to a Plethora of Data The Economist: The Data Deluge Historical records of student achievement Historical records of student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a developmental scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner An Introduction to SGPs

20 core ca ca le le 800 S re 2014 co

co re le ca ro th nt i le 14 ce er th 700 re co 600 le ca

co re th S ro le le ca nt er ce S th 700 re co 600 le ca

Peer 6 6 0 m i re ade Sco Ac cale 201 75% 75 th Per 400 nt i ce le G wth ro 20 core ca ca le le 800 S re 20 14 co

Th is d 75th s: dat erive perc eer 6 6 0 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 400 nt i ce le G wth ro 20 core ca ca le le 800 S re 20 14 co

eer 5 2 0 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 3S eer all g ro ups th Per 400 nt i ce le G wth ro 20 core ca ca le le 800 S re 2014 co

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 40 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 50 Sc

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2015] and the SGP package [Betebenner et al., 2015] In working with state departments of education, the goal is to build local capacity so analyses can be performed and validated on their own using software available and easily used by the state. Colorado, Massachusetts, Indiana, Nevada, Arizona, Wisconsin, Idaho, Washington, Hawaii, Georgia, West Virginia, Oregon, are states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Creative Commons Share-and-Share Alike License: Software is open source and developed on the GitHub collaborative version control system: https://github.com/CenterForAssessment/SGP Source code for analyses of each state are version controlled on GitHub for complete transparency. For example https://github.com/CenterForAssessment/Colorado Meta-data (e.g., names of achievement levels, CSEMs, ...) for state assessment systems is housed within the package and developed/maintained on GitHub: https://github.com/CenterForAssessment/SGPstateData Damian W. Betebenner An Introduction to SGPs

Student Growth Percentile Adoption Adopted Model Considering Adoption Investigating Model

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner An Introduction to SGPs

Advanced Reading Achievement Proficient ● DEMO Reading Scale Score ● Catch Up Target Growth Part Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Typical 40th − 60th Unsatisfactory Low 20th − 39th Very Low 1st − 19th Grade 3 Grade 4 Scale Score 449 Achievement Level Achievement Target Unsatisfactory Proficient (529) Achievement Growth Percentile Growth Level Growth Target Catch Up (64) Growth

Advanced Reading Achievement DEMO Reading ● Scale Score Proficient ● Catch Up Growth Target Level Percentiles Very High 81st − 99th Part Proficient High 61st − 80th ● Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Scale Score 449 533 Achievement Level Achievement Target Unsatisfactory Part Proficient Proficient (562) Achievement Growth Percentile 69 Growth Level Growth Target High Catch Up (76) Growth

Advanced Reading Achievement Proficient ● ● ● Catch Up Target DEMO Reading Scale Score Growth Part Proficient Level Percentiles Very High 81st − 99th High 61st − 80th Typical 40th − 60th Unsatisfactory Low 20th − 39th Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Grade 6 Scale Score 449 533 579 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient (586) Achievement Growth Percentile 69 85 Growth Level Growth Target High Very High Catch Up (71) Growth

Advanced Reading ● Move Up Achievement Target DEMO Reading ● Scale Score Keep Up ● Target Proficient Growth Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Keep Up (76)/Move Up (99) Growth

Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

The Big Picture Student Growth Percentiles and Accountability School Accountability Future Directions References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner An Introduction to SGPs

DEMONSTRATION School Performance DEMO Growth & Achievement 2012−2013 DEMO Mathematics Higher Achievement Higher Achievement School Size 2012−2013 Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile

DEMONSTRATION School Performance DEMO Growth & Prior Achievement 2012−2013 DEMO Mathematics 2011−2012 Prior Percent at/above Proficient Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile

DEMONSTRATION School Performance DEMO Growth & Achievement 2012−2013 DEMO Reading Higher Achievement Higher Achievement School Size 2012−2013 Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile

DEMONSTRATION School Performance DEMO Growth & Prior Achievement 2012−2013 DEMO Reading 2011−2012 Prior Percent at/above Proficient Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile

The Big Picture Student Growth Percentiles and Accountability Future Directions References Web 2.0: The Decade of Data Visualization Chart Wars http://infosthetics.com/archives/2010/01/ chart_wars_the_political_power_of_data_ visualization.html Hans Rosling/Gapminder http://www.youtube.com/watch?v=jbkSRLYSojo Journalism in the Age of Data http://datajournalism.stanford.edu/ Damian W. Betebenner An Introduction to SGPs

The Big Picture Student Growth Percentiles and Accountability Future Directions References Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner An Introduction to SGPs

The Big Picture Student Growth Percentiles and Accountability Future Directions References References Betebenner, D. W., Iwaarden, A. V., Domingue, B., and Shang, Y. SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.3-3.0. R Development Core Team (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner An Introduction to SGPs

About this copy

An Introduction to Student Growth Percentiles: RLI UK — Center for Assessment RLI, January 15, 2015.

This is an image-based copy of the original beamer slides; builds and animations are flattened.