Talks archive, 2015–2019 – An Introduction to Student Growth Percentiles and the SGP Package

The Big Picture Student Growth Percentiles and Accountability Future Directions References An Introduction to Student Growth Percentiles and the SGP Package Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Utah State Board of Education, Salt Lake City, Utah January 10th, 2016 Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth From a Paucity to a Plethora of Data The Economist: The Data Deluge Historical records of student achievement Historical records of student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner An Introduction to SGPs

500 co 22 500 re le ca le ca S re 23 co

co re le th ca ro le 23 nt ce er 20 th re 500 co le ca

co re th ro le i le ca nt er ce S th re 500 co le ca

Peer 5 0 0 m i re ade Sco Ac cale 202 75% 75th er ce nt le G wth ro 500 co 22 500 re S le S ca le S ca S re 20 23 co

Th is d 75th s: dat erive perc eer 5 0 0 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 202 75% 75th er ce nt le G wth ro 500 co 22 500 re S le S ca le S ca S re 20 23 co

eer 3 0 0 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 2S eer all g ro ups 75 th er nt ce le G wth ro 500 co 22 500 re le ca le ca S re 23 co

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 10 r ce 0 nt il e Gr ow th Sc or al 50 0 Sc Sc 30 al or Sc

Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 10 75 er ce 0 nt 0 il e Gr ow th Sc or al 50 0 Sc Sc 30 al or Sc

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2023] and the SGP package [Betebenner et al., 2024] In working with state departments of education, the goal is to build local capacity so analyses can be performed and validated on their own using software available and easily used by the state. Colorado, Massachusetts, Indiana, Nevada, Arizona, Wisconsin, Idaho, Washington, Hawaii, Georgia, West Virginia, Oregon, are states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner An Introduction to SGPs

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Creative Commons Share-and-Share Alike License: Software is open source and developed on the GitHub collaborative version control system: https://github.com/CenterForAssessment/SGP Source code for analyses of each state are version controlled on GitHub for complete transparency. For example https://github.com/CenterForAssessment/Colorado Meta-data (e.g., names of achievement levels, CSEMs, ...) for state assessment systems is housed within the package and developed/maintained on GitHub: https://github.com/CenterForAssessment/SGPstateData Damian W. Betebenner An Introduction to SGPs

Student Growth Percentile Use Extensive Use Some Use Investigating Use

The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner An Introduction to SGPs

Advanced Reading Achievement DEMO Reading ● Scale Score Proficient Growth ● Catch Up Target Catch Up ●● Level Percentiles Target ● Very High 81st − 99th Part Proficient High 61st − 80th ● Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Scale Score 449 533 Achievement Level Achievement Target Unsatisfactory Part Proficient Proficient (538) Proficient (562) Achievement Growth Percentile 69 Growth Level Growth Target High Catch Up (75) Catch Up (76) Growth

Billy Gonzales Wenning Arts Elementary School How to interpret this student growth & achievement report Mathematics Advanced Achievement DEMO ● DEMO Mathematics ● Move Up Scale Score Scale Score ● Target ● Catch Up Target Keep Up ● Growth Target Proficient ● Level Percentiles DEMO Very High 81st − 99th Achievement High 61st − 80th Part Proficient Levels Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Student Grade 3 Grade 4 Grade 5 Growth 2015−2016 2016−2017 2017−2018 Percentile Scale Score 438 447 Achievement Level Achievement Target Proficient Part Proficient Proficient (482)/Advanced (503) Proficient (494) Achievement Growth Percentile 18 Catch Up/Keep Up Growth Level Very Low Growth Move Up/Stay Up Growth Target Keep Up (55)/Move Up (74) Catch Up (60) Targets Suggested Uses Advanced Reading Achievement DEMO Reading ● Scale Score ● Review past growth to assess student academic progress toward Proficient Growth ● Catch Up DEMO achievement goals. Catch Up ●● Target Level Percentiles Target ● ● Develop remediation or enrich− Very High 81st − 99th ment plans based on rate of Part Proficient High 61st − 80th growth needed to reach higher DEMO achievement levels. ● Typical 40th − 60th Low 20th − 39th ● Identify the rate of progress Unsatisfactory Very Low 1st − 19th needed in order to reach or maintain proficient status Grade 3 Grade 4 Grade 5 on the DEMO next year. 2015−2016 2016−2017 2017−2018 Scale Score 449 533 Achievement Level Achievement Target Unsatisfactory Part Proficient Proficient (538) Proficient (562) Achievement Growth Percentile 69 Growth Level Growth Target High Catch Up (75) Catch Up (76) Growth For more information please visit the Student Growth Percentiles Package (the SGP Package) at www.sgp.io or contact dbetebenner@nciea.org Cooperatively developed by the SGP Package & the Center for Assessment, Inc. Distributed by the SGP Package.

Advanced Reading Achievement DEMO Reading ● Scale Score Proficient Growth ● Catch Up Target Catch Up ●● Level Percentiles Target ● Very High 81st − 99th Part Proficient High 61st − 80th ● Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Scale Score 449 533 Achievement Level Achievement Target Unsatisfactory Part Proficient Proficient (538) Proficient (562) Achievement Growth Percentile 69 Growth Level Growth Target High Catch Up (75) Catch Up (76) Growth

Utah: 2017 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2017 Achievement Percentile SGP trajectories for a 3rd grade student starting from the Level 1/Level 2 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 EOCT EOCT EOCT Mathematics Mathematics Mathematics Mathematics Mathematics Mathematics Sec Math I Sec Math II Sec Math III

Utah: 2017 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2017 Achievement Percentile SGP trajectories for a 3rd grade student starting from the Level 2/Level 3 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 EOCT EOCT EOCT Mathematics Mathematics Mathematics Mathematics Mathematics Mathematics Sec Math I Sec Math II Sec Math III

Utah: 2017 ELA Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2017 Achievement Percentile SGP trajectories for a 3rd grade student starting from the Level 1/Level 2 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11

Utah: 2017 ELA Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2017 Achievement Percentile SGP trajectories for a 3rd grade student starting from the Level 2/Level 3 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11

The Big Picture Student Growth Percentiles and Accountability School Accountability Future Directions References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner An Introduction to SGPs

Utah School Performance SAGE Growth & Achievement 2016 SAGE Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2016 Percent at/above Proficient 250 Students 500 Students Achievement Summary Statistics Correlation: 0.46 Correlation (N ≥ 10): 0.46 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

Utah School Performance SAGE Growth & Prior Achievement 2016 SAGE Mathematics Higher Achievement Higher Achievement School Size 2015 Prior Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Summary Statistics Correlation: 0.23 Correlation (N ≥ 10): 0.23 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

Utah School Performance SAGE Growth & Achievement 2016 SAGE ELA Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2016 Percent at/above Proficient 250 Students 500 Students Achievement Summary Statistics Correlation: 0.48 Correlation (N ≥ 10): 0.48 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

Utah School Performance SAGE Growth & Prior Achievement 2016 SAGE ELA Higher Achievement Higher Achievement School Size 2015 Prior Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Summary Statistics Correlation: 0.24 Correlation (N ≥ 10): 0.24 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2016 Median Student Growth Percentile

The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking The Colorado Department of Education and the Center for Assessment have been working for the last two years on developing next generation data visualization to accompany growth model data. The goal: Transform conversations about education through active engagement with data (i.e., evidence). Our efforts have received tremendous interest and recognition: Recognized by Adobe for innovative uses of their technology as an Adobe Max Award finalist in October, 2009. 2010 NCME Award for Outstanding Dissemination of Educational Measurement Concepts to the Public. 14 states signing MOUs to co-develop a cloud-based reporting platform in a non-proprietary fashion. Colorado recently devoted $2.5 million of stimulus funds to the development efforts Damian W. Betebenner An Introduction to SGPs

The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References The Decade of Data Visualization Chart Wars http://infosthetics.com/archives/2010/01/ chart_wars_the_political_power_of_data_ visualization.html Hans Rosling/Gapminder http://www.youtube.com/watch?v=jbkSRLYSojo Journalism in the Age of Data http://datajournalism.stanford.edu/ Damian W. Betebenner An Introduction to SGPs

The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner An Introduction to SGPs

The Big Picture Student Growth Percentiles and Accountability Future Directions References References Betebenner, D. W., Iwaarden, A. V., Domingue, B., and Shang, Y. SGP: Student growth percentiles & percentile growth trajectories. Available online complete with source code from https://sgp.io. R Development Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner An Introduction to SGPs

About this copy

An Introduction to Student Growth Percentiles and the SGP Package — Utah Department of Education, January 10, 2016.

This is an image-based copy of the original beamer slides; builds and animations are flattened.