An Introduction to Student Growth Percentiles and the SGP Package

Albuquerque · November 12, 2013

An Introduction to Student Growth Percentiles and the SGP Package — Albuquerque, November 12, 2013.
Published

November 12, 2013

Event
Invited talk
Slides
37
Format
Converted from Beamer

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. The Big Picture Student Growth Percentiles and Accountability Future Directions References An Introduction to Student Growth Percentiles and the SGP Package Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Albuquerque Public Schools, Albuquerque, New Mexico November 12th, 2013 Damian W. Betebenner An Introduction to SGPs
  2. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth From a Paucity to a Plethora of Data The Economist: The Data Deluge Historical records of student achievement Historical records of student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner An Introduction to SGPs
  3. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner An Introduction to SGPs
  4. 20 core ca ca le le 480 S re 2013 co
  5. 20 core ca ca le le 480 S re 2013 co
  6. 20 core ca ca le le 480 S re 2013 co
  7. co re le ca ro th nt i le 13 ce er th 460 re co 440 le ca
  8. co re th S ro le le ca nt er ce S th 460 re co 440 le ca
  9. Peer 4 5 4 m i re ade Sco Ac cale 201 75% 75 th Per 400 nt i ce le G wth ro 20 core ca ca le le 480 S re 20 13 co
  10. Th is d 75th s: dat erive perc eer 4 5 4 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 400 nt i ce le G wth ro 20 core ca ca le le 480 S re 20 13 co
  11. eer 4 2 6 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 2S eer all g ro ups th Per 400 nt i ce le G wth ro 20 core ca ca le le 480 S re 2013 co
  12. Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 40 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 52 Sc
  13. Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 40 75 er ce 0 nt 0 il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 52 Sc
  14. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner An Introduction to SGPs
  15. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically as rare as the high achiever’s. Equally rare does not mean equally good. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner An Introduction to SGPs
  16. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner An Introduction to SGPs
  17. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner An Introduction to SGPs
  18. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2013] and the SGP package [Betebenner et al., 2013] In working with state departments of education, the goal is to build local capacity so analyses can be performed and validated on their own using software available and easily used by the state. Colorado, Massachusetts, Indiana, Nevada, Arizona, Wisconsin, Idaho, Washington, Hawaii, Georgia, West Virginia, Oregon, are states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner An Introduction to SGPs
  19. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Creative Commons Share-and-Share Alike License: Software is open source and developed on the GitHub collaborative version control system: https://github.com/CenterForAssessment/SGP Source code for analyses of each state are version controlled on GitHub for complete transparency. For example https://github.com/CenterForAssessment/Colorado Meta-data (e.g., names of achievement levels, CSEMs, ...) for state assessment systems is housed within the package and developed/maintained on GitHub: https://github.com/CenterForAssessment/SGPstateData Damian W. Betebenner An Introduction to SGPs
  20. Student Growth Percentile Adoption Adopted Model Considering Adoption Investigating Model
  21. The Data Revolution The Big Picture Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner An Introduction to SGPs
  22. Advanced Reading Achievement Very High DEMO Reading ● Scale Score High Typical Proficient Low Growth Level Percentiles Very Low ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth
  23. Billy Gonzales North End Middle School How to interpret this student growth & achievement report Math Advanced Very High Achievement DEMO High ● DEMO Math ● Typical Scale Score Scale Score ● Low ● Growth Proficient Very Low Level Percentiles DEMO Very High 81st − 99th Achievement High 61st − 80th Levels Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Student Growth Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Percentile Scale Score 438 447 489 495 Achievement Level Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 18 64 44 Growth Level Very Low High Typical Growth Suggested Uses Review past growth to assess student academic progress toward Advanced Reading DEMO achievement goals. Achievement ● Develop remediation or enrich− Very High DEMO Reading ment plans based on rate of ● Scale Score High growth needed to reach higher ● Typical DEMO achievement levels. Proficient Low Growth Level Percentiles ● Identify the rate of progress Very Low needed in order to reach or ● Very High 81st − 99th maintain proficient status High 61st − 80th on the DEMO next year. Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 69 85 74 Growth Level High Very High High Growth For more information please visit the Student Growth Percentiles (SGP) at centerforassessment.github.io/SGP/ or contact dbetebenner@nciea.org Cooperatively developed by the SGP & the Center for Assessment, Inc. Distributed by the SGP.
  24. Advanced Reading ● Move Up Achievement Target DEMO Reading ● Scale Score Keep Up ● Target Proficient Growth Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Keep Up (76)/Move Up (99) Growth
  25. Advanced Reading ● Move Up Achievement Target DEMO Reading ● Scale Score Keep Up ● Target Proficient ● Catch Up ● Target Growth Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (592) Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Catch Up (60) Keep Up (76)/Move Up (99) Growth
  26. Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
  27. Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
  28. Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
  29. The Big Picture Student Growth Percentiles and Accountability School Accountability Future Directions References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner An Introduction to SGPs
  30. DEMONSTRATION School Performance DEMO Growth & Achievement 2012−2013 DEMO Mathematics Higher Achievement Higher Achievement School Size 2012−2013 Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile
  31. DEMONSTRATION School Performance DEMO Growth & Prior Achievement 2012−2013 DEMO Mathematics 2011−2012 Prior Percent at/above Proficient Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile
  32. DEMONSTRATION School Performance DEMO Growth & Achievement 2012−2013 DEMO Reading Higher Achievement Higher Achievement School Size 2012−2013 Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile
  33. DEMONSTRATION School Performance DEMO Growth & Prior Achievement 2012−2013 DEMO Reading 2011−2012 Prior Percent at/above Proficient Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2012−2013 Median Student Growth Percentile
  34. The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking The Colorado Department of Education and the Center for Assessment have been working for the last two years on developing next generation data visualization to accompany growth model data. The goal: Transform conversations about education through active engagement with data (i.e., evidence). Our efforts have received tremendous interest and recognition: Recognized by Adobe for innovative uses of their technology as an Adobe Max Award finalist in October, 2009. 2010 NCME Award for Outstanding Dissemination of Educational Measurement Concepts to the Public. 14 states signing MOUs to co-develop a cloud-based reporting platform in a non-proprietary fashion. Colorado recently devoted $2.5 million of stimulus funds to the development efforts Damian W. Betebenner An Introduction to SGPs
  35. The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References The Decade of Data Visualization Chart Wars http://infosthetics.com/archives/2010/01/ chart_wars_the_political_power_of_data_ visualization.html Hans Rosling/Gapminder http://www.youtube.com/watch?v=jbkSRLYSojo Journalism in the Age of Data http://datajournalism.stanford.edu/ Damian W. Betebenner An Introduction to SGPs
  36. The Big Picture Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner An Introduction to SGPs
  37. The Big Picture Student Growth Percentiles and Accountability Future Directions References References Betebenner, D. W., Van Iwaarden, A., Domingue, B., and Shang, Y. SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.2-0.0. R Development Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner An Introduction to SGPs