Talks archive, 2005–2009 – Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability References Student Growth Percentiles: Theory and Calculation Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH NCME Training Session San Diego, California April 13th, 2009 Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background About Us Describing Student Growth About the R Software Student Growth Percentiles and Accountability About Student Growth Percentiles References About the Training Session Purpose is to learn about student growth percentiles and percentile growth projections/trajectories. Purpose is to learn about growth models and all the jargon: value-added, growth-to-standard, Purpose is to learn how to calculate student growth percentiles and percentile growth projections/trajectories using the R software in conjunction with the SGP package (and to show you just how great R is). Purpose is to learn how student growth percentiles can be used within an accountability system (even one that is NCLB compliant). Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background About Us Describing Student Growth About the R Software Student Growth Percentiles and Accountability About Student Growth Percentiles References About Us Damian Betebenner Senior Associate Center at Center for Assessment. Developed student growth percentiles and percentile growth trajectories[Betebenner, 2008]. Jinnie Choi Doctoral Candidate, UC Berkeley. Intern at the Center summer 2008. Helped develop/write the SGP Package for Marie Huchton Doctoral Candidate, University of Colorado, Boulder. Calculates all growth percentile quantities for the Colorado Department of Education as part of their annual analysis of CSAP data. Hi Shin Shim Doctoral Candidate, Georgia Tech. Intern at the Center summer 2008. Helped develop/write the SGP Package for Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background About Us Describing Student Growth About the R Software Student Growth Percentiles and Accountability About Student Growth Percentiles References About the R Software R is an open source, free, statistical software environment (package/language) that is available for source compilation or in many pre-compiled “flavors”. R even had a recent New York Times article written about it http://www.nytimes.com/2009/01/07/technology/ business-computing/07program.html. R can be downloaded from http://cran.r-project.org/ The philosophy behind R/S (John Chambers): To Turn Ideas into Software Quickly and Faithfully I wrote the SGP (Student Growth Percentile) package for R to make it “easy” for people to do growth analyses on large scale assessment data. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background About Us Describing Student Growth About the R Software Student Growth Percentiles and Accountability About Student Growth Percentiles References About Student Growth Percentiles A model/method for describing student achievement change (i.e., growth) normatively. Fully adopted for growth models in Colorado & Massachusetts. Being investigated in Indiana, South Carolina, Rhode Island, and Nevada. Colorado had their student growth percentile based accountability model recently approved for use with AYP as part of the Growth Model Pilot Program. Model allows users to easily see the interplay between normative and standards based interpretations of growth Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model What are the relevant questions? Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Questions form the vocabulary of later conversations Starting with the right questions simplifies development and motivating the proper use of the growth model results. It’s all about the conversations you want to have. The questions set the table for those conversations. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model Descriptive Questions The questions put forward by stakeholders are primarily descriptive. The questions are only marginally associated causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Again, starting with descriptive questions, perhaps the place to begin is with description and a model supporting such ends. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools bring about student growth in excess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Yen’s questions make no mention of effectiveness. Yen’s questions indicate stakeholder interest in a description of student growth. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session The Big Picture Growth Model Background Relevant Questions/Concepts Describing Student Growth Description, Inference, and Causality Student Growth Percentiles and Accountability Growth & Effectiveness References The Colorado Growth Model The Colorado Growth Model How much growth did my student make? Is it enough? An important first step in the development of the Colorado Growth Model was to separate the description of growth from the assignment of responsibility (i.e., accountability). Incorporating growth into accountability follows from the viable description of growth. A viable description of growth facilitates stakeholder engagement and investigations of responsibility for good/bad growth. This leads to greater stakeholder support for accountability. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background Student Growth Percentiles Describing Student Growth Model for Student Growth Percentiles Student Growth Percentiles and Accountability Criterion Referenced Growth References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

600 co 05 600 re S le S ca le ca S re 06 co

Overview of Training Session Growth Model Background Student Growth Percentiles Describing Student Growth Model for Student Growth Percentiles Student Growth Percentiles and Accountability Criterion Referenced Growth References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background Student Growth Percentiles Describing Student Growth Model for Student Growth Percentiles Student Growth Percentiles and Accountability Criterion Referenced Growth References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background Student Growth Percentiles Describing Student Growth Model for Student Growth Percentiles Student Growth Percentiles and Accountability Criterion Referenced Growth References Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. Criticism of a descriptive approach is not done in a general methodological sense but in terms of utility and craftsmanship[Berk, 2003] Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Overview of Training Session Growth Model Background Student Growth Percentiles Describing Student Growth Model for Student Growth Percentiles Student Growth Percentiles and Accountability Criterion Referenced Growth References Combining Normative and Criterion-Referenced Growth Growth adequacy is determined by whether a student’s growth is sufficient to reach/maintain desired achievement levels (e.g., proficiency). Percentile growth projections/trajectories are calculated for each student. These “growth-to-standard” trajectories indicate what it will take for the student to reach/maintain proficiency and other achievement levels. The Colorado Growth Model was recently approved as part of the Growth Model Pilot Program using this approach. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Advanced Math Achievement High CSAP Math ● Scale Score ● Typical ● Growth Proficient Level Percentiles Low High 66th − 99th Part Proficient ● Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 5 Grade 6 Grade 7 Grade 8 Next Year Scale Score 446 468 549 524 Achievement Level Part Proficient Part Proficient Part Proficient Proficient Achievement Growth Percentile 52 96 25 Growth Level Typical High Low Growth

Advanced Reading High Achievement CSAP Reading ● Typical ● Scale Score Proficient Low Growth ● ● Level Percentiles ● High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 5 Grade 6 Grade 7 Grade 8 Next Year Scale Score 589 617 617 657 Achievement Level Proficient Proficient Part Proficient Proficient Achievement Growth Percentile 64 36 83 Growth Level Typical Typical High Growth

Advanced Writing Achievement High CSAP Writing ● Scale Score ● Typical Proficient ● Growth Level Percentiles ● Low High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 5 Grade 6 Grade 7 Grade 8 Next Year Scale Score 495 530 547 524 Achievement Level Part Proficient Proficient Proficient Part Proficient Achievement Growth Percentile 81 50 16 Growth Level High Typical Low Growth

Overview of Training Session Growth Model Background School Accountability Describing Student Growth Systemic Goals Student Growth Percentiles and Accountability References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

Mountain View School District: 2007 CSAP Math School Results Student Growth versus Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch Percentage of Proficient/Advanced Students in School Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students Median of Student Growth Percentiles in School

Mountain View School District: 2007 CSAP Reading School Results Student Growth versus Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch Percentage of Proficient/Advanced Students in School Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students Median of Student Growth Percentiles in School

Mountain View School District: 2007 CSAP Writing School Results Student Growth versus Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch Percentage of Proficient/Advanced Students in School Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students Median of Student Growth Percentiles in School

Overview of Training Session Growth Model Background School Accountability Describing Student Growth Systemic Goals Student Growth Percentiles and Accountability References What Level of Growth is “Good Enough” Normative comparisons of schools or other groups will always be lacking because they fail to describe “good enough”. If the goal is for today’s exemplary growth to become tomorrow’s typical growth, just how much system improvement do we hoping to see. The student growth percentile and percentile growth projection/trajectory information can be combined to provide diagnostic information about the entire system. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Overview of Training Session Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Student Growth Percentiles: Theory and Calculation

About this copy

Student Growth Percentiles: Theory and Calculation — NCME Annual Meeting, April 13, 2009.

This is an image-based copy of the original PDF slides; builds and animations are flattened.