Talks archive, 2010–2014 – New Directions for Student Growth Models

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References New Directions for Student Growth Models Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH The Neag Graduate Student Association (NGSA) Colloquium October 14th, 2010 Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References About Me Past & Present ???? – 2001 Professional Student. Received a Ph.D. in Mathematics in 1995 from University of Wyoming and a Ph.D. in Education (Research & Evaluation Methodology) University of Colorado, Boulder, 2001. 2001 – 2004 Post-Doc. University of Colorado, Boulder 2004 – 2007 Assistant Professor, Boston College, Department of Education Research Measurement and Evaluation 2007 – ???? Senior Associate, National Center for the Improvement of Educational Assessment Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References About Me Past & Present Current Responsibilities Work with state departments of education and affiliated organizations on the development of student growth models. Developed student growth percentiles and percentile growth trajectories[Betebenner, 2008] to help states discuss student growth. Currently refining and sharing these techniques with other states including Massachusetts, Colorado, Nevada, Indiana, South Carolina, Arizona, Rhode Island Current Interests The rise of the data scientist: Data analysis and data visualization and their intersection with education Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References About Data Science Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References About Today’s Talk On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes They say that genius is an infinite capacity for taking pains. It’s a very bad definition, but it does apply to detective work. On Theories of Action Confucius Tell me and I will forget, show me and I might remember, involve me and I will understand. Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts References The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner New Directions for Student Growth Models

Intro Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Growth & Accountability Fundamental Premise “Good” schools/teachers bring about growth in student achievement in excess of that found at “bad” schools/teachers. The inherent believability of this premise is at the heart of current enthusiasm to incorporate growth models into accountability systems. Note that current NCLB accountability invokes the same premise (with achievement substituted for growth). “Good schools/teachers” are commonly referred to as highly effective schools/teachers. It is critical to understand the distinction between growth and effectiveness. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Value-added Analyses/Inferences Who/What is responsible for student growth? Often, analyses of student growth attempt to determine the amount of student progress/growth that can be attributed to the school or teacher. Called Value-added analyses, these techniques attempt to estimate the teacher/school contribution to student achievement. Value-added models quantify effectiveness—purporting to measure the impact on achievement that this school or teacher would have, on average, upon similar students assigned to them. These models and their effectiveness claims are controversial for a number of reasons (EPI (2010), NRC/NAE (2010), Braun (2005), Rubin and Stuart (2004)) . Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Value-added Analyses/Inferences Who/What is responsible for student growth? Value-added models are better described as effectiveness models than as growth models. Value-added is an inference – a causal conclusion drawn from the data. Value-added isn’t a type of statistical model or type of assessment (if I showed you the statistical model formulation or an assessment, from the model formulation or assessment you wouldn’t be able to tell me whether it’s value-added). All growth models can be used for value-added purposes but not vice-versa. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Descriptive Questions Note that the questions from Yen’s (2007) survey are primarily descriptive. The questions are only peripherally associated with causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Beginning with description doesn’t prevent subsequent discussions about responsibility/cause. Beginning with responsibility/cause often prevents discussions about description. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner New Directions for Student Growth Models

20 core ca ca le le 400 S re 2006 co

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2010] and the SGP package [Betebenner, 2010] In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Colorado, Massachusetts, Indiana, and Wisconsin are four states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. Damian W. Betebenner New Directions for Student Growth Models

Value-Added Intro Description, Inference, and Causality Growth & Accountability: Part I Describing Student Growth Growth & Accountability: Part II Student Growth Percentiles Student Growth Percentiles and Accountability Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner New Directions for Student Growth Models

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

NCIEA  CDE Colorado Department of Education High Math Advanced ● Typical Achievement ● Low CSAP Math ● Scale Score ● ● Growth Level Percentiles Proficient High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 515 517 589 603 Achievement Level Proficient Proficient Advanced Proficient Achievement Growth Percentile 15 92 76 Growth Level Low High High Growth 2008 CSAP Growth & Achievement Report Advanced High Reading Typical Achievement CSAP Reading Low ● Scale Score Proficient Growth Level Percentiles ● High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 567 620 652 657 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 72 89 72 Growth Level High High High Growth Advanced High Writing Achievement ● Typical CSAP Writing ● Scale Score Low ● ● Growth Proficient Level Percentiles Example Student 3020 High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 478 517 519 574 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 77 44 86 Growth Level High Typical High Growth

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Intro Growth & Accountability: Part I School Accountability Growth & Accountability: Part II Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner New Directions for Student Growth Models

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

Intro Growth & Accountability: Part I School Accountability Growth & Accountability: Part II Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References Growth, Effectiveness, and Value-Added “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). RttT has placed teacher responsibility for student progress as a primary issue. In teacher performance evaluations, even when large scale assessment results are available, depending upon the state they constitute, on average, about a third of the overall teacher rating. Damian W. Betebenner New Directions for Student Growth Models

2009 CSAP Elementary School Mathematics: School Poverty by School Achievement (r = −0.71) 80 Higher Achievement Percent at/above Proficient Lower Achievement Percent Free/Reduced Lunch Students at School

2009 CSAP Elementary School Mathematics: School Poverty by School Median SGP (r = −0.22) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School

2009 CSAP Elementary School Reading: School Poverty by School Median SGP (r = −0.25) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School

Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain 1 Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch

Composite VA Gain by % Poverty School % Poverty

Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile 4th Quintile Median Student Growth Percentile 70 ●●●● 5th Quintile ●●●● School Size 25 Students ● 50 Students 100 Students ●● ● 250 Students ● ● 500 Students Value−Added Estimate from Layered Model

Figure 5a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 1st Quintile Median Student Growth Percentile ●● ● School Size ● ●● 25 Students ●● 50 Students ●● ● ● ● ● 100 Students 250 Students 500 Students Value−Added Estimate from Layered Model

Figure 6a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 2nd Quintile Median Student Growth Percentile 60 ●●● School Size ● 25 Students ●● 50 Students ●● ● 100 Students ● 250 Students ● ● ● 500 Students Value−Added Estimate from Layered Model

Figure 7a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 3rd Quintile Median Student Growth Percentile 60 ●●●● School Size ●● 25 Students ● ● ● ● 50 Students ● 100 Students ● ● ●● 250 Students 30 ● ●●● ● 500 Students Value−Added Estimate from Layered Model

Figure 8a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement ●● 4th Quintile Median Student Growth Percentile 60 ● ● ● ●● ● ● School Size 25 Students 50 ● 50 Students 100 Students ●●● 250 Students 30 ● 500 Students Value−Added Estimate from Layered Model

Figure 9a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement Median Student Growth Percentile 5th Quintile ● ● ● School Size ●● ● 25 Students 50 ● ●●●●●●● ● ● 50 Students 100 Students ● ● 250 Students 30 ● ● ● 500 Students Value−Added Estimate from Layered Model

Figure 10a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement Median Student Growth Percentile ● 10th Decile 60 ● ● ● School Size ●● 25 Students 50 ● ● ●● 50 Students 100 Students 250 Students 30 ● 500 Students Value−Added Estimate from Layered Model

Intro Growth & Accountability: Part I School Accountability Growth & Accountability: Part II Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References What’s Next? Colorado, Massachusetts, and Indiana have publicly released growth results. The response has been overwhelmingly positive. Calculating of the quantities is the easy part. Getting people to understand them and use them correctly is the hard part. Colorado, Massachusetts, Indiana, Virginia, and other states are investigating a multi-state platform for data visualization and reporting built from a common growth analysis. Damian W. Betebenner New Directions for Student Growth Models

The Colorado Growth Model Adopted Model Near Model Adoption Interested in Model

Intro Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2010). SGP: Student Growth Percentile and Percentile Growth Projection/Trajectory Functions. R package version 0.0-6. Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Technical report, Educational Testing Service, Princeton, New Jersey. Economic Policy Institute (2010). Problems with the use of student test scores to evaluate teachers. Technical report, Washington, D.C. Downloaded August 30th, 2010 from http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. National Research Council and National Academy of Education Getting value of out value-added: Report of a workshop. Technical report, Washington, D.C. Commitee on Value-Added Methodology for Instructional Improvement. R Development Core Team (2010). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1):103–116. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner New Directions for Student Growth Models

About this copy

New Directions for Student Growth Models — UCONN Colloquium, October 14, 2010.

This is an image-based copy of the original beamer slides; builds and animations are flattened.