Talks archive, 2005–2009 – New Directions for Student Growth Models

Overview Growth Model Background Student Growth Percentiles and Accountability References New Directions for Student Growth Models Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH The Neag Graduate Student Association (NGSA) Colloquium October 14th, 2010 Damian W. Betebenner New Directions for Student Growth Models

Overview Growth Model Background About Me Student Growth Percentiles and Accountability References About Me Past & Present ???? – 2001 Professional Student. Received a Ph.D. in Mathematics in 1995 from University of Wyoming and a Ph.D. in Education (Research & Evaluation Methodology) University of Colorado, Boulder, 2001. 2001 – 2004 Post-Doc. University of Colorado, Boulder 2004 – 2007 Assistant Professor, Boston College, Department of Education Research Measurement and Evaluation 2007 – ???? Senior Associate, National Center for the Improvement of Educational Assessment Damian W. Betebenner New Directions for Student Growth Models

Overview Growth Model Background About Me Student Growth Percentiles and Accountability References About Me Past & Present Current Responsibilities Work with state departments of education and affiliated organizations on the development of student growth models. Developed student growth percentiles and percentile growth trajectories[Betebenner, 2008] to help states discuss student growth. Currently refining and sharing these techniques with other states including Massachusetts, Colorado, Nevada, Indiana, South Carolina, Arizona, Rhode Island Current Interests The rise of the data scientist: Data analysis and data visualization and their intersection with education Damian W. Betebenner New Directions for Student Growth Models

Overview Growth Model Background About Me Student Growth Percentiles and Accountability References About Data Science Damian W. Betebenner New Directions for Student Growth Models

Overview Growth Model Background About Me Student Growth Percentiles and Accountability References About Today’s Discussion Growth Models: What’s the big deal? Student growth percentiles and percentile growth projections/trajectories A growth model isn’t an end, it’s just a means embedded within a larger theory of action (e.g., high-stakes accountability) Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Questions Note that the questions put forward by stakeholders are primarily descriptive The questions are only peripherally associated with causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Again, starting with descriptive questions, perhaps the place to begin is with description and a model supporting such ends. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner New Directions for Student Growth Models

20 core ca ca le le 400 S re 2006 co

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. Damian W. Betebenner New Directions for Student Growth Models

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Combining Normative and Criterion-Referenced Growth Growth adequacy is determined by whether a student’s growth is sufficient to reach/maintain desired achievement levels (e.g., proficiency). Percentile growth projections/trajectories are calculated for each student using the most recent historical MCAS longitudinal student growth analyses. These “growth-to-standard” trajectories indicate what it will take for the student to reach/maintain proficiency and other achievement levels. This approach to quantifying “adequate” growth (done by Colorado) was approved by the USED for use in AYP determinations as part of the Growth Model Pilot Program. Damian W. Betebenner New Directions for Student Growth Models

Overview School Accountability Growth Model Background Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner New Directions for Student Growth Models

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

Overview School Accountability Growth Model Background Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools bring about student growth in excess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Damian W. Betebenner New Directions for Student Growth Models

2009 CSAP Elementary School Mathematics: School Poverty by School Achievement (r = −0.71) 80 Higher Achievement Percent at/above Proficient Lower Achievement Percent Free/Reduced Lunch Students at School

2009 CSAP Elementary School Mathematics: School Poverty by School Median SGP (r = −0.22) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School

2009 CSAP Elementary School Reading: School Poverty by School Median SGP (r = −0.25) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School

Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain 1 Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch

Composite VA Gain by % Poverty School % Poverty

Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile 4th Quintile Median Student Growth Percentile 70 ●●●● 5th Quintile ●●●● School Size 25 Students ● 50 Students 100 Students ●● ● 250 Students ● ● 500 Students Value−Added Estimate from Layered Model

Figure 5a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 1st Quintile Median Student Growth Percentile ●● ● School Size ● ●● 25 Students ●● 50 Students ●● ● ● ● ● 100 Students 250 Students 500 Students Value−Added Estimate from Layered Model

Figure 6a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 2nd Quintile Median Student Growth Percentile 60 ●●● School Size ● 25 Students ●● 50 Students ●● ● 100 Students ● 250 Students ● ● ● 500 Students Value−Added Estimate from Layered Model

Figure 7a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement 3rd Quintile Median Student Growth Percentile 60 ●●●● School Size ●● 25 Students ● ● ● ● 50 Students ● 100 Students ● ● ●● 250 Students 30 ● ●●● ● 500 Students Value−Added Estimate from Layered Model

Figure 8a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement ●● 4th Quintile Median Student Growth Percentile 60 ● ● ● ●● ● ● School Size 25 Students 50 ● 50 Students 100 Students ●●● 250 Students 30 ● 500 Students Value−Added Estimate from Layered Model

Figure 9a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement Median Student Growth Percentile 5th Quintile ● ● ● School Size ●● ● 25 Students 50 ● ●●●●●●● ● ● 50 Students 100 Students ● ● 250 Students 30 ● ● ● 500 Students Value−Added Estimate from Layered Model

Figure 10a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size > 50) Prior Achievement Median Student Growth Percentile ● 10th Decile 60 ● ● ● School Size ●● 25 Students 50 ● ● ●● 50 Students 100 Students 250 Students 30 ● 500 Students Value−Added Estimate from Layered Model

Overview School Accountability Growth Model Background Growth & Effectiveness Student Growth Percentiles and Accountability Concluding Remarks References What’s Next? Colorado and Massachusetts have publicly released growth results. The response has been overwhelmingly positive. Calculating of the quantities is the easy part. Getting people to understand them and use them correctly is the hard part. Colorado, Massachusetts, Indiana, Arizona, and other states are investigating a multi-state platform for data visualization and reporting built from a common growth analysis. Damian W. Betebenner New Directions for Student Growth Models

Overview Growth Model Background Student Growth Percentiles and Accountability References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner New Directions for Student Growth Models

About this copy

New Directions for Student Growth Models — UCONN Colloquium, November 16, 2009.

This is an image-based copy of the original beamer slides; builds and animations are flattened.