Talks archive, 2005–2009 – Student Growth with the MCAS An Overview of Student Growth Percentiles

Overview Growth Model Background Student Growth Percentiles, Accountability and Program Evaluation References Student Growth with the MCAS An Overview of Student Growth Percentiles Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Malden, Massachusetts June 9th, 2009 Damian W. Betebenner Student Growth with the MCAS

Overview Growth Model Background About Me Student Growth Percentiles, Accountability and Program Evaluation References Relevant Topics What questions does the Massachusetts Growth Model address? What are student growth percentiles and percentile growth projections/trajectories?, What role do the results from the Massachusetts Growth Model play in program evaluations like those associated with ELT. Damian W. Betebenner Student Growth with the MCAS

Overview Growth Model Background About Me Student Growth Percentiles, Accountability and Program Evaluation References About Me Damian Betebenner Senior Associate Center at Center for Assessment. Developed student growth percentiles and percentile growth trajectories[Betebenner, 2008] to help states discuss student growth. Currently refining and sharing these techniques with other states including Massachusetts, Colorado, Nevada, Indiana, South Carolina, Arizona, Rhode Island. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Questions Note that the questions put forward by stakeholders are primarily descriptive The questions are only peripherally associated with causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Again, starting with descriptive questions, perhaps the place to begin is with description and a model supporting such ends. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner Student Growth with the MCAS

600 co 05 600 re S le S ca le ca S re 06 co

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Overview Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles, Accountability and Program Evaluation Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Combining Normative and Criterion-Referenced Growth Growth adequacy is determined by whether a student’s growth is sufficient to reach/maintain desired achievement levels (e.g., proficiency). Percentile growth projections/trajectories are calculated for each student using the most recent historical MCAS longitudinal student growth analyses. These “growth-to-standard” trajectories indicate what it will take for the student to reach/maintain proficiency and other achievement levels. This approach to quantifying “adequate” growth (done by Colorado) was approved by the USED for use in AYP determinations as part of the Growth Model Pilot Program. Damian W. Betebenner Student Growth with the MCAS

Advanced Reading High Achievement CSAP Reading ● Typical ● Scale Score Proficient Low Growth ● ● Level Percentiles ● High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 5 Grade 6 Grade 7 Grade 8 Next Year Scale Score 589 617 617 657 Achievement Level Proficient Proficient Part Proficient Proficient Achievement Growth Percentile 64 36 83 Growth Level Typical Typical High Growth

Overview Growth Model Background School Accountability Student Growth Percentiles, Accountability and Program Evaluation Growth & Effectiveness References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Student Growth with the MCAS

Boston: 2008 MCAS ELA School Results Student Growth versus Current Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch 2008 Percentage at/above Proficient (Current Achievement) Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent 70 ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students 2008 Median of Student Growth Percentiles in School

Boston: 2008 MCAS Math School Results Student Growth versus Current Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch 2008 Percentage at/above Proficient (Current Achievement) Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent 70 ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students 2008 Median of Student Growth Percentiles in School

Overview Growth Model Background School Accountability Student Growth Percentiles, Accountability and Program Evaluation Growth & Effectiveness References Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools bring about student growth in excess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Damian W. Betebenner Student Growth with the MCAS

Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain 1 Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch

2008 CSAP Reading: School Median Student Growth Percentile versus School Percent Free/Reduced Lunch School Size 50 Students 100 Students Median of Student Growth Percentiles in School 200 Students 70 500 Students 1,000 Students Percent Free/Reduced Eligible Students in School

2008 CSAP Math: School Median Student Growth Percentile versus School Percent Free/Reduced Lunch School Size 50 Students 100 Students Median of Student Growth Percentiles in School 200 Students 70 500 Students 1,000 Students Percent Free/Reduced Eligible Students in School

Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile 4th Quintile Median Student Growth Percentile 70 ●●●● 5th Quintile ●●●● School Size 25 Students ● 50 Students 100 Students ●● ● 250 Students ● ● 500 Students Value−Added Estimate from Layered Model

Overview Growth Model Background Student Growth Percentiles, Accountability and Program Evaluation References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Student Growth with the MCAS

About this copy

Student Growth with the MCAS An Overview of Student Growth Percentiles — Massachusetts Department of Education, June 9, 2009.

This is an image-based copy of the original beamer slides; builds and animations are flattened.