Talks archive, 2005–2009 – Student Growth with the MCAS An Overview of Student Growth Percentiles

Growth Model Background Student Growth Percentiles and Accountability References Student Growth with the MCAS An Overview of Student Growth Percentiles Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Rocky Hill, Connecticut October 22nd, 2009 Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Questions Note that the questions put forward by stakeholders are primarily descriptive The questions are only peripherally associated with causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Again, starting with descriptive questions, perhaps the place to begin is with description and a model supporting such ends. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner Student Growth with the MCAS

600 co 05 600 re S le S ca le ca S re 06 co

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. Damian W. Betebenner Student Growth with the MCAS

Relevant Questions/Concepts Description, Inference, and Causality Growth Model Background Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles References Model & Software for Student Growth Percentiles Criterion Referenced Growth Combining Normative and Criterion-Referenced Growth Growth adequacy is determined by whether a student’s growth is sufficient to reach/maintain desired achievement levels (e.g., proficiency). Percentile growth projections/trajectories are calculated for each student using the most recent historical MCAS longitudinal student growth analyses. These “growth-to-standard” trajectories indicate what it will take for the student to reach/maintain proficiency and other achievement levels. This approach to quantifying “adequate” growth (done by Colorado) was approved by the USED for use in AYP determinations as part of the Growth Model Pilot Program. Damian W. Betebenner Student Growth with the MCAS

Growth Model Background School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness References Concluding Remarks Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Student Growth with the MCAS

Mountain View School District: 2007 CSAP Math School Results Student Growth versus Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch Percentage of Proficient/Advanced Students in School Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students Median of Student Growth Percentiles in School

Mountain View School District: 2007 CSAP Reading School Results Student Growth versus Student Achievement by Free/Reduced Lunch Percentage School Percent Free/Reduced Lunch Percentage of Proficient/Advanced Students in School Lower Growth Higher Growth ● Less than 20 percent ● 20 to 40 percent Higher Achievement Higher Achievement ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students Lower Growth Higher Growth 500 Students Lower Achievement Lower Achievement 1,000 Students Median of Student Growth Percentiles in School

Growth Model Background School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness References Concluding Remarks Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools bring about student growth in excess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Damian W. Betebenner Student Growth with the MCAS

Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain 1 Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch

Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile ● ● 4th Quintile Median Student Growth Percentile ● 5th Quintile School Size 25 Students ● ● 50 Students 100 Students 250 Students 500 Students Value−Added Estimate from Layered Model

Growth Model Background School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness References Concluding Remarks The Massachusetts Growth Model will publicly release result at the end of this month. Calculating of the quantities is the easy part. Getting people to use and understand them correctly is the hard part. Colorado, Massachusetts, Indiana, Arizona, and other states are investigating a multi-state platform for data visualization and reporting built from a common growth analysis. Damian W. Betebenner Student Growth with the MCAS

Growth Model Background Student Growth Percentiles and Accountability References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Student Growth with the MCAS

About this copy

Student Growth with the MCAS An Overview of Student Growth Percentiles — Northeastern Educational Research Association, October 22, 2009.

This is an image-based copy of the original beamer slides; builds and animations are flattened.