Talks archive, 2005–2009 – Student Growth Percentiles: Toward a Normative Understanding of Student Growth

bg=White Overview Normative Student Growth Descriptive/Prescriptive Growth References Student Growth Percentiles: Toward a Normative Understanding of Student Growth Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Growth Models – A Joint Exploration Lawrence, Kansas May 20th, 2008 Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References What Do Emminent Statisticians Recommend John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. George E. P. Box All models are wrong but some are useful. Recommendation: Start at the end What are the questions and uses Kansas would like student growth analyses to address? Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References What are the relevant questions? Different analysis techniques answer (or fail to answer) different, but often related, questions—it is critical to understanding these different questions. Answers to these different questions often lead to different uses. Starting with the right questions simplifies both the development of appropriate analyses and motivating the proper use of the results. However, the “right questions” often vary by stakeholder group. Identifying the questions you will answer sets the stage for subsequent conversations that will take place. Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns for each group. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References What are the relevant questions for parents? Parent Questions: Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References What are the relevant questions for teachers? Teacher Questions: Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References What are the relevant questions for administrators? Administrator Questions: Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References Why Growth for School Accountability? A fundamental premise associated with using student growth for school accountability is that “good” schools bring about student growth in excess of that found at “bad” schools. The inherent believability of this premise is at the heart of current enthusiasm to incorporate growth models into state accountability systems. Students attending such schools—commonly referred to as highly effective/ineffective schools—tend to demonstrate extraordinary growth that is causally attributed to the school or teachers instructing the students. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Relevant Questions Descriptive/Prescriptive Growth Growth Models References Value-added Models/Inferences The primary thrust of growth analyses has been to determine, using sophisticated statistical techniques, the amount of student progress/growth that can be justifiably attributed to the school or teacher. Such analyses, often called value-added analyses, attempt to estimate the teacher/school contribution to student achievement. Value-added models provide effectiveness quantities that purport to quantify the impact on achievement that this school or teacher would have, on average, upon similar students assigned to them. The foremost example being the value-added models developed by Dr. William Sanders and used in many districts and states nationwide to infer teacher or school effectiveness. These models and their effectiveness claims are controversial for a number of reasons [Braun, 2005, Rubin et al., 2004]. Value-added models are better described as effectiveness than as growth models. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Actual versus Aspirational Student Growth Descriptive/Prescriptive Growth Describing Actual Student Growth References Actual versus Aspirational Student Growth It seems simple enough: Establish what constitutes a year’s growth in a year’s time for each student. This simplicity belies a subtle complexity. To adequately address the notion of a year’s growth, actual growth must be distinguished from aspirational growth. Describe What is a current year’s growth? Prescribe What should a current year’s growth be? Answering the second question establishes a threshold distinguishing adequate from inadequate growth. To make such a distinction requires an answer to the first question which defines a norm: What is the range of growth currently observed? Aspirational growth should be possible—This is Linn’s existence proof applied at the individual level [Linn, 2003]. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Actual versus Aspirational Student Growth Descriptive/Prescriptive Growth Describing Actual Student Growth References Describing Actual Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Numerous States believe that placing their tests on a vertical scale will simplify the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. However, the 4 inch increase becomes fully understandable only when placed in context of growth of other children between ages 3 and 4. In the following discussion student growth percentiles are introduced to provide a normative context for interpreting student growth. Damian W. Betebenner Normative Student Growth

600 co 05 600 re S le S ca le ca S re 06 co

bg=White Overview Normative Student Growth Actual versus Aspirational Student Growth Descriptive/Prescriptive Growth Describing Actual Student Growth References Student Growth Percentiles As the figures indicate, there is a distribution of 2006 scale scores associated with each 2005 scale score. The average (or median) of the conditional density could be used as a point estimate of the expected 2006 scale score given the student’s 2005 performance. Similarly, the middle 50% of each density gives a range of scores where one would expect such a student’s score to reside 50% of the time. Using quantile regression techniques, conditional densities using more than a single prior score can be estimated. Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. For example, a student growth percentile of 60 indicates the student’s latest scale score exceeded 60 percent of his/her counterparts with equal prior scores. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Actual versus Aspirational Student Growth Descriptive/Prescriptive Growth Describing Actual Student Growth References Student Growth Percentiles Student growth percentiles norm each student’s growth relative to their academic peers. The growth percentiles are estimated using quantile regression techniques (Betebenner & Shang, 2007; Koenker, 2005). The percentile scale on which the growth percentiles lie allows for easy comparison of all students. For example, consider a low achieving student demonstrating 90th percentile growth and a high achieving student demonstrating 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. In this sense, the low achiever’s growth is more exemplary than the high achiever’s. However, judgment of whether the student’s growth was adequate requires external criteria. Damian W. Betebenner Normative Student Growth

bg=White Overview Normative Student Growth Descriptive/Prescriptive Growth References References Betebenner, D. W. and Shang, Y. (2007). Reference growth charts for educational outcomes. Paper presented at the 2007 AERA Annual Conference, Chicago, IL. Retrieved May 12th, 2007 from http://monticello.bc.edu/Betebenner. Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Technical report, Educational Testing Service, Princeton, New Jersey. Koenker, R. (2005). Quantile Regression. Cambridge University Press, Cambridge. Linn, R. L. (2003). Accountability: Responsibility and reasonable expectations. Technical report, Center for the Study of Evaluation, CRESST, Los Angeles, CA. Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1):103–116. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Normative Student Growth

About this copy

Student Growth Percentiles: Toward a Normative Understanding of Student Growth — Kansas Department of Education, May 2008.

This is an image-based copy of the original PDF slides; builds and animations are flattened.