Talks archive, 2010–2014 – Student Growth Percentiles Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Student Growth Percentiles and Accountability Student Growth Percentiles and Accountability Future Directions References Student Growth Percentiles Norm- and Criterion-Referenced Student Growth Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH WERA Spring Conference Puget Sound, WA March 29, 2012 Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Student Growth Percentiles and Accountability Future Directions References From a Paucity to a Plethora of Data Data Data Everywhere (The Economist, 2010) More and better organized data: mega, giga, tera, peta, exa, zetta, yotta! Historical records of student achievement. Linkage of student achievement to student demographics, teachers, schools, educational programs, . . . Stakeholder interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Student Growth Percentiles and Accountability Future Directions References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judgment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Why Growth? Student Growth Percentiles and Accountability Relevant Questions/Concepts Student Growth Percentiles and Accountability Future Directions References The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Why Growth? Student Growth Percentiles and Accountability Relevant Questions/Concepts Student Growth Percentiles and Accountability Future Directions References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Why Growth? Student Growth Percentiles and Accountability Relevant Questions/Concepts Student Growth Percentiles and Accountability Future Directions References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Why Growth? Student Growth Percentiles and Accountability Relevant Questions/Concepts Student Growth Percentiles and Accountability Future Directions References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Why Growth? Student Growth Percentiles and Accountability Relevant Questions/Concepts Student Growth Percentiles and Accountability Future Directions References What are the relevant questions for policy makers? Policy Maker Questions Can/how should student growth be used in judgments of educator quality? Can/how should student growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for context and student demographics need to be considered? Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Student Growth Percentiles and Accountability Future Directions References Growth & Accountability: SGP/The Colorado Growth Model How much growth & is it enough? Colorado begins conversations about education/educator quality by answering two questions: How much: How much growth did a student make? Is is enough: Is it enough growth to reach or maintain proficiency? Answering these two questions in a credible and clear manner sets the stage for subsequent accountability discussions. Colorado purposely separated the description (what is) from the determination of responsibility (who/what “done it”). This separation promoted greater ownership of results and participation in finding root causes. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Student Growth Percentiles and Accountability Future Directions References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

20 core ca ca le le 600 S re 2012 co

20 core ca ca le le 600 S re 2010 co

co th 500 re ro i le le nt ca er ce th 500 re co 400 le ca

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Student Growth Percentiles and Accountability Future Directions References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 75th percentile growth and a high achieving student with 25th percentile growth. The low achieving student grew at a rate exceeding 75 percent of their academic peers. The high achieving student grew at a rate exceeding just 25 percent of their academic peers. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Student Growth Percentiles and Accountability Future Directions References Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach is being used by a number of states as part of their ESEA waiver application and was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

Example Student 12 (1234567812) Sample Middle School Advanced Math Achievement High CSAP Math ● Scale Score Typical ● Growth Proficient ● Level Percentiles Low High 66th − 99th ● Typical 35th − 65th Part Proficient Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 379 388 444 462 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 16 69 51 Growth Level Low High Typical Growth Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth Advanced Writing Achievement High CSAP Writing ● Scale Score Typical ● Growth Proficient Level Percentiles ● Low High 66th − 99th Part Proficient ● Typical 35th − 65th Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 425 449 469 504 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 55 53 79 Growth Level Typical Typical High Growth Copyright 2009, Colorado Department of Education/Center for Assessment, Inc.

West Virginia: 2010−2011 Mathematics Norm & Criterion Referenced Growth to College Readiness Percentile Growth Trajectory to College Readiness College Ready Scale Score Not College Ready 2010−2011 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Student Growth Percentiles and Accountability Teacher & School Accountability Student Growth Percentiles and Accountability Future Directions References Going from Students to Classrooms/Schools It’s of interest to examine schools and classrooms where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school, classroom or other grouping calculate the median of the student growth percentiles. If students were randomly sampled, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Student Growth Percentiles and Accountability Descriptive Accountability Student Growth Percentiles and Accountability Future Directions References Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Student Growth Percentiles and Accountability Descriptive Accountability Student Growth Percentiles and Accountability Future Directions References Descriptive Accountability “This is the difference between a retrospective question of identifying fault as opposed to a prospective strategy to engineer some corrective measure, almost independent of considering whether there was blame-worthiness. And to move away from the blame-worthiness paradigm toward something that is more regulatory in nature where one might seize upon disparities or circumstances that are for some reason deemed unacceptable and engineer the interventions needed to bring about the necessary change. . . . It’s the no-fault gap closing strategy in which the effort is to build a consensus about a vision of an improved society rather than figure out where’s the person we want to pillory.” Christopher Edley (2006) Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

The Big Picture Growth & Accountability Growth & Accountability: SGP/The Colorado Growth Model Student Growth Percentiles and Accountability Student Growth Percentiles and Accountability Future Directions References References Edley, C. (2006). Educational “Opportunity" is the highest civil rights priority. So what should researchers and lawyers do about it? Retrieved June 22, 2006 from the World Wide Web: http://www.softconference.com/MEDIA/WMP/260407/#43.010. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. The Economist and Cukier, K. (2010). Data data everywhere. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner SGP: Norm- and Criterion-Referenced Student Growth

About this copy

Student Growth Percentiles Norm- and Criterion-Referenced Student Growth — Washington Educational Research Association, March 29, 2012.

This is an image-based copy of the original beamer slides; builds and animations are flattened.