Talks archive, 2010–2014 – Student Growth Percentiles

The Big Picture Why Growth? Growth and Accountability References Student Growth Percentiles Damian W. Betebenner @DBetebenner DBetebenner@nciea.org National Center for the Improvement of Educational Assessment Dover, NH Archdiocese of Baltimore, Baltimore, Maryland August 13th, 2012 D. W. Betebenner Student Growth Percentiles

The Big Picture Why Growth? Growth and Accountability References About Me Damian Betebenner Senior Associate National Center for the Improvement of Educational Assessment. Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. Currently refining and sharing these techniques more than 30 other states. D. W. Betebenner Student Growth Percentiles

The Big Picture Why Growth? The Data Revolution Growth and Accountability Lessons Learned References From a Paucity to a Plethora of Data The Economist: The Data Deluge More and better organized data: mega, giga, tera, peta, exa, zetta, yotta! Historical records of student achievement Linkage of student achievement to student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. D. W. Betebenner Student Growth Percentiles

The Big Picture Why Growth? The Data Revolution Growth and Accountability Lessons Learned References A Recurring Metaphor Data as a window and a mirror D. W. Betebenner Student Growth Percentiles

The Big Picture Why Growth? The Data Revolution Growth and Accountability Lessons Learned References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judgment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Growth Mindset (Dweck:2006). Students with a “growth mindset” outperform those with a “fixed mindset”. D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References What are the relevant questions for policy makers? Policy Maker Questions Can/how should student growth be used in judgments of educator quality? Can/how should student growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for student demographics need to be considered? D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References Describing Student Growth Measuring and communicating student growth is not a simple task. Some believe creating the perfect scale (e.g., height in inches) solves the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a norm-referenced context for describing student growth. D. W. Betebenner Student Growth Percentiles

600 co 11 600 re le ca le ca S re 12 co

co th re ro i le le ce nt ca er th re 600 co le ca

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: SGP = Probability of Current Achievement given Prior Achievement As such, growth percentiles describe the rarity of a student’s current achievement taking account of their prior achievement. D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is as exemplary (in a probabilistic sense) as the high achiever’s. However: Judgments about the adequacy of student growth require external criteria. D. W. Betebenner Student Growth Percentiles

The Big Picture Relevant Questions/Concepts Why Growth? Describing Student Growth Growth and Accountability Student Growth Percentiles References Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. D. W. Betebenner Student Growth Percentiles

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

Alexandro Sosa Sample School How to interpret this student growth & achievement report Language Math Very High Very High Achievement Achievement 4th Quartile 4th Quartile AoB SAT 10 High AoB SAT 10 Language High AoB SAT 10 Math ● Scale Score Typical Scale Score Scale Score ● Typical Low Low Growth Growth Very Low ● Level Percentiles ● Very Low Level Percentiles AoB SAT 10 Very High 81st − 99th Very High 81st − 99th 3rd Quartile 3rd Quartile Achievement High 61st − 80th High 61st − 80th Levels Typical 40th − 60th Typical 40th − 60th 2nd Quartile Low 20th − 39th 2nd Quartile ● Low 20th − 39th Very Low1st − 19th Very Low1st − 19th 1st Quartile 1st Quartile Student Grade 3 Grade 4 Grade 5 Grade 6 Next Year Grade 3 Grade 4 Grade 5 Grade 6 Next Year Growth 2008 2009 2010 2011 2008 2009 2010 2011 Percentile Scale Score 583 652 648 679 Scale Score 588 603 643 646 Achievement Level 1st Quartile 4th Quartile 3rd Quartile 4th Quartile Achievement Achievement Level 2nd Quartile 2nd Quartile 3rd Quartile 2nd Quartile Achievement Growth Percentile −− 66 88 Growth Percentile −− 80 24 Growth Level −− High Very High Growth Growth Level −− High Low Growth Suggested Uses ● Review past growth to assess student academic progress toward Reading Science Very High AoB SAT 10 achievement goals. Achievement Very High Achievement ● Develop remediation or enrich− 4th Quartile 4th Quartile High AoB SAT 10 Reading AoB SAT 10 Science ment plans based on rate of Typical ● Scale Score ● Scale Score ● Low growth needed to reach higher High AoB SAT 10 achievement levels. Very Low Growth Typical Growth ● Low Level Percentiles Level Percentiles ● Identify the rate of progress needed in order to reach or Very High 81st − 99th Very Low Very High 81st − 99th 3rd Quartile 3rd Quartile ● maintain proficient status ● High 61st − 80th High 61st − 80th on the AoB SAT 10 next year. Typical 40th − 60th Typical 40th − 60th 2nd Quartile Low 20th − 39th 2nd Quartile Low 20th − 39th Very Low1st − 19th Very Low1st − 19th 1st Quartile 1st Quartile Grade 3 Grade 4 Grade 5 Grade 6 Next Year Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 608 626 643 679 Scale Score 627 608 632 654 Achievement Level 2nd Quartile 2nd Quartile 2nd Quartile 3rd Quartile Achievement Achievement Level 3rd Quartile 2nd Quartile 2nd Quartile 3rd Quartile Achievement Growth Percentile −− 46 91 Growth Percentile −− 42 52 Growth Level −− Typical Very High Growth Growth Level −− Typical Typical Growth For more information please visit the Archdiocese of Baltimore at www.archbalt.org/schools or contact 1−410−547−5515 Cooperatively developed by the Archdiocese of Baltimore & the Center for Assessment, Inc. Distributed by the Archdiocese of Baltimore.

AOB: 2011 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 675 90 50 2011 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

AOB: 2011 Language Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory Scale Score 2011 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References Going from Students to Groups of Students It’s of interest to examine schools/classrooms/groups where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a groups of students (for example, a school) calculate the median of the student growth percentiles. If students were randomly selected, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. D. W. Betebenner Student Growth Percentiles

AOB School Performance AOB Growth & Prior Achievement 2011 AOB Reading Higher Achievement X Higher Achievement School Size XX X X X 2010 Prior Percent at/above Proficient Lower Growth Higher Growth X 50 Students X XXXX X X 100 Students XX X X X 250 Students X 500 Students X XX X X Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth and Achievement 2011 AOB Reading Higher Achievement X X X X X X Higher Achievement School Size XX Lower Growth X X XX X X Higher Growth 50 Students X XX X 100 Students 2011 Percent at/above Proficient X XX 250 Students 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth & Prior Achievement 2011 AOB Mathematics Higher Achievement X X Higher Achievement School Size 2010 Prior Percent at/above Proficient Lower GrowthX Higher Growth 50 Students X XX X X 100 Students 250 Students XX X 500 Students X XX Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth and Achievement 2011 AOB Mathematics Higher Achievement X X X HigherX Achievement School Size Lower Growth X XX X X Higher Growth 50 Students XX X 100 Students XXX X X 2011 Percent at/above Proficient X X 250 Students X 500 Students Achievement Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth & Prior Achievement 2011 AOB Reading Higher Achievement Higher Achievement School Size 2010 Prior Percent at/above Proficient Lower Growth Higher Growth 50 Students 100 Students 250 Students X 500 Students XX X X X X X XX XXX X Achievement Lower Growth X Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth and Achievement 2011 AOB Reading Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2011 Percent at/above Proficient 250 Students X 500 Students X XX XX XX Achievement Lower Growth X Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth & Prior Achievement 2011 AOB Mathematics Higher Achievement Higher Achievement School Size 2010 Prior Percent at/above Proficient Lower Growth X Higher Growth 50 Students 100 Students 250 Students 500 Students X XX X X XX Achievement Lower Growth X X Higher Growth Lower Achievement Lower Achievement Growth 2011 Median Student Growth Percentile

AOB School Performance AOB Growth and Achievement 2011 AOB Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students X X 100 Students 2011 Percent at/above Proficient 250 Students 500 Students X XX X X XX X Achievement Lower Growth X Higher Growth Lower Achievement X Lower Achievement Growth 2011 Median Student Growth Percentile

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References Growth, Effectiveness, and Value-Added “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). RttT has placed teacher responsibility for student progress as a primary issue. D. W. Betebenner Student Growth Percentiles

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References Value-added Analyses/Inferences Who/What is responsible for student growth? Often, analyses of student growth attempt to determine the amount of student progress/growth that can be attributed to the school or teacher. Called Value-added analyses, these techniques attempt to estimate the teacher/school contribution to student achievement. Value-added models quantify effectiveness—purporting to measure the impact on achievement that this school or teacher would have, on average, upon similar students assigned to them. These models and their effectiveness claims are controversial for a number of reasons (EPI (2010), NRC/NAE (2010), Braun (2005), Rubin and Stuart (2004)) . D. W. Betebenner Student Growth Percentiles

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References Value-added Analyses/Inferences Who/What is responsible for student growth? Value-added models are better described as effectiveness models than as growth models. Value-added is an inference – a causal conclusion drawn from the data. Value-added isn’t a type of statistical model or type of assessment (if I showed you the statistical model formulation or an assessment, from the model formulation or assessment you wouldn’t be able to tell me whether it’s value-added). All growth models can be used for value-added purposes but not vice-versa. D. W. Betebenner Student Growth Percentiles

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References Descriptive Questions Note that the questions from Yen’s (2007) survey are primarily descriptive. The questions are only peripherally associated with causation. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Beginning with description doesn’t prevent subsequent discussions about responsibility/cause. Beginning with responsibility/cause often prevents discussions about description. D. W. Betebenner Student Growth Percentiles

The Big Picture School Accountability Why Growth? Student Growth and Value-Added Growth and Accountability Description, Inference, and Causation References A Recurring Metaphor Data as a window and a mirror D. W. Betebenner Student Growth Percentiles

The Big Picture Why Growth? Growth and Accountability References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Technical report, Educational Testing Service, Princeton, New Jersey. Economic Policy Institute (2010). Problems with the use of student test scores to evaluate teachers. Technical report, Washington, D.C. Downloaded August 30th, 2010 from http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf. National Research Council and National Academy of Education Getting value of out value-added: Report of a workshop. Technical report, Washington, D.C. Commitee on Value-Added Methodology for Instructional Improvement. Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1):103–116. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. D. W. Betebenner Student Growth Percentiles

About this copy

Student Growth Percentiles — Archdiocese of Baltimore, August 13, 2012.

This is an image-based copy of the original beamer slides; builds and animations are flattened.