Talks archive, 2010–2014 – The Maine Growth Model: Changing Conversations about Education

The Big Picture Guiding Principals and Questions History: The Colorado Growth Model Next Steps The Maine Growth Model Changing Conversations about Education Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire September 29th, 2010 Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions History: The Colorado Growth Model Next Steps About Me Dr. Damian Betebenner, PhD Senior Associate at the Center for Assessment (NCIEA). Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009]. In the process of refining and sharing these techniques with other states including Colorado, Massachusetts, Indiana, Virginia, New York and 15 other states in various stages of investigation/adoption. Interested in the rise of the data sciences/scientist: Data analysis and data visualization and their use within education Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Why Student Growth? History: The Colorado Growth Model Accountability & Student Growth Next Steps The Attractiveness of Student Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another–not by status at either point time alone. Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Why Student Growth? History: The Colorado Growth Model Accountability & Student Growth Next Steps Accountability & Student Growth Growth versus Status Enthusiasm for growth in accountability stems from the belief that teacher & school quality are more closely related to student growth than status. Enthusiasm for growth also stems from its potential diagnostic uses. How do we judge the use of student growth related measures within an accountability system? Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Guiding Principals History: The Colorado Growth Model Guiding Questions Next Steps Guiding Principals John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. George E. P. Box All models are wrong but some are useful. Start at the end and work backward The best model answers the questions of greatest relevance to stakeholders. Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Guiding Principals History: The Colorado Growth Model Guiding Questions Next Steps Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. Identify the conversations you want to have. The questions set the table for those conversations. See Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators Damian W. Betebenner The Maine Growth Model

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps History: The Colorado Growth Model How much growth did a student make? Is it enough? An important first step in the development of the Colorado Growth Model was to begin discussions by separating the description of growth from discussions of responsibility/accountability. Incorporating growth into accountability followed more easily from the accepted description of growth. The description of growth facilitated stakeholder engagement and investigations of responsibility for good/bad growth. That in turn led to greater stakeholder support for particular forms of accountability. Damian W. Betebenner The Maine Growth Model

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. NECAP does not posess a vertical scale, so simply subtracting scale scores is not defensible. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth and change require context. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a norm-referenced basis for describing student growth. Damian W. Betebenner The Maine Growth Model

20 core ca ca le le 400 S re 2006 co

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria together with standard setting. Damian W. Betebenner The Maine Growth Model

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps Growth: Norms & Standards The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s Growth Model Pilot Program application. Damian W. Betebenner The Maine Growth Model

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

NCIEA  CDE Colorado Department of Education High Math Advanced ● Typical Achievement ● Low CSAP Math ● Scale Score ● ● Growth Level Percentiles Proficient High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 515 517 589 603 Achievement Level Proficient Proficient Advanced Proficient Achievement Growth Percentile 15 92 76 Growth Level Low High High Growth 2008 CSAP Growth & Achievement Report Advanced High Reading Typical Achievement CSAP Reading Low ● Scale Score Proficient Growth Level Percentiles ● High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 567 620 652 657 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 72 89 72 Growth Level High High High Growth Advanced High Writing Achievement ● Typical CSAP Writing ● Scale Score Low ● ● Growth Proficient Level Percentiles Example Student 3020 High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 478 517 519 574 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 77 44 86 Growth Level High Typical High Growth

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps From Students to Groups of Students It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) one calculates the median of the student growth percentiles. If students were randomly assigned to schools, one expects to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner The Maine Growth Model

Maine: 2009−10 NECAP Growth & Achievement Growth and Achievement NECAP Math Performance by School Poverty 100 Higher Achievement Higher Achievement School Size 90 Lower Growth X X Higher Growth 50 Students XXX 100 Students XX XX 250 Students XX X X XX X X Fall 2009 Percent at/above Proficient X 500 Students XX X XX X X X X XX XX X XX X X X X XXX X X X X XX X X X XXX XXXX X XX X X XX XX 70 X X XXX X X X XX X X X X XXXX X XX X X XX X X X XX XX X XX X XXX XXX X XX XX X XX XXXX X X X X XX X XXX X X X X XX XXX X X XX X X X XXX XXX X X X XX X X X XX X X XX X X X X XX X X X XX X XX X XXX XXX X XX X Percent Free/Reduced XX XX XX X X X XX X X XX XXXX XX XX X X X X XX X X X X XX X X XX X X X XX X X XX X X X XX X Lunch Students XX X XX X XXX X X X XX X XX X XX X X XX X X XX X X XXX X X X XX XX XX XX X X X ● Less than 20 percent XXX X ● 20 to 40 percent XX X X ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent Lower Growth Higher Growth 0 Lower Achievement Lower Achievement Fall 2009 Median Student Growth Percentile

Maine: 2009−10 NECAP Growth & Achievement Growth and Achievement Reading Performance by School Poverty 100 Higher Achievement Higher Achievement School Size Lower Growth Higher Growth X 50 Students 90 XX XXX XXX 100 Students XX XX X XX XX X 250 Students X X XX X X X XX 80 X XX X XXXXX X X XXXX XX Fall 2009 Percent at/above Proficient XX XXXX X X XX XXX X X X 500 Students X X XXX XXXXXX XXX XXX XXXXXX X X X XXX XX XXXX X X X X X X XXXX X XXX X XX XX XX X X XX XX X XXXX X X XX X XX XX XX XX XX XXX XX X XXX X XX XXX XXX X X X XX X X XX X XX X X XX XXXX XXXXXXXXXXXX X X XX X XX XXXX X XXX X XX XXX X XX XXXX XX X X XXXXXX X XX X X X XXXX X X XXX X 60 X X XX X X X XX X X X X XX XX X XX XX X X X X XX XX X X XX Percent Free/Reduced X XX XX X X XX X XX X XX X XX X Lunch Students ● Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent 20 ● More than 80 percent Lower Growth Higher Growth 0 Lower Achievement Lower Achievement Fall 2009 Median Student Growth Percentile

Maine: 2009−10 NECAP Growth & Achievement Growth and Achievement Reading Performance by School Poverty 100 Higher Achievement Higher Achievement School Size 90 Lower Growth Higher Growth 50 Students 100 Students X X 250 Students Fall 2009 Percent at/above Proficient X X 500 Students X X XX X XX X XX XX X X X X XXX X X XX X X XX X X X XX XX X Percent Free/Reduced X XXX X Lunch Students ● Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent 20 ● More than 80 percent Lower Growth Higher Growth 0 Lower Achievement Lower Achievement Fall 2009 Median Student Growth Percentile

The Big Picture Describing Student Growth Guiding Principals and Questions Combining Actual and Aspirational Growth History: The Colorado Growth Model Growth & Effectiveness Next Steps Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools bring about student growth in excess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Stakeholder questions provided by Yen make no mention of effectiveness placing a greater emphasis on description High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School/Principal Effectiveness. Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Growth & Accountability History: The Colorado Growth Model References Next Steps Next Steps for Growth Data Calculating growth quantities is the first and easiest step in the development and deployment of a growth model. The end goal is to change conversations about education through the examination and use of data. Reporting data to stakeholders throughout the state in engaging and meaningful ways is essential to transforming conversations about education. Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Growth & Accountability History: The Colorado Growth Model References Next Steps Descriptive Accountability “Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality.” R. L. Linn (2008) Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Growth & Accountability History: The Colorado Growth Model References Next Steps Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner The Maine Growth Model

The Big Picture Guiding Principals and Questions Growth & Accountability History: The Colorado Growth Model References Next Steps References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner The Maine Growth Model

About this copy

The Maine Growth Model: Changing Conversations about Education — Maine Department of Education, September 29, 2010.

This is an image-based copy of the original beamer slides; builds and animations are flattened.