The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References Changing Conversations about Education Using Growth Data to Inform Practice Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire American Association of School Administrators Virtual Webinar September 11, 2011 Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Future Directions References From a Paucity to a Plethora of Data Data Data Everywhere (The Economist, 2010) More and better organized data: mega, giga, tera, peta, exa, zetta, yotta! Historical records of student achievement. Linkage of student achievement to student demographics, teachers, schools, educational programs, . . . Stakeholder interest in examining student achievement over time (student growth) derives from data availability.. Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Future Directions References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judgment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Questions & Models Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Questions & Models Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Questions & Models Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Questions & Models Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for policy makers? Administrator Questions Can/how should growth be used in judgments of educator quality? Can/how should growth be used in judgments of education (e.g., school) quality? Are students on track for career and college readiness? What adjustments for student demographics need to be considered? Damian W. Betebenner Changing Conversations about Education

The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Criterion Referenced Growth References Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. Damian W. Betebenner Changing Conversations about Education

600 co 09 600 re le ca le ca S re 10 co

600 co 09 600 re le ca le ca S re 10 co

600 co 09 600 re le ca le ca S re 10 co

co th re ro i le le ce nt ca er th re 600 co le ca

The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Criterion Referenced Growth References Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. Damian W. Betebenner Changing Conversations about Education

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness Future Directions References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Changing Conversations about Education

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness Future Directions References Growth, Effectiveness, and Value-Added “Good teachers/schools” are often called highly effective teachers/schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). RttT has placed teacher responsibility for student progress as a primary issue. In teacher performance evaluations, even when large scale assessment results are available, depending upon the state they constitute, on average, about a third of the overall teacher rating. Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking The Colorado Department of Education and the Center for Assessment have been working for the last two years on developing next generation data visualization to accompany growth model data. The goal: Transform conversations about education through active engagement with data (i.e., evidence). Our efforts have received tremendous interest and recognition: Recognized by Adobe for innovative uses of their technology as an Adobe Max Award finalist in October, 2009. 2010 NCME Award for Outstanding Dissemination of Educational Measurement Concepts to the Public. 14 states signing MOUs to co-develop a cloud-based reporting platform in a non-proprietary fashion. Colorado recently devoted $2.5 million of stimulus funds to the development efforts Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References The Decade of Data Visualization Chart Wars http://infosthetics.com/archives/2010/01/ chart_wars_the_political_power_of_data_ visualization.html Hans Rosling/Gapminder http://www.youtube.com/watch?v=jbkSRLYSojo Journalism in the Age of Data http://datajournalism.stanford.edu/ Damian W. Betebenner Changing Conversations about Education

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner Changing Conversations about Education

Student Growth Percentile Adoption Adopted Model Near Model Adoption Interested in Model

The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References References Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Technical report, Educational Testing Service, Princeton, New Jersey. Economic Policy Institute (2010). Problems with the use of student test scores to evaluate teachers. Technical report, Washington, D.C. Downloaded August 30th, 2010 from http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf. Edley, C. (2006). Educational “Opportunity" is the highest civil rights priority. So what should researchers and lawyers do about it? Retrieved June 22, 2006 from the World Wide Web: http://www.softconference.com/MEDIA/WMP/260407/#43.010. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. National Research Council and National Academy of Education Getting value of out value-added: Report of a workshop. Technical report, Washington, D.C. Commitee on Value-Added Methodology for Instructional Improvement. Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1):103–116. The Economist and Cukier, K. (2010). Data data everywhere. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Changing Conversations about Education

About this copy

Changing Conversations about Education: Using Growth Data to Inform Practice — Renaissance Short Cycle, September 11, 2011.

This is an image-based copy of the original beamer slides; builds and animations are flattened.