Exploring Student Growth in Missouri: Changing Conversations about Education

Missouri Department of Education · September 16, 2010

Exploring Student Growth in Missouri: Changing Conversations about Education — Missouri Department of Education, September 16, 2010.
Published

September 16, 2010

Event
State agency
Slides
37
Format
Converted from PDF

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. The Big Picture Guiding Principles and Questions The Colorado Growth Model New Directions Exploring Student Growth in Missouri Changing Conversations about Education Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire Missouri Growth Model Technical Advisory Committee Missouri DESE, Kansas City, Missouri September 16th, 2010 Damian W. Betebenner Student Growth in Missouri
  2. The Big Picture Guiding Principles and Questions The Colorado Growth Model New Directions About Me Dr. Damian Betebenner, PhD - Senior Associate at the Center for Assessment (NCIEA). - Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009]. - Currently refining and sharing these techniques with other states including Colorado, Massachusetts, Arizona, Indiana, and 15 other states in various stages of investigation/adoption. Damian W. Betebenner Student Growth in Missouri
  3. The Colorado Growth Model Adopted Model Near Model Adoption Interested in Model
  4. The Big Picture The Data Revolution Guiding Principles and Questions Why Growth? The Colorado Growth Model Accountability and Growth New Directions From a Paucity to a Plethora of Data The Economist: The Data Deluge Historical records of student achievement Historical records of student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. Damian W. Betebenner Student Growth in Missouri
  5. The Big Picture The Data Revolution Guiding Principles and Questions Why Growth? The Colorado Growth Model Accountability and Growth New Directions The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Damian W. Betebenner Student Growth in Missouri
  6. The Big Picture The Data Revolution Guiding Principles and Questions Why Growth? The Colorado Growth Model Accountability and Growth New Directions Accountability & Growth Growth versus Status Enthusiasm for growth in accountability stems from the belief that growth and teacher/school quality are more closely related than status. Enthusiasm for growth also stems from its potential diagnostic uses. How do we judge the use of growth related measures within an accountability system? Damian W. Betebenner Student Growth in Missouri
  7. The Big Picture Guiding Principles and Questions Guiding Principles The Colorado Growth Model Guiding Questions New Directions Guiding Principles John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. George E. P. Box All models are wrong but some are useful. Start at the end and work backward The best model answers the questions of greatest relevance to stakeholders. Damian W. Betebenner Student Growth in Missouri
  8. The Big Picture Guiding Principles and Questions Guiding Principles The Colorado Growth Model Guiding Questions New Directions Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. See Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators Damian W. Betebenner Student Growth in Missouri
  9. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness The Colorado Growth Model How much growth did a student make? Is it enough? An important first step in the development of the Colorado Growth Model was to begin discussions by separating the description of growth from discussions of responsibility/accountability. Incorporating growth into accountability followed more easily from the accepted description of growth. The description of growth facilitated stakeholder engagement and investigations of responsibility for good/bad growth. That in turn led to greater stakeholder support for particular forms of accountability. Damian W. Betebenner Student Growth in Missouri
  10. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Describing Student Growth Discussing student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth and change require context. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a norm-referenced basis for describing student growth. Damian W. Betebenner Student Growth in Missouri
  11. 20 core ca ca le le 400 S re 2006 co
  12. 20 core ca ca le le 400 S re 2006 co
  13. 20 core ca ca le le 400 S re 2006 co
  14. 20 core ca ca le le 400 S re 2006 co
  15. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria together with standard setting. Damian W. Betebenner Student Growth in Missouri
  16. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, within a generalized additive model [Hastie and Tibshirani, 1990]. Given assessment scores for t occasions, (t ≥ 2), the τ-th conditional quantile for Yt based upon Yt −1 , Yt −2 , . . . , Y1 is given by QYt (τ|Yt −1 , . . . , Y1 ) = φij (Yj )βij (τ) where φi ,j , i = 1, 2, 3 and j = 1, . . . , t − 1 denote the B-spline basis functions. Quantile regression is used to model the complete distribution of the response variable (current achievement), as opposed to just the conditional mean. B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner Student Growth in Missouri
  17. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Student Growth Percentile Model Properties Does not require a vertical scale No distributional assumptions associated with achievement. Robust to ceiling/floor effects on the exam. Invariant to mono-tone transformations of scale—does not require interval scale. Robust to outliers. For example, states with LOSS scores at 0 but a scale range that “really” begins at 300 to 400. Uses full historical record of student achievement data based upon a 1 grade/1 year progression. Quantile crossing is adjusted for via the procedure of Dette and Volgushev (2009). [Dette and Volgushev, 2009] Damian W. Betebenner Student Growth in Missouri
  18. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Accuracy and Precision of SGP results Discussing accuracy and precision of estimates/calculations requires stipulation of a chance process. There are several possible chance processes at work. For example, measurement, students, equating, Measurement error seems most relevant. Results in a “double whammy” [Carroll et al., 1995, p. 22], impacting both accuracy and precision. Bias is negligible at individual level, particularly in situations with 2 or more prior year scores. Currently using simulation to express precision based upon published CSEM statistics for state assessments at both individual and group level. Based upon multiple conversations with states, fear that confidence intervals lead people to make “bad inferences with confidence”. Damian W. Betebenner Student Growth in Missouri
  19. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2010] and the SGP package [Betebenner, 2010] In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Colorado, Massachusetts, Indiana, and Wisconsin are four states (that I know of) who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. Damian W. Betebenner Student Growth in Missouri
  20. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. Damian W. Betebenner Student Growth in Missouri
  21. Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth
  22. NCIEA  CDE Colorado Department of Education High Math Advanced ● Typical Achievement ● Low CSAP Math ● Scale Score ● ● Growth Level Percentiles Proficient High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 515 517 589 603 Achievement Level Proficient Proficient Advanced Proficient Achievement Growth Percentile 15 92 76 Growth Level Low High High Growth 2008 CSAP Growth & Achievement Report Advanced High Reading Typical Achievement CSAP Reading Low ● Scale Score Proficient Growth Level Percentiles ● High 66th − 99th Typical 36th − 65th Part Proficient Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 567 620 652 657 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 72 89 72 Growth Level High High High Growth Advanced High Writing Achievement ● Typical CSAP Writing ● Scale Score Low ● ● Growth Proficient Level Percentiles Example Student 3020 High 66th − 99th Part Proficient Typical 36th − 65th Low 1st − 35th Unsatisfactory Grade 4 Grade 5 Grade 6 Grade 7 Next Year Scale Score 478 517 519 574 Achievement Level Part Proficient Proficient Proficient Proficient Achievement Growth Percentile 77 44 86 Growth Level High Typical High Growth
  23. 2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
  24. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness From Students to Groups of Students Aristotle We understand best those things we see emerge from their very beginnings. It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) one calculates the median of the student growth percentiles. If students were randomly assigned to schools, one expects to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. Damian W. Betebenner Student Growth in Missouri
  25. District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School
  26. Boston−Charlestown School Performance Growth and Achievement MCAS ELA by School Poverty Higher Achievement X X Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2009 Percent at/above Proficient 250 Students 500 Students X X X X 1000 Students XX X 60 X XX X X X XX X X X XX X X School Percent X X X Free/Reduced Lunch X XX X ● Less than 20 percent XX ● 20 to 40 percent XX X X X X X X X X ● 40 to 60 percent X X XX XX ● 60 to 80 percent Achievement X XX XXX X ● More than 80 percent Lower Growth X XX X Higher Growth Lower Achievement Lower Achievement Growth 2009 Median Student Growth Percentile
  27. The Big Picture Describing Student Growth Guiding Principles and Questions Model & Software for Student Growth Percentiles The Colorado Growth Model Combining Actual and Aspirational Growth New Directions Growth & Effectiveness Growth, Effectiveness, and Value-Added Fundamental Premise “Good” schools (or teachers) bring about student growth in ex- cess of that found at “bad” schools. “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). Assuming the above premise is, at least, partly true. How do we judge the correctness of a value-added model? Damian W. Betebenner Student Growth in Missouri
  28. 2009 CSAP Elementary School Mathematics: School Poverty by School Achievement (r = −0.71) 80 Higher Achievement Percent at/above Proficient Lower Achievement Percent Free/Reduced Lunch Students at School
  29. 2009 CSAP Elementary School Mathematics: School Poverty by School Median SGP (r = −0.22) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School
  30. 2009 CSAP Elementary School Reading: School Poverty by School Median SGP (r = −0.25) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School
  31. Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain 1 Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch
  32. Composite VA Gain by % Poverty School % Poverty
  33. The Big Picture Growth & Accountability Guiding Principles and Questions Web 2.0 Technologies The Colorado Growth Model References New Directions Next Steps for Growth Data Confucius Tell me and I will forget, show me and I may remember, involve me and I will understand. Calculating growth quantities is the first and easiest step in the development and deployment of a growth model. The end goal is to change conversations about education through the examination and use of data. To this end, Colorado has been inspired by the Web 2.0 user-centered design philosophies and technologies. The goal is to produce data visualization and social networking tools to turn data into information for all education stakeholders and ultimately change conversations about education. Damian W. Betebenner Student Growth in Missouri
  34. The Big Picture Growth & Accountability Guiding Principles and Questions Web 2.0 Technologies The Colorado Growth Model References New Directions Descriptive Accountability “Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality.” R. L. Linn (2008) Damian W. Betebenner Student Growth in Missouri
  35. The Big Picture Growth & Accountability Guiding Principles and Questions Web 2.0 Technologies The Colorado Growth Model References New Directions Web 2.0: Data Visualization and Social Networking The Colorado Department of Education and the Center for Assessment have been working for the last two year on developing next generation data visualization to accompany growth model data. The goal: Transform conversations about education through active engagement with data (i.e., evidence). Our efforts have received tremendous interest and recognition: Received (just announced) the 2010 annual award for Outstanding Dissemination of Educational Measurement Concepts to the Public by the National Council on Measurement in Education. Recognized by Adobe for innovative uses of their technology as an Adobe Max Award finalist in October, 2009. Multiple states signing MOUs to co-develop a cloud-based reporting platform in a non-proprietary fashion. Colorado recently devoted $2.5 million of stimulus funds to the development efforts Damian W. Betebenner Student Growth in Missouri
  36. The Big Picture Growth & Accountability Guiding Principles and Questions Web 2.0 Technologies The Colorado Growth Model References New Directions Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 Damian W. Betebenner Student Growth in Missouri
  37. The Big Picture Growth & Accountability Guiding Principles and Questions Web 2.0 Technologies The Colorado Growth Model References New Directions References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2010). SGP: Student Growth Percentile and Percentile Growth Projection/Trajectory Functions. R package version 0.0-5. Carroll, R. J., Ruppert, D., and Stefanski, L. A. (1995). Measurement Error in Nonlinear Models. Chapman & Hall, London. Dette, H. and Volgushev, S. (2009). Non-crossing non-paramatric estimates of quantile curves. Journal of the Royal Statistical Society B, 70(3):609–627. Hastie, T. J. and Tibshirani, R. J. (1990). Generalized Additive Models. Chapman & Hall/CRC, New York, NY. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. R Development Core Team (2010). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. Damian W. Betebenner Student Growth in Missouri