Student Growth Percentiles

Hawaii Department of Education · December 8, 2015

Student Growth Percentiles — Hawaii Department of Education, December 8, 2015.
Published

December 8, 2015

Event
State agency
Slides
61
Format
Converted from PDF

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References Student Growth Percentiles Damian W. Betebenner @DBetebenner DBetebenner@nciea.org National Center for the Improvement of Educational Assessment Dover, NH Hawaii DOE SGP Training Session D. W. Betebenner Student Growth Percentiles
  2. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References About Me Damian Betebenner Senior Associate National Center for the Improvement of Educational Assessment. Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009, Betebenner, 2012]. Currently refining and sharing these techniques with approximately two dozen other states. D. W. Betebenner Student Growth Percentiles
  3. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References About Today’s Training Session An overview of growth models and student growth percentiles An introduction to R The basics of using the SGP package More advanced uses of the SGP package D. W. Betebenner Student Growth Percentiles
  4. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References Things we’ll need for today The R software environment installed on your computer. Get from: http://cran.r-project.org/ The SGP package. Githubhomepage: IUUQTHJUIVCDPN$FOUFS'PS"TTFTTNFOU4(1 http://DFOUFSGPSBTTFTTNFOU.github.com/SGP/ CRANpage:http://cran.r-project.org/web/packages/SGP/ D. W. Betebenner Student Growth Percentiles
  5. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Future Directions References From a Paucity to a Plethora of Data The Economist: The Data Deluge More and better organized data: mega, giga, tera, peta, exa, zetta, yotta! Historical records of student achievement Linkage of student achievement to student demographics, teachers, schools, educational programs, . . . Stakeholder interest in an examination of this longitudinal data Interest in examining student achievement over time (student growth) derives from data availability. D. W. Betebenner Student Growth Percentiles
  6. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Future Directions References A Recurring Metaphor Data as a window and a mirror D. W. Betebenner Student Growth Percentiles
  7. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II The Data Revolution Student Growth Percentiles and Accountability Lessons Learned Future Directions References Lessons Learned On Questions John Tukey It is better to have an approximate answer to the right question than a precise answer to the wrong question. On Statistical Models George E. P. Box All models are wrong but some are useful. On Understanding Aristotle We understand best those things we see emerge from their very beginnings. On Accountability Sherlock Holmes It is a capital mistake to theorize before you have all the evidence. It biases the judgment. On Theories of Action Confucius Tell me and I will forget, show me and I will remember, involve me and I will understand. D. W. Betebenner Student Growth Percentiles
  8. The Big Picture Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References The Attractiveness of Growth What is growth and why measure it? Student learning is a central goal of education. Assessments of student achievement provide evidence of the current status of student knowledge and understanding. Learning is demonstrated by growth in student achievement from one point in time to another point in time–not by status at either point time alone. Growth Mindset (Dweck:2006). Students with a “growth mindset” outperform those with a “fixed mindset”. D. W. Betebenner Student Growth Percentiles
  9. The Big Picture Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References Guiding Questions Growth models address specific questions Different growth analysis techniques are good at answering different questions. It is critical to understand these different questions. Different questions lead to different conversations which lead to different uses and outcomes. Changing conversations about education Starting with the right questions simplifies development and motivating the proper use of the growth model results. The questions set the table for those conversations. Yen (2007) for an excellent list of questions derived from a survey of parent, teachers and administrators D. W. Betebenner Student Growth Percentiles
  10. The Big Picture Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for parents? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Parent Questions Did my child make a year’s worth of progress in a year? Is my child growing appropriately toward meeting state standards? Is my child growing as much in Math as Reading? Did my child grow as much this year as last year? D. W. Betebenner Student Growth Percentiles
  11. The Big Picture Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for teachers? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Teacher Questions Did my students make a year’s worth of progress in a year? Did my students grow appropriately toward meeting state standards? How close are my students to becoming Proficient? Are there students with unusually low growth who need special attention? D. W. Betebenner Student Growth Percentiles
  12. The Big Picture Growth & Accountability: Part I Why Growth? Growth & Accountability: Part II Different Models/Different Questions Student Growth Percentiles and Accountability Relevant Questions/Concepts Future Directions References What are the relevant questions for administrators? Yen (2007), from a state survey of parents, teachers and administrators, compiled a list of frequently voiced questions/concerns by stakeholder group. Administrator Questions Did the students in our district/school make a year’s worth of progress in all content areas? Are our students growing appropriately toward meeting state standards? Does this school/program show as much growth as that one? Can I measure student growth even for students who do not change proficiency categories? Can I pool together results from different grades to draw summary conclusions? D. W. Betebenner Student Growth Percentiles
  13. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Growth & Accountability Growth versus Status Enthusiasm for growth in accountability stems from the belief that growth and teacher/school quality are more closely related than status. Enthusiasm for growth also stems from its potential diagnostic uses. D. W. Betebenner Student Growth Percentiles
  14. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Growth & Accountability Fundamental Premise “Good” schools/teachers bring about growth in student achievement in excess of that found at “bad” schools/teachers. The inherent believability of this premise is at the heart of current enthusiasm to incorporate growth models into accountability systems. Note that current NCLB accountability invokes the same premise (with achievement substituted for growth). “Good schools/teachers” are commonly referred to as highly effective schools/teachers. It is critical to understand the distinction between growth and effectiveness. D. W. Betebenner Student Growth Percentiles
  15. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Value-added Analyses/Inferences Who/What is responsible for student growth? Often, analyses of student growth attempt to determine the amount of student progress/growth that can be attributed to the school or teacher. Called Value-added analyses, these techniques attempt to estimate the teacher/school contribution to student achievement. Value-added models quantify effectiveness—purporting to measure the impact on achievement that this school or teacher would have, on average, upon similar students assigned to them. These models and their effectiveness claims are controversial for a number of reasons (EPI (2010), NRC/NAE (2010), Braun (2005), Rubin and Stuart (2004)) . D. W. Betebenner Student Growth Percentiles
  16. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Value-added Analyses/Inferences Who/What is responsible for student growth? Value-added models are better described as effectiveness models than as growth models. Value-added is an inference – a causal conclusion drawn from the data. Value-added isn’t a type of statistical model or type of assessment (if I showed you the statistical model formulation or an assessment, from the model formulation or assessment you wouldn’t be able to tell me whether it’s value-added). All growth models can be used for value-added purposes but not vice-versa. D. W. Betebenner Student Growth Percentiles
  17. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Descriptive Questions Note that the questions from Yen’s (2007) survey are primarily descriptive. The questions are only peripherally associated with causality. High stakes accountability has transformed questions about student growth into questions about responsibility/cause: Teacher and School Effectiveness. Beginning with description doesn’t prevent subsequent discussions about responsibility/cause. Beginning with responsibility/cause often prevents discussions about description. D. W. Betebenner Student Growth Percentiles
  18. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Descriptive Accountability Accountability system results can have value without making causal inferences about school quality, solely from the results of student achievement measures and demographic characteristics. Treating the results as descriptive information and for identification of schools that require more intensive investigation of organizational and instructional process characteristics are potentially of considerable value. Rather than using the results of the accountability system as the sole determiner of sanctions for schools, they could be used to flag schools that need more intensive investigation to reach sound conclusions about needed improvements or judgments about quality [Linn, 2008, p. 21]. D. W. Betebenner Student Growth Percentiles
  19. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Causal Explanation versus Causal Attribution Hilton (1990) distinguished between causal explanation and causal attribution. Causal explanation is conversational/inter-psychic whereas causal attribution is not. The quality of a causal attribution is determined by its truth content whereas the quality of a causal explanation is based upon both relevance and truth content. Is accountability an enterprise directed toward causal attribution (i.e., a branch of research/program evaluation) or an enterprise enabling causal explanation? D. W. Betebenner Student Growth Percentiles
  20. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Describing Student Growth Measuring student growth, even with a vertical scale, is not a simple task. Some believe a vertical scale simplifies the task of measuring student growth. Even with an interval (or ratio) scale, growth is not easy to interpret. Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. Student growth percentiles were developed to provide a normative context for describing student growth. D. W. Betebenner Student Growth Percentiles
  21. Two Years of Test Scores: The Simplest Case 20 core ca ca le le 500 S re 2012 co
  22. Accounting for where a student starts 20 core ca ca le le 500 S re 2012 co
  23. Identifying relative growth 20 core ca ca le le 500 S re 2012 co
  24. A Student Growth Percentile of 40 th Per nt i ce le G wth ro 20 core ca ca le le 500 S re 20 12 co
  25. A Student Growth Percentile of 75 th Per 100 nt i ce le G wth ro 20 core ca ca le le 500 S re 20 12 co
  26. Academic Peers start from the same place Peer 3 6 0 m i re cade e Sco A cal 2 75% 75th Per 100 nt i ce le G wth ro 20 core ca ca le le 500 S re 20 12 co
  27. All data is used to determine a student growth percentile Th is d 7 5 t h s: dat erive perc eer 3 6 0 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer a ll cade e Sco g ro ups A cal 2 75% 75th Per 100 nt i ce le G wth ro 20 core ca ca le le 500 S re 20 12 co
  28. All data is used to determine a student growth percentile eer 2 2 5 cPe= Th m i r is d 7 5 t h ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 1S eer a ll g ro ups th Per 100 nt i ce le G wth ro 20 core ca ca le le 500 S re 2012 co
  29. Low & High Achieving students demonstrate growth Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 10 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 20 Sc
  30. Low & High Achieving students demonstrate growth Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 10 75 er ce 0 nt 0 il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 20 Sc
  31. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. D. W. Betebenner Student Growth Percentiles
  32. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 90th percentile growth and a high achieving student with 10th percentile growth. The low achieving student grew at a rate exceeding 90 percent of similar students. The high achieving student grew at a rate exceeding just 10 percent of similar students. The low achiever’s growth is more exemplary (probabilistically) than the high achiever’s. Judgments about the adequacy of student growth require external criteria. D. W. Betebenner Student Growth Percentiles
  33. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. D. W. Betebenner Student Growth Percentiles
  34. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score D. W. Betebenner Student Growth Percentiles
  35. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2012] and the SGP package [Betebenner et al., 2012] Licensed under Creative Commons Share-and-Share Alike Licenses: In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Colorado, Massachusetts, Nevada, Indiana, West Virginia, Wisconsin are states who have state staff capable of generating the student growth percentiles together with aggregations that follow from their calculation. State ownership/oversight of the SGP calculations has been a big plus in terms of buy-in from the state, enhancing their ability to explain and work with local stakeholders. D. W. Betebenner Student Growth Percentiles
  36. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Description, Inference, and Causality Richard Berk in Regression Analysis, A Constructive Critique (2003) provides an account of the (ab)use of regression Three Cheers for Description Descriptive models are judged by their utility Two Cheers for Inference Inferential models are judged based upon their ability to fulfill statistical criteria associated with generalizing from a sample to a population. What is the chance process? One Cheer for Causality Causal models are judged, in addition to inferential issues, by an external theory which plausibly relates causes/interventions and effects/outcomes. D. W. Betebenner Student Growth Percentiles
  37. The Big Picture Value-Added Growth & Accountability: Part I Description, Inference, and Causality Growth & Accountability: Part II Describing Student Growth Student Growth Percentiles and Accountability Student Growth Percentiles Future Directions Model & Software for Student Growth Percentiles References Criterion Referenced Growth Establishing growth-standards based upon growth-norms The most common adequacy criterion are judging growth toward an achievement goal (i.e., growth-to-standard). Results from student growth percentile analyses can be used to calculate percentile growth trajectories for each student. These trajectories indicate what future rates of growth will lead to and are used to make adequacy judgments. This growth-to-standard approach was approved as part of Colorado’s successful application to the Growth Model Pilot Program. D. W. Betebenner Student Growth Percentiles
  38. Kevin Flowers Sample School Exceeded How to interpret this student growth & achievement report Exceeds High Met Typical SBA Scale Score Nearly Met ● Low Meets SBA Achievement Approaches Levels Not Met Well Below HSA SBA SBA SBA SBA Student Math Math Math Math Math Growth Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Percentile Scale Score 309 2513 2502 2598 Achievement Level Meets Met Nearly Met Met Achievement Growth Percentile 86 19 91 Growth Level High Low High Growth Suggested Uses Exceeded ● Review past growth to assess student academic progress toward SBA achievement goals. Exceeds High Met ● Develop remediation or enrich− ment plans based on rate of ● Typical growth needed to reach higher Nearly Met SBA achievement levels. Low ● Identify the rate of progress ● Meets needed in order to reach or maintain proficient status on the SBA next year. ● ● Not Met Approaches Well Below HSA SBA SBA SBA SBA ELA ELA ELA ELA ELA Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 290 2377 2465 2548 Achievement Level Approaches Not Met Nearly Met Met Achievement Growth Percentile 28 70 94 Growth Level Low High High Growth For more information please visit the Hawaii State Department of Education (HIDOE) at www.doe.k12.hi.us or contact 808−586−3230 Cooperatively developed by HIDOE & the Center for Assessment, Inc. Distributed by HIDOE.
  39. Micah Chavez Sample School 2 Exceeds How to interpret this student growth & achievement report Math Move Up Move Up ● Target ● Achievement Target ● HSA Meets HSA Math ● ● ● Scale Score Scale Score ● Keep Up ●● Keep Up Growth Target Target Level Percentiles HSA High 66th − 99th Achievement Approaches Levels ● Typical 35th − 65th Low 1st − 34th Well Below Student Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Growth 2010 2011 2012 2013 2014 Percentile Scale Score 263 317 322 308 Achievement Level Achievement Target Approaches Meets Meets Meets Meets (315.4)/Exceeds (359) Meets (315.4)/Exceeds (366.7) Achievement Growth Percentile 97 88 48 Catch Up/Keep Up Growth Level High High Typical Growth Move Up/Stay Up Growth Target Keep Up (45)/Move Up (77) Keep Up (44)/Move Up (83) Targets Exceeds Suggested Uses Reading Move Up ● Target Achievement Meets ● HSA Reading ● Scale Score ● Review past growth to assess ● student academic progress toward ● Keep Up Target Growth HSA achievement goals. ● Catch Up ●● Target ● Level Percentiles Develop remediation or enrich− ● Approaches High 66th − 99th ment plans based on rate of growth needed to reach higher Typical 35th − 65th HSA achievement levels. ● Identify the rate of progress Well Below Low 1st − 34th needed in order to reach or maintain proficient status Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 on the HSA next year. 2010 2011 2012 2013 2014 Scale Score 243 287 294 303 Achievement Level Achievement Target Well Below Approaches Approaches Meets Meets (270) Meets (300)/Exceeds (352.8) Achievement Growth Percentile 87 78 75 Growth Level Growth Target High High High Catch Up (50) Keep Up (52)/Move Up (94) Growth For more information please visit the Hawaii State Department of Education (HIDOE) at www.doe.k12.hi.us or contact 808−586−3230 Cooperatively developed by the HIDOE & the Center for Assessment, Inc. Distributed by the HIDOE.
  40. Hawaii: 2017 Mathematics Norm & Criterion Referenced Growth & Achievement Percentile Growth Trajectory No Grade 9 Assessment No Grade 10 Assessment Scale Score 2017 Achievement Percentile 2200 SGP trajectories for a 3rd grade student starting from the Level 1/Level 2 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11
  41. Hawaii: 2017 Reading Norm & Criterion Referenced Growth & Achievement No Grade 10 Assessment No Grade 9 Assessment Percentile Growth Trajectory Scale Score 2017 Achievement Percentile SGP trajectories for a 3rd grade student starting from the Level 1/Level 2 cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 11
  42. Hawaii: 2013 Reading Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment 2013 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
  43. Hawaii: 2013 Reading Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment 2013 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
  44. Hawaii: 2013 Mathematics Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment Percentile Growth Trajectory 2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
  45. Hawaii: 2013 Mathematics Norm & Criterion Referenced Growth & Achievement No Grade 9 Assessment 2013 Achievement Percentile Percentile Growth Trajectory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10
  46. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness Future Directions References Going from Students to Schools It’s of interest to examine schools where students demonstrate, on average, extraordinarily high and low student growth. To summarize the student growth percentiles associated with a school (or other grouping) calculate the median of the student growth percentiles. If students were randomly assigned to schools, expect to see a median of 50. Values greatly above or below 50 are of interest in identifying best practices or providing extra support. Examining growth with achievement sheds new light on school performance. D. W. Betebenner Student Growth Percentiles
  47. Hawaii School Performance SBA Growth & Prior Achievement 2017 SBA Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2016 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Summary Statistics Correlation: 0.31 Correlation (N ≥ 10): 0.31 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2017 Median Student Growth Percentile
  48. Hawaii School Performance SBA Growth & Prior Achievement 2017 SBA Reading Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 2016 Prior Percent at/above Proficient 50 Students 100 Students 250 Students 500 Students Achievement Summary Statistics Correlation: 0.27 Correlation (N ≥ 10): 0.27 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2017 Median Student Growth Percentile
  49. Hawaii School Performance SBA Growth & Achievement 2017 SBA Mathematics Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2017 Percent at/above Proficient 250 Students 500 Students Achievement Summary Statistics Correlation: 0.53 Correlation (N ≥ 10): 0.53 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2017 Median Student Growth Percentile
  50. Hawaii School Performance SBA Growth & Achievement 2017 SBA Reading Higher Achievement Higher Achievement School Size Lower Growth Higher Growth 50 Students 100 Students 2017 Percent at/above Proficient 250 Students 500 Students Achievement Summary Statistics Correlation: 0.43 Correlation (N ≥ 10): 0.43 Lower Growth Higher Growth Lower Achievement Lower Achievement Growth 2017 Median Student Growth Percentile
  51. District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School
  52. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II School Accountability Student Growth Percentiles and Accountability Growth & Effectiveness Future Directions References Growth, Effectiveness, and Value-Added “Good schools” are often called highly effective schools. What’s the relationship between growth and effectiveness? Effectiveness indicates who/what is responsible for the growth (value-added models). RttT has placed teacher responsibility for student progress as a primary issue. In teacher performance evaluations, even when large scale assessment results are available, depending upon the state they constitute, on average, about a third of the overall teacher rating. D. W. Betebenner Student Growth Percentiles
  53. 2009 CSAP Elementary School Mathematics: School Poverty by School Achievement (r = −0.71) 80 Higher Achievement Percent at/above Proficient Lower Achievement Percent Free/Reduced Lunch Students at School
  54. 2009 CSAP Elementary School Mathematics: School Poverty by School Median SGP (r = −0.22) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School
  55. 2009 CSAP Elementary School Reading: School Poverty by School Median SGP (r = −0.25) 80 Higher Average Growth Median Student Growth Percentile 20 Lower Average Growth Percent Free/Reduced Lunch Students at School
  56. Economic Disadvantage vs School Effectiveness TN Elementary Schools 2008 Average Yearly Achievement Gain Math & Reading/Language Arts in Z units from TN Current State Average Percent Free & Reduced Lunch
  57. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Descriptive Accountability “This is the difference between a retrospective question of identifying fault as opposed to a prospective strategy to engineer some corrective measure, almost independent of considering whether there was blame-worthiness. And to move away from the blame-worthiness paradigm toward something that is more regulatory in nature where one might seize upon disparities or circumstances that are for some reason deemed unacceptable and engineer the interventions needed to bring about the necessary change. . . . It’s the no-fault gap closing strategy in which the effort is to build a consensus about a vision of an improved society rather than figure out where’s the person we want to pillory.” C. Edley (2006) D. W. Betebenner Student Growth Percentiles
  58. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking The Colorado Department of Education and the Center for Assessment have been working for the last four years on developing next generation data visualization to accompany growth model data. The goal: Transform conversations about education through active engagement with data (i.e., evidence). Our efforts have received tremendous interest and recognition: Recognized by Adobe for innovative uses of their technology as an Adobe Max Award finalist in October, 2009. 2010 NCME Award for Outstanding Dissemination of Educational Measurement Concepts to the Public. 18 states signing MOUs to co-develop a cloud-based reporting platform in a non-proprietary fashion. Colorado recently devoted $2.5 million of stimulus funds to the development efforts D. W. Betebenner Student Growth Percentiles
  59. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References The Decade of Data Visualization Chart Wars http://infosthetics.com/archives/2010/01/ chart_wars_the_political_power_of_data_ visualization.html Hans Rosling/Gapminder http://www.youtube.com/watch?v=jbkSRLYSojo Journalism in the Age of Data http://datajournalism.stanford.edu/ D. W. Betebenner Student Growth Percentiles
  60. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Descriptive Accountability Student Growth Percentiles and Accountability Web 2.0 Technologies Future Directions References Web 2.0: Data Visualization and Social Networking With a collaborative spirit, with a collaborative platform where people can upload data, explore data, compare solutions, discuss the results, build consensus, we can engage passionate people, local communities, media and this will raise–incredibly–the amount of people who can understand what is going on. And this would have fantastic outcomes: the engagement of people, especially new generations; it would increase knowledge, unlock statistics, improve transparency and accountability of public policies, change culture, increase numeracy, and in the end, improve democracy and welfare. E. Giovannini, Chief Statistician, OECD. June 2007 D. W. Betebenner Student Growth Percentiles
  61. The Big Picture Growth & Accountability: Part I Growth & Accountability: Part II Student Growth Percentiles and Accountability Future Directions References References Berk, R. A. (2003). Regression Analysis: A Constructive Critique. Sage, Thousand Oaks, CA. Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W. (2012). Growth, standards, and accountability. In Cizek, G., editor, Setting Performance Standards: Foundations, Methods, and Innovations, pages 439–450. Routledge, New York. Betebenner, D. W., Iwaarden, A. V., and Domingue, B. (2012). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 0.8-0.0. Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Technical report, Educational Testing Service, Princeton, New Jersey. Economic Policy Institute (2010). Problems with the use of student test scores to evaluate teachers. Technical report, Washington, D.C. Downloaded August 30th, 2010 from http://epi.3cdn.net/724cd9a1eb91c40ff0_hwm6iij90.pdf. Edley, C. (2006). Educational “Opportunity" is the highest civil rights priority. So what should researchers and lawyers do about it? Retrieved June 22, 2006 from the World Wide Web: http://www.softconference.com/MEDIA/WMP/260407/#43.010. Hilton, D. J. (1990). Conversational processes and causal explanation. Psychological Bulletin, 107(1):65–81. Linn, R. L. (2008). Educational accountability systems. In The Future of Test-Based Educational Accountability, pages 3–24. Taylor & Francis, New York. National Research Council and National Academy of Education Getting value of out value-added: Report of a workshop. Technical report, Washington, D.C. Commitee on Value-Added Methodology for Instructional Improvement. R Development Core Team (2012). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Rubin, D. B., Stuart, E. A., and Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1):103–116. Yen, W. M. (2007). Vertical scaling and No Child Left Behind. In Dorans, N. J., Pommerich, M., and Holland, P. W., editors, Linking and Aligning Scores and Scales, pages 273–283. Springer, New York. D. W. Betebenner Student Growth Percentiles