Understanding the West Virginia Growth Model

REL West Virginia · 2015

Understanding the West Virginia Growth Model — REL West Virginia, 2015.
Published

January 1, 2015

Event
Invited talk
Slides
85
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Understanding the West Virginia Growth Model February 20, 2015 Morgantown, West Virginia
  2. Welcome and Overview Justin Baer, Ph.D. Director, REL Appalachia
  3. What is a REL? • A REL is a regional educational laboratory. • There are 10 RELs across the country. • The REL program is administered by the U.S. Department of Education, Institute of Education Sciences (IES). • A REL serves the education needs of a designated region. • The REL works in partnership with the region’s school districts, state departments of education, and others to use data and research to improve academic outcomes for students.
  4. What is a REL?
  5. REL Appalachia’s mission • Meet the applied research and technical assistance needs of Kentucky, Tennessee, Virginia, and West Virginia. • Conduct empirical research and analysis. • Bring evidence-based information to policymakers and practitioners: – Inform policy and practice—for states, districts, schools, and other stakeholders. – Focus on high-priority, discrete issues and build a body of knowledge over time. http://www.RELAppalachia.org Follow us! @REL_Appalachia
  6. Workshop goals • Provide participants with the research foundation and background of the WV Growth Model • Increase participants’ understanding of specific applications of the growth model in WV schools. • Increase participants’ understanding of how the WV Growth Model and forthcoming WV A-F school grading system are linked.
  7. West Virginia Growth Model: Changing Conversations about Student Performance Dr. Damian Betebenner Senior Associate The National Center for the Improvement of Educational Assessment
  8. West Virginia Growth Model • The existence of annual student testing together with unique student identifiers forms the foundation for the calculation of student growth. • Over the last decade, most states have begun calculation of longitudinal student growth using large scale state assessments. • The analysis of student growth is controversial because of the uses to which it is put: Teacher & School Evaluation. • Development of the West Virginia Growth Model involved the creation of a diagnostically useful quantity that could be used (or not used) at a later time for many purposes.
  9. West Virginia Growth Model • Research on the SGP model began in 2007 (Betebenner, 2008) • Based upon work in Colorado, the norm-referenced model was extended to include a criterion-referenced component. (Betebenner, • Work in other states incorporating the model into state reporting and accountability systems occurred in subsequent years and student growth became more common across state assessment systems. (Betebenner, 2012)
  10. West Virginia Growth Model • Where you are is a function of where you started and how quickly you’ve travelled since. • Student growth and student attainment/achievement are related in this way. • Distance=Rate x Time • Those who start behind tend to stay behind and those that start ahead tend to stay ahead.
  11. West Virginia Growth Model • The concept of student growth isn’t as difficult to understand and popular perception suggests. • The calculation may be complicated but know how to calculate student growth is not synonymous with understanding student growth: – The number is difficult to calculate but is simple to understand conceptually. – Height and weight percentiles associated with children are difficult to calculate but are simple to understand conceptually. • Computational details are important but often obscure a simple conceptual basis.
  12. West Virginia Growth Model • There are 3 general approaches of calculating student growth: – Growth as an amount of change (i.e., gain). – Growth as a norm/peer referenced quantity. – Growth as a criterion referenced quantity.
  13. West Virginia Growth Model • Growth as an amount of change: – What could be more simple? – Like height or weight just calculate the amount of change. – Requires a vertical scale (which West Virginia has up to this year and will have going forward with SBAC). – Growth = Score Time 2 - Score Time 1 • Not as simple as it seems: – Comparisons between students often leads to wrong conclusions – Interpreting change requires context.
  14. West Virginia Growth Model Stepping outside of education for an example
  15. West Virginia Growth Model • Which high jumper improved more: – Judy improved 2 inches. – Anna improved 6 inches. – An inch is an inch!!! – But is an inch really an inch??? – 2 inches for an elite jumper is far more remarkable than 6 inches for a novice.
  16. West Virginia Growth Model • The same holds true for the vertically scaled WESTEST 2. Average Scale Score Change 2013 to 2014 Top 10% Reading Grade 4 2013 -7.5 Bottom 10% Reading Grade 4 2013 +24.5 Top 10% Mathematic Grade 4 2013 -12.6 Bottom 10% Mathematics Grade 4 2013 +18.4 • Just as an inch isn’t necessarily an inch; a scale score point isn’t necessarily a scale score point.
  17. West Virginia Growth Model • This is analogous to height/weight measures for infants. • Growth amounts are tricky to interpret and require context. An infant male toddler is measured at 2 and 3 years of age and is shown to have grown 4 inches. The magnitude of increase—4 inches —is a well understood quantity that any parent can grasp and calculate at home using a simple yardstick. However, parents leaving their pediatrician’s office knowing only how much their child has grown would likely be wanting for more information: Parents are not interested in an absolute magnitude of growth, but instead in a norm-referenced quantity locating that 4 inch increase alongside the height increases of similar children.
  18. West Virginia Growth Model • Peer-based analyses are also called norm-referenced analyses. • Like with infant height/weight charts, peers are children/student with similar characteristics. • For growth analyses using large scale assessments, peers are students with (at least) the same scale score history on the same tests. • Peer based norms are invaluable in providing context for stakeholders to understand student growth.
  19. West Virginia Growth model cal ore e Sc Sc ore al e Source: West Virginia Department of Education/ NCIEA
  20. West Virginia Growth model cal e e or Sc or e Sc e c al Source: West Virginia Department of Education/ NCIEA
  21. West Virginia Growth model cal e e Sc c or or e S e al 4 Sc Source: West Virginia Department of Education/ NCIEA
  22. West Virginia Growth model cal e e Sc c or or e S e al 4 Sc Source: West Virginia Department of Education/ NCIEA
  23. West Virginia Growth model cal e e Sc c or or e S e al 4 Sc Source: West Virginia Department of Education/ NCIEA
  24. West Virginia Growth model cal e e Sc c or or e S e al 4 Sc Source: West Virginia Department of Education/ NCIEA
  25. West Virginia Growth Model • Peer-based growth calculations are only as good as the assessments on which they are based. • With quality assessments, peer based analyses can allow for all students to demonstrate growth. • However, ceiling or floor effects on a test can impede the analysis of growth for students at the extremes. • The goal is to be able to demonstrate for stakeholders student growth that is exemplary as well as that which is problematic.
  26. West Virginia Growth Model • Growth is achievement over time Source: West Virginia Department of Education/ NCIEA
  27. West Virginia Growth Model • Peer based comparisons (norms) are helpful but limited by their very nature. • One needs criteria as well as norms to have a complete understanding. • A criterion-referenced interpretation examines the quantity against a set of standards. • Student performance on the WESTEST 2 within the 5 performance levels is a familiar example of a criterion referenced interpretation of attainment/achievement.
  28. West Virginia Growth Model • Criterion-referenced growth is also referred to as growth-to- standard. • The standards used to examine growth are often the state’s performance standards. • Peer based analyses (e.g., SGP) are used in many states as part of growth-to-standard analyses. • Growth-to-standard analyses produce growth targets determining whether a student’s growth is sufficient to reach/maintain state standards. • Absent norms, criterion-referenced growth produces results similar to achievement indicating: – Students that start behind tend to stay behind. – Students that start ahead tend to stay ahead.
  29. West Virginia Growth Model Source: West Virginia Department of Education/ NCIEA
  30. West Virginia Growth Model Source: West Virginia Department of Education/
  31. West Virginia Growth Model Source: West Virginia Department of Education/ NCIEA
  32. West Virginia Growth Model Source: West Virginia Department of Education/ NCIEA
  33. West Virginia Growth Model • Most of the controversy surrounding student growth analyses centers on their (high stakes) uses: – Teacher evaluation – School evaluation/accreditation – Principal evaluation • There is broad disagreement on whether results derived from state summative assessments are appropriate for personnel or institutional evaluation. • Measures and uses often get conflated leading to push back against the measures (annual state assessments) themselves. • States tailor their use of SGPs to their particular state policies.
  34. West Virginia Growth Model • Like most states, West Virginia will transition to a new state assessment in • West Virginia is a part of the Smarter Balanced (SBAC) consortium and will use their test. • The transition from the WESTEST 2 to SBAC have some impact on the growth analyses: – It will have minimal impact upon the norm-referenced analyses. – It will have a larger impact upon the criterion-referenced (growth-to- standard) analyses. • Even with two distinct tests (WESTEST 2 & SBAC), it’s possible to monitor the relative progress of students. • Monitoring growth-to-standard during the transition is more difficult for two reasons: – The standards have not been officially established – Data is not yet available on how students have progressed relative to the new test and standards.
  35. West Virginia Growth Model • New tests, performance standards, and accountability systems have made the use of growth results for high stakes purposes even more controversial. • Many states are diminishing/eliminating state assessment results from accountability calculations during the assessment transition. • Stakeholders often conflate the test itself with the use of the test. • The challenges associated with the use of growth during the assessment transition are more about communication than about technical issues.
  36. Questions?
  37. Break
  38. The West Virginia Growth Model: What it Means for Your School Timothy Butcher Coordinator, Office of Assessment and Accountability West Virginia Department of Education
  39. Results Results • Student Growth – Approximately 182,000 tested students per year – Approximately 174,000 students are accountable under growth model – Growth data observed for all students in grades 4-11 – Adequate growth data obtained for all students in grades 4-10
  40. Results Results • Student observed growth (grades 4-11) – Most meaningful at the student level – State Level Results: ▪ 35% of students in the state show low growth ▪ 30% of students show typical growth ▪ 35% of students show high growth – District and school results are more meaningful because it varies – Next slide shows available reports
  41. Growth Model Reports: Summary Source: West Virginia Department of Education Internal WVEIS WOW Application
  42. Growth Model Reports: Observed Source: West Virginia Department of Education Internal WVEIS WOW Application
  43. Growth Model Reports: Adequate Source: West Virginia Department of Education Internal WVEIS WOW Application
  44. Growth Model Reports: Observed Growth Source: West Virginia Department of Education Internal WVEIS WOW Application
  45. Growth Model Reports: Grade Source: West Virginia Department of Education Internal WVEIS WOW Application
  46. Results Results • Student Adequate Growth (grades 4-10) – Are students demonstrating enough growth to meet grade level expectations in the future? – Not be proficient now, but considered to be on track – If proficient, are you keeping up or on track to fall behind?
  47. Results: Math Results: Math Not Proficient Proficient 24% are 61% are Catching Up Keeping Up 76% are Falling 39% are Falling Behind Behind Source: West Virginia Department of Education
  48. Results: RLA Results: Reading and Language Arts Not Proficient Proficient 31% are Catching 66% are Keeping Up Up 69% are Falling 34% are Falling Behind Behind Source: West Virginia Department of Education
  49. Growth Model Reports: Adequate Growth Source: West Virginia Department of Education Internal WVEIS WOW Application
  50. Growth Model Reports: Mathematics by Grade Source: West Virginia Department of Education Internal WVEIS WOW Application
  51. Growth Model Reports: Reading/LA by Grade Source: West Virginia Department of Education Internal WVEIS WOW Application
  52. Data Data • Review your own data to determine how many students are: – Keeping up with grade level expectations – Catching up to grade level expectations – Falling Behind grade level expectations – Staying Behind grade level expectations
  53. Available sources of data Available Sources of Data • Private site for ZoomWV for District Personnel and Principals • WVEIS on the Web (WOW) Principal and Teacher Access
  54. Why Growth? Why Growth? • Why growth? 4 Driving Questions 1. How much growth did my student exhibit? 2. What does this growth mean? 3. How much growth does my student need to be proficient in the future? 4. How likely is this growth? • Student Reports – the primary benefit of the WV Growth Model
  55. Why Growth? Why Growth? • Why growth? – Growth tells us the what ▪ Contextualizes student changes in performance ▪ Identifies successful progress for an individual student, regardless of performance – Growth does NOT tell us why ▪ Concretizes data-driven decision making ▪ Builds interest into where we should look next
  56. The Student Growth Report The Student Growth Report Source: West Virginia Department of Education Sample Report 56
  57. Why Growth? Why Growth? • Electronic reports are available (student level) bundled at the grade, school, and district level. • Reports are printed by your counties.
  58. Student Growth ReportsStudent Growth Reports • Keep conversations simple. – Drive questions towards the why: ▪ Why is my student exhibiting high/typical/low growth? ▪ Why is my student catching up/keeping up? ▪ Why is my student falling behind/staying behind? – Get people to target standards analysis – Must be ongoing through the year to assess a student’s learning trajectory
  59. Questions?
  60. Break-Out Session: Growth Model Lydotta Taylor, Ed.D.. Alliance Lead, REL Appalachia
  61. LUNCH
  62. Debriefing of Break-Out Session Lydotta Taylor, Ed.D.. Alliance Lead, REL Appalachia
  63. WV’s A-F Grading System for Schools & Districts Carla Howe, Ph.D. Interim Executive Director of Data Governance and Accountability West Virginia Department of Education
  64. Purpose of A-F • Provides clear and concise information on how well schools are doing • Puts student achievement as the central focus of our education accountability and accreditation system • Assists the Governor, the Legislature, the WVDE, and county boards of education in ensuring: – that high quality educational standards and annual performance measures and progress are met by all schools and school systems – and that a thorough and efficient system of schools is provided.
  65. A-F Policy Establishes an accountability system for West Virginia Public Schools and an approval process for school systems based on: (1) measures of student performance (2) a system of reviews through which school and school system quality is examined and publically reported
  66. A-F Components • Proficiency • Growth • Graduation Rates for schools with a Grade 12
  67. A-F Point Values A-F Point Values Component Points Possible Total Points Possible Proficiency Mathematics 200 Reading 200 Observed Growth Mathematics 100 Reading 100 Adequate Growth Mathematics 100 Reading 100 Accelerated Growth 1000 points possible Mathematics 100 Reading 100 Graduation Rate 1200 points possible Four-year rate 100 for schools with Grade Five-year rate 100 12 Source: West Virginia Department of Education 67
  68. Proficiency Proficiency rates for mathematics and reading are calculated with the following criteria for each subject independently. • Students with full academic year (FAY) status • Students enrolled in grades 3-11 • Students with a valid assessment result • At least 10 FAY students for each subject at the school level To calculate the proficiency components: • Sum the number of students (that met the criteria above) that were proficient, and divide by the total number of students (that met the criteria above) for each subject. • Multiply the proficiency rate for each subject by 200 to determine the number of points earned out of the total 200.
  69. Observed Growth To calculate observed growth for each subject independently, the following criteria must be met: • Students Enrolled in grades 4-11 • Students who are FAY in the current year • Students with valid scale scores in the current and prior year to get growth results • 10 FAY students with growth calculations at the school level. This calculation takes the sum of students with typical and high growth by subject; then divides by the total number of students with growth measures at a school by subject. Each rate is then multiplied by 100 to determine the number of points earned out of the total 100.
  70. Adequate Growth To calculate adequate growth for each subject independently, the following criteria must be met: • Students Enrolled in grades 4-11 • Students who are FAY in the current year • Students with valid scale scores in the current and prior year to get growth results • 10 FAY students with growth calculations at the school level. This calculation takes the sum of students with a Yes indicator in either the Keeping Up or Catching Up category for each subject (reading and mathematics), then divides by the total number of students with growth measures at a school by subject. Each rate is then multiplied by 100 to determine the number of points earned out of the total 100.
  71. Accelerated Growth In order to calculate Accelerated Growth of the Lowest 25% of students, first, the LOW25 subgroup must be identified.
  72. Identifying the LOW25 subgroup To identify the LOW25 subgroup in a school, the following conditions are to be satisfied independently for mathematics and reading scores: • Students enrolled in grades 4-11 in the current year • Students with a full academic year (FAY) indicator in the current year • Students that take state assessments
  73. Low25 subgroup (cont.) To calculate the LOW25 subgroup, the following steps will be taken for reading and mathematics independently. 1. Count the total number of students who are FAY by grade level in the current year. 2. Determine the number of students in each grade that equals 25% of the accountable student population with growth results. – In cases where the result is a decimal, round up to the next whole number. 3. Using the students’ prior year scale scores, rank the students’ scale scores from lowest to highest by grade. 4. Select the number of students as determined from Step 2 starting with the lowest scale score. − If a student included in the LOW25 subgroup has the same scale score as a student(s) who just missed the 25% cutoff, include the student(s) in the LOW25 subgroup.
  74. LOW25 subgroup (cont.) 5. Set a LOW25 indicator for the students identified as the lowest performers in each grade. 6. Combine students indicated as being in the LOW25 subgroup across all grades into one LOW25 subgroup for the school. 7. If there are fewer than 10 students in the LOW25 subgroup, add an additional student participating in the state assessment per grade across all grades until the minimum of at least 10 FAY students can be met.
  75. Calculating Accelerated Growth To calculate accelerated growth for each subject independently, the following criteria must be met: • A minimum of 10 FAY students in the LOW25 subgroup Using current year growth results, calculate the accelerated growth rate: 1. Sum the LOW25 students across grades that have High growth. 2. Divide by the total number of LOW25 students. 3. Multiple by 100 to determine the number of points earned out of the total 100.
  76. Graduation Rates • The certified graduation rates will be included as part of the A-F report card by converting the graduation rate to points. • Graduation rates are certified during the October 1 Certified Collection for the prior year. These rates will then carry forward and be part of the report card for the year in which the rates are certified. June 2014 graduates and Summer 2014 graduates are certified in October 2014. The rates that are certified will be part of the score card for the 2014-15 school year.
  77. Adding it Up! • A school’s scores for each component will be summed as shown in prior referenced point value table. • Schools will be ranked from highest points earned to the lowest points earned (elementary/middle schools will be ranked separately from high schools). • Based on the points earned rankings, stanines will be calculated for elementary/middle schools, and stanines will be calculated for high schools in Year 1.
  78. What is a stanine? • Stanine (STAndard NINE) is a method of scaling test scores on a nine-point standard scale where the distribution of scores is divided into nine intervals and where the mean lies at the center of the fifth interval.
  79. Assign a letter grade • The letter grade associated with each stanine will be applied to each school within the stanine as shown in the example below. • The percentages in the example below show the approximate percent of schools that will fall into each stanine.
  80. Grade & Point Spans for future years • Once grades have been assigned to schools, the point spans will be established for future years. • The school with the lowest points and the school with the highest points in a letter grade span will create the point span.
  81. Questions?
  82. Break-Out Session: Accountability Lydotta Taylor, Ed.D.. Alliance Lead, REL Appalachia
  83. Debriefing of Break-Out Session Lydotta Taylor, Ed.D.. Alliance Lead, REL Appalachia
  84. Expert Panel Dr. Damian Betebenner, Panelist Timothy Butcher, Panelist Dr. Carla Howe, Panelist Dr. Lydotta Taylor, Facilitator
  85. Wrap-Up and Closing Remarks Stakeholder Feedback Survey Dr. Justin Baer Director, REL Appalachia Dr. Lydotta Taylor Alliance Lead, REL Appalachia