Three Examples: Using MGPs in Educator Evaluations

Hawaii Department of Education · March 27, 2014

Three Examples: Using MGPs in Educator Evaluations — Hawaii Department of Education, March 27, 2014.
Published

March 27, 2014

Event
State agency
Slides
19
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Three Examples: Using MGPs in Educator Evaluations Damian Betebenner and Elena Diaz-Bilello Center for Assessment A Presentation for the Teacher Leader Work Group
  2. Common Approaches Used to Evaluate MGPs in EES Most common approaches pursued by districts and states using MGPs in educator evaluations or for school accountability discussed in last meeting: Option 1 - Fixed (i.e., MGP based) cuts Option 2 - Adding confidence intervals around the median SGP or building them into the cuts (margin of error)
  3. Examples of both options from 3 different places • Option 1 - Fixed MGPs: New Jersey • Option 2A – Confidence intervals around the median SGP: Denver Public Schools • Option 2B – Building the margin of error into the cuts: Rhode Island
  4. Option 1: New Jersey • Set up standard setting panels consisting of teachers in different districts • Panels Reviewed: a. MGP distributions across schools in the district b. MGP distributions across classrooms in the state c. Performance profiles of students with different SGP levels
  5. New Jersey • Panel used performance profiles to identify cuts that would recognize different levels of MGP growth. • First established consensus on what classified low, typical or high growth • Also set one more cut to differentiate lower levels of typical growth from higher levels of typical growth • Translated MGP values from 1-99 scale to the 1-4 rating scale
  6. Other sources consulted • In addition to reviewing their own data, NJ DOE consulted with other states using a similar fixed approach • Consulted with the TAC to develop the scale Note: Have already communicated to teachers and administrators that cuts set could be adjusted in future years based on emerging information and best practices
  7. Points assigned to MGPs in NJ
  8. Option 2A: Denver Public Schools Like NJ DOE, groups assembled to review growth data and agreed to define cuts that establish thresholds for: a.Low Growth – below 40th percentile b.Typical Growth – 40th to below the 60th percentile c.High Growth – 60th percentile and above
  9. Computed Confidence Intervals (CI) around a Teacher MGP
  10. Evaluating growth with CIs • Look at MGP in each content area • The lower limit of CI in each content area • The upper limit of CIs in each content area • Evaluate these points relative to the cut- points used to classify 3 different levels of growth • Teacher can earn 5 levels of growth classified as A (lowest)to E (highest level of growth)
  11. The MGP Scoring Model Used in DPS 40th 60th percentile percentile
  12. Considerations: Hard conversations about using CIs • Technical justifications to use CIs • Not an easy task to communicate CI concept to the field • Decision to go with CIs governed by belief that CIs met the “fairness principle” of the EES Currently, still experiencing difficulty helping teachers to understand the CI concept
  13. Option 2B: Rhode Island • Rhode Island initially wished to not use a confidence interval • Considerations of imprecision in the median SGPs led to a desire to build in some measure of confidence • The state decided on using an overall confidence interval as opposed to using teacher specific confidence intervals
  14. Caterpillar Plots for Hawaii Teachers: Median SGP < 35 including 1 standard error 9.5% of the teachers have median SGP more than 1 standard error below 35 using BOTH content areas Median Teacher Student Growth Percentile +/− 1 Standard Error Median SGP Median SGP Teacher Rank
  15. Caterpillar Plots for Hawaii Teachers: Median SGP < 35 including 1 standard error 11.7% of the teachers have median SGP more than 1 standard error below 35 using Mathematics Median Teacher Student Growth Percentile +/− 1 Standard Error Median SGP Median SGP Teacher Rank
  16. Caterpillar Plots for Hawaii Teachers: Median SGP < 35 including 1 standard error 5.2% of the teachers have median SGP more than 1 standard error below 35 using Reading Median Teacher Student Growth Percentile +/− 1 Standard Error Median SGP Median SGP Teacher Rank
  17. Option 2B: Rhode Island • Note different length caterpillar “legs”. These are the teacher specific CSEMs. • The median SGP aggregates across both content areas. • With 1 standard error, there is 84% certainty that the median SGP is actually below 35. The “average” standard error is approximately 5 • Rhode Island built this imprecision into the overall cut (now 30, instead of 35), with the goal of simplifying explanation.
  18. Take-away points from case studies In options 1, 2a, 2b: • Growth discussions are focused on classifying levels of growth taking place in classrooms • Evaluations based on criterion-referenced definitions of growth (the cuts used) baked into the MGP distribution • Review of data always used to help inform the placement of cuts • Classifications of growth made with stakeholder groups involved
  19. Guiding questions • When considering the cuts used in many other places, how do these correspond to the levels of growth expectations set in your system? • How did you justify setting the lowest growth threshold at the 5th percentile or at another percentile rank (lowest cut used for percentile rank approach)? What are the implications for this decision? • To what extent do you think it is fair to assign a different rating category to a teacher if