MCAS TAC Oct2012 V6

Massachusetts Technical Advisory Committee · October 2012

MCAS TAC Oct2012 V6 — Massachusetts Technical Advisory Committee, October 2012.
Published

October 25, 2012

Event
Technical advisory committee
Slides
49
Format
Converted from PowerPoint

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Thursday, October 25, 2012
  2. MCAS growth Topics
  3. Agenda ■ (30 min) DAMIAN Overview of SGPs – where we are, where we are going Baseline SGPs – showcase grade 10 results in ELA ■ Efficacy of the B-Splines for adjusting growth expectations at the tails of the distributions Growth and achievement charts – ■ discussion of purpose of the SGPs – discussion of the descriptive capacity of SGPs ■ (30 min) Bob – History and Current Application of the SGPs ■ (30 min) Kathy – Utility and Validity of SGPs ■ (30 min) DAMIAN Reliability: Accuracy (How much are we off from the true value) & Precision (Signal to noise) ■ Arise at the individual levels for summary data: For individuals or groups ■ A discussion about use cases and its relationship to precision & reliability ■ Future Applications Baseline SGPs Proposed for use in educator evaluation (2012, informally – DESE will provide guidance on this in 2013) Future validity studies
  4. An Overview of the Student Growth Percentile Methodology
  5. About me Damian Betebenner ■ Senior Associate at the Center for Assessment ■ Developed student growth percentiles and percentile growth trajectories to help states and educational associations utilize student growth in decision making. ■ Currently refining and sharing these techniques with over 30 states including Colorado, New York, Rhode Island, New Hampshire, Maine, …
  6. Describing Student Growth ■ Measuring student growth, even with a vertical scale, is not a simple task. ■ Some believe a vertical scale simplifies the task of measuring student growth. ■ Even with an interval (or ratio) scale, growth is not easy to interpret. ■ Consider, for example, height. A child might grow 4 inches between ages 3 and 4. 4 inches is a well understood quantity. The 4 inch increase becomes really meaningful only when understood alongside the growth of other 3 to 4 year olds. ■ Student growth percentiles were developed to provide a norm- referenced description of student growth (i.e., a growth norm).
  24. History and Current Applications of the SGPs
  25. Why did we need a growth model ■ NCLB required ELA and Mathematics tests in all grades 3-8 beginning in 2006 ■ Longitudinal data systems allowed schools and districts to view assessment trends over time ■ Inferences made by comparing scaled scores from one grade to another are invalid, but were being made with increasing frequency
  26. 2012 MCAS Percent Proficient by Grade ELA Mathematics
  27. Growth Model Roll-out Recap Spring 2009 SGP Pilot ■ A Department staff-member was assigned to each of 10 volunteer districts to help principals and curriculum coordinators interpret data and use the Education Data Warehouse ■ Reports were developed and modified based on feedback from pilot districts ■ Districts struggled to interpret Growth to Standard Data, which was removed from the system ■ Pilot districts helped Department staff develop FAQ and interpretive materials for the statewide release
  28. Growth Model Roll-out Recap October 2009: Statewide release ■ Growth Reports from 2008 and 2009 added to School and District Profiles ■ The Department pledged not to use Growth Data in the accountability system while school and district officials familiarized themselves with the data ■ Student level reporting in the Education Data Warehouse hampered by very slow technology ■ 475 Data Warehouse users statewide were able to run 3,000 reports in a peak week
  29. Growth Model Roll-out Recap ■ 2010 First Commendations given to schools with two consecutive years of median Growth above 60 in ELA & Math ■ Massachusetts’ Race to the Top application proposes using Growth data as significant part of the educator evaluation system – Two thirds of the state’s districts sign up for RttT 17% of teachers in Massachusetts have 20 or more growth scores in their primary subject ■ June 2010-2011 Student Assessment oversees $460,000 project to upgrade and renovate the EDW ■ The Department pilots the collection of student, course and educator data
  30. Growth Model Roll-out Recap ■ Use of the renovated EDW burgeons to 150,000 reports in September 2011 then 268,000 in September 2012 About half of all reports feature growth ■ 99,000 users per hour review public MCAS Growth and Performance Reports on September 19, 2012 ■ Every Principal in Massachusetts is required to attend one of seven MCAS workshops where a growth model is explained in detail – 96.5% attend
  31. Growth Model Roll-out Recap ■ The state’s NCLB waiver uses ELA and Mathematics growth as two of 11 indicators in the Progress and Performance Index used to make accountability ratings ■ The state proposes using SGPs along with at least one other indicator to identify trends and patterns in Educator Effectiveness
  32. Growth in the NCLB Waiver Median SGP Range Alternate pathways Number of points 60-99 15+ point change in SGP 100 50.5-59.5 10-14 point change 75 41-50 1-9 point change 50 Schools must average 75 points across all indicators and for their High Needs Subgroup to be classified in Level 1
  33. Growth in the Educator Evaluation Proposal
  34. 2010 to 2011 English and Elementary Teachers’ two year growth trends (Moderate = 40 to 59.5) ELA 2011 Teacher Growth Category Higher Moderate Lower Total Count 210 126 31 367 Higher % of Total 18.7% 11.2% 2.8% 32.7% ELA 2010 Pilot Growth Count 144 307 90 541 Moderate Category % of Total 12.8% 27.3% 8.0% 48.1% Count 19 107 90 216 Lower % of Total 1.7% 9.5% 8.0% 19.2% Count 373 540 211 1124 Total % of Total 33.2% 48.0% 18.8% 100.0% R = .52; The attrition of teachers with low growth in the 2010 pilot was higher (19.8%) than for teachers with high growth (14.7%)
  35. 2010 to 2011 Mathematics and Elementary Teachers’ two year growth trends (Moderate = 40 to 59.5) Math Teachers 2011 Growth Category Higher Moderate Lower Total Count 195 132 14 341 Higher % of Total 18.7% 12.6% 1.3% 32.6% Math Teachers 2010 Count 127 244 103 474 Moderate Growth Category % of Total 12.2% 23.3% 9.9% 45.4% Count 27 82 121 230 Lower % of Total 2.6% 7.8% 11.6% 22.0% Count 349 458 238 1045 Total % of Total 33.4% 43.8% 22.8% 100.0% R = .56; The attrition of teachers with low growth in the 2010 pilot was higher (18.4%) than for teachers with high growth (12.8%)
  36. SGP Utility and Validity
  37. Validity and Utility of SGPs – An examination of consequential validity evidence Overall: Promote a defensible way of describing and understanding changes in achievement. Intended Outcomes Degree Goal is Achieved 1. Growth indicator that is correlated 1. SGPs are highly correlated to gains with improvement on MCAS exams on MCAS and to changes in performance levels 2. Growth indicator that is uncorrelated to student 2. SGPs have very low correlations demographic measures with student demographic measures at both the individual and the school level 3. All students have an equal opportunity to show growth 3. Due to the use of B-Spline fitting, unfettered by floor and ceiling SGPs are largely unaffected by effects floor and ceiling effects; almost all students have an equal opportunity to grow
  38. SGPs have standardized measurement properties: Scatterplot matrix: ELA Change scores are negatively correlated with prior year’s MCAS SSs (r=-.29 to -.50 ) Meaning: Regression to the Mode/Mean SGPs areSGPs NOT arecorrelated with prior NOT correlated MCAS SSs (r=0), with eachbut are (r=0), other positively but correlatedhighly with current (r=.49 correlated withto .56) and futurechange (r=.22 scores to .25) MCAS SSs (.75 to .81) Meaning: Student growth is unrelated to prior achievement levels Sample: ~61,000+ students starting in grade 5 in 2009 and ending in grade 8 in
  39. SGPs explain student achievement patterns Math Proficiency Levels 2009 and 2012 by Student Average Growth 2010 to 2012 Math Growth Average #10#3 SGP Categories: Very Grand Very Low: 1-16 Math PL 2012 Very Low Low Moderate High High Total Low : 17-33 Warning 1,076 3,450 5,840 247 30 10,643 Moderate: 34-66 Needs Improvement 529 3,549 12,943 1,535 175 18,731 High: 67-82 Proficient 202 2,373 14,514 3,030 421 20,540 Very High:83-99 Advanced 19 452 9,234 4,466 1,023 15,194 Grand Total 1,826 9,824 42,531 9,278 1,649 65,108 Math Proficiency Levels 2009 and 2012 by Student Average Growth 2010 to 2012 The majority of Math Growth Average #10#3 students achieving Math PL 2012 Very Low Low Moderate High Very High high growth from Warning 59% 35% 14% 3% 2% 2010-12 end up Needs Improvement 29% 36% 30% 17% 11% scoring Advanced Proficient 11% 24% 34% 33% 26% in 2012. Advanced 1% 5% 22% 48% 62% Grand Total 100% 100% 100% 100% 100%
  40. SGPs explain student achievement patterns Math Proficiency Levels 2009 and 2012 by Student Average Growth 2010 to 2012 While students in all Math PL 2012 performance Math Growth Average Needs categories in 2009 Math PL 2009 #10#2 Warning Improvement Proficient Advanced Grand Total have an equal chance Warning Low 18% 1% 19% Moderate 46% 18% 0% 0% 64% at growth… High 3% 12% 3% 0% 17% Needs Improvement Low 11% 6% 0% 0% 17% Students achieving Moderate 7% 45% 14% 0% 66% high average growth High 0% 3% 12% 2% 18% Proficient Low 2% 11% 3% 0% 17% are 5X more likely to Moderate 0% 12% 44% 9% 65% move out of the Failing High 0% 0% 5% 13% 18% level. Advanced Low 0% 3% 11% 3% 17% Moderate 0% 0% 17% 48% 65% High 0% 18% 18% Students achieving low Grand Total 16% 29% 31% 23% 100% average growth are 4X more likely to fall from the Proficient level.
  41. SGPs help identify students who gain or lose ground on MCAS performance levels Students 2009: 216 starting out in 2012: 240 SGP10: 99 needs SGP11: 20 SGP10: 99 SGP12: 16 improvement in SGP11: 62 Mod. Growth 2012: 228 2009 with high SGP12: 23 Students average growth Mod. Growth SGP10: 1 move up a out in starting SGP11: 28 failing in 2009 performance SGP12: 78 level with high in 2012 Mod. Growth average growth move up a performance level in 2012
  42. SGPs predominantly describe growth, not student demographic information Proportion of Explained Variance, Stepwise Linear Regression R R^2 Variables Entered 0.777 0.604 MB1: MATG2009 0.945 0.893 MB2: MATG2009 *Math SGPs 2010-12 0.948 0.899 MB3: MATG2009 *MSGP 2010-12 *Student Demographics Student demographics explains .6% residual variance.
  43. SGPs provide an important indicator of school effectiveness Hierarchical Regression Coefficients & (Standard Errors)* Variance Components of Hierarchical Regression Models* Null Model MB1 MB2 MB3 Null Model MB1 MB2 MB3 Intercept -.081 -.031 -013 .120 (.023) (.011) (.004) (.005) Variance Components MATG2009 (GMC) .858 .869 .812 L1 Student (W) 1.422 0.602 0.178 0.167 (.003) (.002) (.002) L2 School (B) 0.443 0.107 0.008 0.008 MSGP 2010 (GMC) .011 .011 Additional Variance Explained L1 Student (W) 57.64% 87.51% 88.26% MSGP 2011 (GMC) .010 .010 (.000) (.000) L2 School (B) 75.88% 98.13% 98.13% MSGP2012 (GMC) .019 .019 Deviance (-2LL) 197,274 144,388 68,422 65,261 (.000) (.000) Change in -2LL 52,886 75,966 3,161 freelunch_sum -0.02 # of parameters 3 4 7 20 Significance 0.000 0.000 0.000 0.000 title1_sum -.033 (.002) Intra-class Correlation lep_sum .007 ICC 23.8% (.006) *Dependent Variable: Math Theta 2012 lep_flep_sum -.022 sped_sum -.053 Race_AA -.086 Race_A .060 24% of Variance is Race_H -.025 87.5% of between schools Addition varianceof for Race_M demographic students and 98% of Race_O variables variance for explains schools is an accom_l_sum additional described by .75% of the 2009 accom_c_sum variance MCAS scores forthe and students accom_m_sum to 2011 and SGPs .001% of (.006) variance for schools
  44. SGPs maximize benefits and minimize weaknesses of student growth data ■ Prior to the SGPs, individuals had no easy way to determine if they performed better or worse on MCAS from one grade to the next ■ Change scores demonstrate regression to the mode ■ SGPs are strongly correlated with change (r = .85), but are more stable ■ SGPs are strongly associated with future performance on MCAS ■ SGPs are very weakly correlated with student demographics ■ Given MCAS’ lack of floor/ ceiling effects, SGPs allow all students ~equal chance to grow Size denotes change scores – range from -42 to 48
  45. Correlations: SGPs, Change Scores, and Background Variables Correlations: SGPs with Change Scores and Student Background Variables msgp.09 msgp.10 msgp.11 msgp.12 esgp.09 esgp.10 esgp.11 esgp.12 msgp.09 1 0.00 0.00 0.00 esgp.09 1 0.00 0.00 0.00 msgp.10 0.00 1 0.00 0.00 esgp.10 0.00 1 0.00 0.00 msgp.11 0.00 0.00 1 0.00 esgp.11 0.00 0.00 1 0.01 msgp.12 0.00 0.00 0.00 1 esgp.12 0.00 0.00 0.01 1 mscaleds.09 0.52 0.00 0.00 0.00 escaleds.09 0.56 0.00 0.00 0.00 mscaleds.10 0.28 0.47 0.00 0.00 escaleds.10 0.25 0.53 0.00 0.00 mscaleds.11 0.27 0.26 0.44 0.00 escaleds.11 0.23 0.24 0.51 0.00 mscaleds.12 0.26 0.24 0.22 0.40 escaleds.12 0.24 0.24 0.22 0.49 MathGrowthAverage 0.51 0.51 0.51 0.51 ELAGrowthAverage 0.50 0.51 0.51 0.51 MathGrowthAverage.10 0.00 0.59 0.58 0.59 ELAGrowthAverage.10 0.00 0.59 0.58 0.59 MedGrMath 0.48 0.48 0.48 0.48 MedGrELA 0.48 0.48 0.48 0.49 MathChange2010 -0.43 0.84 0.00 0.00 ELAChange2010 -0.50 0.81 0.00 0.00 MathChange2011 -0.01 -0.38 0.84 0.00 ELAChange2011 -0.08 -0.54 0.75 0.00 MathChange2012 0.00 -0.04 -0.44 0.81 ELAChange2012 0.01 0.00 -0.48 0.81 freelunch_sum -0.10 -0.08 -0.07 -0.07 freelunch_sum -0.10 -0.06 -0.06 -0.04 title1_sum -0.09 -0.05 -0.04 -0.06 title1_sum -0.08 -0.06 -0.04 -0.01 lep_sum -0.03 0.02 0.02 0.01 lep_sum -0.04 0.02 0.02 0.02 lep_flep_sum -0.02 0.03 0.03 0.01 lep_flep_sum -0.02 0.04 0.03 0.03 sped_sum -0.11 -0.10 -0.07 -0.03 sped_sum -0.10 -0.10 -0.08 -0.03 Race_AA -0.01 -0.03 -0.01 -0.02 Race_AA -0.04 -0.02 0.01 0.01 Race_A 0.07 0.07 0.06 0.04 Race_A 0.06 0.07 0.05 0.07 Race_H -0.07 -0.04 -0.02 -0.04 Race_H -0.05 -0.01 -0.02 -0.02 Race_M 0.00 -0.01 0.00 -0.01 Race_M -0.01 0.00 0.00 0.00 Race_O 0.00 0.00 0.00 0.00 Race_O 0.00 0.00 0.00 0.00 accom_l_sum -0.11 -0.10 -0.08 -0.05 accom_l_sum -0.09 -0.10 -0.10 -0.06 accom_c_sum -0.03 -0.04 -0.03 -0.03 accom_c_sum -0.02 -0.04 -0.03 -0.04 accom_m_sum -0.12 -0.09 -0.08 -0.05 accom_m_sum -0.10 -0.10 -0.09 -0.05 Math N = 62000+, ELA N = 61,000+
  46. SGP Precision
  47. Precision of SGPs
  48. Precision of SGPs
  49. Precision of SGPs