ON THE PRECISION OF MCAS SGPS Update on results presented to April 2013 TAC

BACKGROUND • October 2012: Derived and reported precision results from simulations based upon resampling at the state level. • Fall 2012: State to transition to anchored SGP analyses (i.e., baseline referenced) from previous cohort based analyses. • April 2013: Derived and reported precision results based upon resampling at the individual level for both cohort and anchored analyses. • October 2013: Update of 2013 MCAS results based upon methodology discussed April 2013

METHOD • Precision estimates for individual SGPs are derived from resampling of SGPs using CSEMs from MCAS assessment. • Student scores (current and rst prior) are simulated using CSEM and SGPs are re-calculated based upon perturbed scores. • Source code for the simulation algorithm is available on GitHub at: https:// github.com/CenterForAssessment/SGP/blob/master/R/csemScoreSimulator.R and CSEMs are available in the embedded SGPstateData meta-data object. • 100 simulated SGPs are calculated for each student. Standard deviation of the simulated SGPs is used as the standard error associated with the individual SGP. • Simulated SGPs are calculated for both cohort and anchored SGPs. Anchored SGPs utilize CSEMs averaged across multiple years. fi

RESULTS 2013 MCAS Median SGP (anchored) Standard Errors by grade and content area. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 10 ELA --- 5.62 6.21 6.24 6.06 6.44 6.19 Mathematics --- 5.76 5.99 6.11 6.15 6.05 5.86

RESULTS • The standard error associated with individual SGPs is approximately 6 for each grande and content area. • The maximum standard error associated with individual SGPs is 8.85. • In general, there is greater precision associated with extremely low/high SGPs.

RESULTS: STANDARD ERROR BY PRIOR SCORE SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 4 Std Error ELA Grade 5 Std Error ELA Grade 6 Std Error Growth Theta (Prior) Growth Theta (Prior) Growth Theta (Prior) SGP SGP SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 7 Std Error ELA Grade 8 Std Error ELA Grade 10 Std Error Growth Theta (Prior) Growth Theta (Prior) Growth Theta (Prior) SGP SGP SGP

RESULTS: STANDARD ERROR BY PRIOR SCORE SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 4 Std Error MATHEMATICS Grade 5 Std Error MATHEMATICS Grade 6 Std Error Growth Theta (Prior) Growth Theta (Prior) Growth Theta (Prior) SGP SGP SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 7 Std Error MATHEMATICS Grade 8 Std Error MATHEMATICS Grade 10 Std Error Growth Theta (Prior) Growth Theta (Prior) Growth Theta (Prior) SGP SGP SGP

About this copy

On the Precision of MCAS SGPs — Massachusetts Department of Education, October 2013.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.