On the Precision of MCAS SGPs: Massachusetts TAC

Massachusetts Department of Education · April 9, 2013

On the Precision of MCAS SGPs: Massachusetts TAC — Massachusetts Department of Education, April 9, 2013.
Published

April 9, 2013

Event
State agency
Slides
43
Format
Converted from Beamer

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Background References Appendix On the Precision of MCAS SGPs Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Massachusetts TAC Cambridge, Massachusetts April 9th, 2013 Damian W. Betebenner On the Precision of MCAS SGPs
  2. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Background Based upon feedback from stakeholders and to be consistent with current reporting of precision of individual level data, over the past year Massachusetts DESE has investigated methodologies for reporting precision. October 2012: Derived and reported precision results from simulations based upon resampling at the state level. Fall 2012: State to transition to anchored SGP analyses (i.e., baseline referenenced) from previous cohort based analyses. April 2013: Derived and reported precision results based upon resampling at the individual level for both cohort and anchored analyses. Damian W. Betebenner On the Precision of MCAS SGPs
  3. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Given a student’s prior scale scores and the associated conditional density, their current scale score corresponds to a percentile of that conditional distribution. This percentile is the student’s growth percentile. Growth percentiles are closely related to estimating the probability of observing a student’s current achievement taking account of their past achievement: Pr(Current Achievement |Past Achievement ). As such, growth percentiles describe the rarity of a student’s current achievement conditional upon their prior achievement. Damian W. Betebenner On the Precision of MCAS SGPs
  4. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Student Growth Percentiles Should we be surprised with a child’s current achievement given their prior achievement? Student growth percentiles answer this question. Consider a low achieving student with 50th percentile growth and a high achieving student with 50th percentile growth. The low achieving student grew at a rate exceeding 50 percent of similar students. The high achieving student grew at a rate exceeding just 50 percent of similar students. The low achiever’s growth is probabilistically equivalent to the high achiever’s. Equally rare does not imply equally good! Judgments about the adequacy of student growth require external criteria. That is, criterion referencing. Damian W. Betebenner On the Precision of MCAS SGPs
  5. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Model for Student Growth Percentiles Student growth percentiles are calculated using quantile regression with B-spline smoothing, a generalized additive model [Betebenner, 2008, Betebenner, 2009]. Quantile regression is used to model the complete distribution of the response variable (current achievement) B-spline smoothing is uses to accommodate non-linearity and heteroscedasticity of the data. The model/method is descriptive and all about data fit—it’s a data mining procedure. The model quantifies distance = rate · time probabilistically. Damian W. Betebenner On the Precision of MCAS SGPs
  6. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Model for Student Growth Percentiles Conditional Decile Regression Curves Conditional Decile Regression Lines 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score Damian W. Betebenner On the Precision of MCAS SGPs
  7. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Software for Student Growth Percentiles Student Growth Percentiles and Percentile Growth Trajectories are calculated using the open source R Software Environment [R Development Core Team, 2013] and the SGP package [Betebenner et al., 2013] Creative Commons Share-and-Share Alike Licenses: In working with state departments of education, the goal is to build state capacity so states can perform and validate their own analyses using software available and easily used by the state. Software is open source and developed on the GitHub collaborative version control system: https://github.com/dbetebenner/SGP CSEMs (and other meta-data) for MCAS is maintained within the package and developed/maintained using the GitHub collaborative version control system: https://github.com/dbetebenner/SGPstateData Damian W. Betebenner On the Precision of MCAS SGPs
  8. Background Quick Student Growth Percentiles References Model & Software for Student Growth Percentiles Appendix Simulation Methodology SGP precision is calculated by creating 100 SGPs for each student by perturbing current and immediate prior growth theta scores. Growth theta scores are linear transformations of theta scores. CSEMs associated with theta scores are transformed to growth theta scale and used to simulated growth theta scores for each student. QYt (τ|Yt −1 , . . . , Y1 ) = φij (Yj )βij (τ), (1) Ytperturbed = Yt +  where t ∼ N (0, CSEMYt ) Yt −1perturbed = Yt −1 +  where t −1 ∼ N (0, CSEMYt −1 ) Precision estimates are derived for both cohort and anchored SGPs. Results are provided in the following figures and in Table 2 of the associated paper. Damian W. Betebenner On the Precision of MCAS SGPs
  9. SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 5 Std Error Growth Theta (Prior) SGP
  10. SGP Standard Error by SGP and Current Growth Theta SGP ELA Grade 5 Std Error Growth Theta (Current) SGP
  11. SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 6 Std Error Growth Theta (Prior) SGP
  12. SGP Standard Error by SGP and Current Growth Theta SGP ELA Grade 6 Std Error Growth Theta (Current) SGP
  13. SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 7 Std Error Growth Theta (Prior) SGP
  14. SGP Standard Error by SGP and Current Growth Theta SGP ELA Grade 7 Std Error Growth Theta (Current) SGP
  15. SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 8 Std Error Growth Theta (Prior) SGP
  16. SGP Standard Error by SGP and Current Growth Theta SGP ELA Grade 8 Std Error Growth Theta (Current) SGP
  17. SGP Standard Error by SGP and Prior Growth Theta SGP ELA Grade 10 Std Error Growth Theta (Prior) SGP
  18. SGP Standard Error by SGP and Current Growth Theta SGP ELA Grade 10 Std Error Growth Theta (Current) SGP
  19. SGP Standard Error by SGP and Prior Growth Theta Growth Theta (Prior) SGP
  20. SGP Standard Error by SGP and Current Growth Theta Growth Theta (Prior) SGP
  21. SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 5 Std Error Growth Theta (Prior) SGP
  22. SGP Standard Error by SGP and Current Growth Theta SGP MATHEMATICS Grade 5 Std Error Growth Theta (Current) SGP
  23. SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 6 Std Error Growth Theta (Prior) SGP
  24. SGP Standard Error by SGP and Current Growth Theta SGP MATHEMATICS Grade 6 Std Error Growth Theta (Current) SGP
  25. SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 7 Std Error Growth Theta (Prior) SGP
  26. SGP Standard Error by SGP and Current Growth Theta SGP MATHEMATICS Grade 7 Std Error Growth Theta (Current) SGP
  27. SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 8 Std Error Growth Theta (Prior) SGP
  28. SGP Standard Error by SGP and Current Growth Theta SGP MATHEMATICS Grade 8 Std Error Growth Theta (Current) SGP
  29. SGP Standard Error by SGP and Prior Growth Theta SGP MATHEMATICS Grade 10 Std Error Growth Theta (Prior) SGP
  30. SGP Standard Error by SGP and Current Growth Theta SGP MATHEMATICS Grade 10 Std Error Growth Theta (Current) SGP
  31. SGP Standard Error by SGP and Prior Growth Theta Growth Theta (Prior) SGP
  32. SGP Standard Error by SGP and Current Growth Theta Growth Theta (Prior) SGP
  33. Background References Appendix References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Betebenner, D. W., Van Iwaarden, A., and Domingue, B. (2013). SGP: An R Package for the Calculation and Visualization of Student Growth Percentiles & Percentile Growth Trajectories. R package version 1.0-4.0. R Development Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Damian W. Betebenner On the Precision of MCAS SGPs
  34. 20 core ca ca le le 280 S re 2012 co
  35. 20 core ca ca le le 280 S re 2012 co
  36. 20 core ca ca le le 280 S re 2012 co
  37. co re le ca ro th nt i le 12 ce er th 260 re co 240 le ca
  38. co re th S ro le le ca nt er ce S th 260 re co 240 le ca
  39. Peer 2 5 4 m i re ade Sco Ac cale 201 75% 75 th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 20 12 co
  40. Th is d 75th s: dat erive perc eer 2 5 4 acaa and from ntile c P = dem co a cu ic p vers ll the t m i re eer all ade Sco g ro Ac cale ups 201 75% 75 th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 20 12 co
  41. eer 2 2 6 cPe= Th m i r is d 75th ade Sco dat erive perc acaa and from ntile Ac cale dem co a cu ic p vers ll the t 1S eer all g ro ups th Per 200 nt i ce le G wth ro 20 core ca ca le le 280 S re 2012 co
  42. Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps 75 th Pe 20 r ce 0 nt il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 22 Sc
  43. Th is e dade 75 a c t a r iv t h a d a ed p emnd f er ic coro cmen peve atil errs ll e c gralltheut ou ps % 7 5 th P 20 75 er ce 0 nt 0 il e Gr ow th Sc 0 or al Sc Sc 0 0 al or 22 Sc