MASSACHUSETTS A DISCUSSION OF STUDENT GROWTH Damian W Betebenner National Center for the Improvement of Educational Assessment Malden, MA June 14th, 2016

ISSUES/QUESTIONS • Caveats/concerns about how we should and should not use the growth data for 2016, for reporting, accountability, and for other purposes (such as research). • What will be possible to do for growth for the 2017 test year, since that will be our first year under the new accountability system. Will we be able to calculate growth at all? Will it only be comparable within test history streams (e.g., MCAS 2015 -> PARCC 2016 -> NGMCAS 2017 vs. PARCC 2015 -> PARCC 2016 -> NGMCAS 2017)? • Thoughts/options regarding “growth to standard” or other ways of getting at growth that would help us make comparisons across schools for accountability purposes that feel fairer to stakeholders without setting different growth expectations for different subgroups of students • If we are going to continue using SGPs we need to overcome the rhetoric that the metric has been widely discredited. Thoughts?

SGP/GROWTH BASICS • Subtraction isn’t a growth model. • Give anyone two numbers and ask them about “growth”, and they revert to a 4th grader and subtract the numbers — regardless of whether the scores are on a vertical scale. • SGPs are uncorrelated with the prior student achievement. I.e., regardless of where a student starts, they have an equal chance of demonstrating low/high growth. • SGPs are uncorrelated from year to year. I.e., low/high growth for a student in one year doesn’t tell anything about growth in other years. • Despite individual SGPs being uncorrelated with prior achievement, that doesn’t imply aggregates of SGPs (e.g., median SGP at school level) are uncorrelated with prior achievement. • Median SGPs can be correlated with prior achievement because of actual differences in schools, measurement error. • Correlations in MA very low (~0.2) compared to some other states (~0.2 to ~0.5).

CAVEATS/CONCERNS ABOUT HOW WE SHOULD AND SHOULD NOT USE THE GROWTH DATA FOR 2016, FOR REPORTING, ACCOUNTABILITY, AND FOR OTHER PURPOSES. • There is (should be) a 1-1 correspondence between caveats/concerns for using growth data as for using status data. • Scale based approach to running growth analyses in current year is the best (least worst) option available to MA to create comparable results across groups taking different tests. • Mode effect will continue to be a concern. The Center will be investigating mode effects in early July with Rhode Island.

What will be possible to do for growth for the 2017 test year? • The current solution for 2016 is to use scaling to “make interchangeable” MCAS and PARCC results for 2015 and 2016. • If that works, then the transition to NGMCAS will be greatly simplified. • With mode effects in 2015 and the possibility of them in 2016, the specter of mode effects will continue to lurk.

Thoughts/options regarding “growth to standard” or other ways of getting at growth that would help us make comparisons across schools for accountability purposes that feel fairer to stakeholders without setting different growth expectations for different subgroups of students • Important to distinguish differential impact, bias and fairness. • Norm- and criterion referenced growth are opposite sides of the same coin. • Criterion-referenced growth is usually harder for lower achieving schools to reach and easier for high achieving schools (those that start behind generally stay behind and those that start ahead generally stay ahead). • Criterion-referenced growth more important in when policy cannot tolerate “growth to nowhere” (e.g., ELP testing).

Thoughts/options regarding “growth to standard” or other ways of getting at growth that would help us make comparisons across schools for accountability purposes that feel fairer to stakeholders without setting different growth expectations for different subgroups of students • Adjustments (note, growth is just adjusted/conditional status) • Individual level with individual level adjustment — different norm-groups/different individual level expectations • Individual level with group adjustment — different norm-groups/different individual level expectations • Group level with group adjustment — 2nd stage adjustment/different group level expectations

BASEBALL: PAYROLL VS. AVG WINS

If we are going to continue using SGPs we need to overcome the rhetoric that the metric has been wid • Here’s a two x two matrix in which to locate criticisms: Individual Group Accuracy ASA/AERA Precision Sireci ASA/AERA • Different uses and different criticisms. Sireci’s criticisms are usually directed at the individual yet recently is now talking about group/teacher level growth. • Messaging of use should depend upon different uses (parent report for use for accountability).

About this copy

Massachusetts a Discussion of Student Growth — Massachusetts Department of Education, June 16, 2016.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.