MCAS TO PARCC Considerations Regarding student growth during assessment transitions Damian W Betebenner National Center for the Improvement of Educational Assessment Cambridge, MA May 13th, 2014

GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher evaluation • School/District/State evaluation • Other research/diagnostic purposes

September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August

REVIEW: THREATS TO CALCULATION • MCAS and PARCC assessments are different assessments reported on different scales. • Longitudinal analysis techniques differ in their requirements regarding scales over time. Covariate Adjustment Models Growth Curve Modeling Less Stringent More Stringent • Conversion to z-scores (or other scale transformations) do not make the MCAS & PARCC results equivalent.

REVIEW: THREATS TO CALCULATION • Longitudinal analyses are impacted by different tests as long as the results are included either as dependent or independent variables. • That is, the results of students taking different exams are, in general, not comparable for either status or growth. • Other threats to longitudinal analyses include prominent ceiling/floor effects on the new exams. • Unlikely to see ceiling effects but floor effects are likely (probably not in Massachusetts).

REVIEW: FLAVORS OF GROWTH • SGP accommodates both Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs • Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • These growth metrics often have different data requirements. • SGP calculation is robust to scale transformations. It unlikely that the transition from the MCAS scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated for each content area and grade. • Massachusetts use of anchored SGPs will be impacted by the assessment transition and will force use of cohort referenced SGPs until sufficient data is available on the new scale.

REVIEW: THREATS TO CALCULATION • SGPs don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A significant ceiling/floor associated with a test (either PARCC or a state test used as a prior) can lead to bad model fit. These growth metrics often have different data requirements. • Barring significant ceiling/floor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.

REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate an SGP doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating SGP use will require defining the purpose. • Current state uses include: • Teacher evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school accreditation.

REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Criterion-referenced growth not available until year 2 of PARCC. • Assessment results from different assessments (state and PARCC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC field testing.

About this copy

MCAS to PARCC — Massachusetts Department of Education, May 13, 2014.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.