MCAS to PARCC

Massachusetts Department of Education · May 13, 2014

MCAS to PARCC — Massachusetts Department of Education, May 13, 2014.
Published

May 13, 2014

Event
State agency
Slides
10
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. MCAS TO PARCC Considerations Regarding student growth during assessment transitions Damian W Betebenner National Center for the Improvement of Educational Assessment Cambridge, MA May 13th, 2014
  2. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher evaluation • School/District/State evaluation • Other research/diagnostic purposes
  3. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  4. REVIEW: THREATS TO CALCULATION • MCAS and PARCC assessments are different assessments reported on different scales. • Longitudinal analysis techniques differ in their requirements regarding scales over time. Covariate Adjustment Models Growth Curve Modeling Less Stringent More Stringent • Conversion to z-scores (or other scale transformations) do not make the MCAS & PARCC results equivalent.
  5. REVIEW: THREATS TO CALCULATION • Longitudinal analyses are impacted by different tests as long as the results are included either as dependent or independent variables. • That is, the results of students taking different exams are, in general, not comparable for either status or growth. • Other threats to longitudinal analyses include prominent ceiling/floor effects on the new exams. • Unlikely to see ceiling effects but floor effects are likely (probably not in Massachusetts).
  6. REVIEW: FLAVORS OF GROWTH • SGP accommodates both Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs • Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • These growth metrics often have different data requirements. • SGP calculation is robust to scale transformations. It unlikely that the transition from the MCAS scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated for each content area and grade. • Massachusetts use of anchored SGPs will be impacted by the assessment transition and will force use of cohort referenced SGPs until sufficient data is available on the new scale.
  7. REVIEW: THREATS TO CALCULATION • SGPs don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A significant ceiling/floor associated with a test (either PARCC or a state test used as a prior) can lead to bad model fit. These growth metrics often have different data requirements. • Barring significant ceiling/floor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.
  8. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate an SGP doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating SGP use will require defining the purpose. • Current state uses include: • Teacher evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school accreditation.
  9. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Criterion-referenced growth not available until year 2 of PARCC. • Assessment results from different assessments (state and PARCC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC field testing.