Growth During Transition

CCSSO · June 2014

Growth During Transition — CCSSO, June 2014.
Published

June 27, 2014

Event
Conference
Slides
18
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GROWTH DURING TRANSITION Damian W Betebenner National Center for the Improvement of Educational Assessment CCSSO National Conference on Student Assessment June 27, 2014 New Orleans, Louisiana
  2. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher & Leader evaluation • School/District/State evaluation • Other diagnostic purposes
  3. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth
  4. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Before: 2013-2014
  5. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1: 2014-2015
  6. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1I: 2015-2016
  7. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth After: 2016-2017
  8. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  9. September October November December January February March April May June July August SBAC Field Testing & Scoring SBAC Preliminary Standard Setting Assessments and Digital Summative Assessment Final Standard Setting Library Available Administration Summative Assessment Administration Summative Assessment Administration September October November December January February March April May June July August
  10. THE ASSESSMENT TRANSITION WHAT GETS “BROKEN” • Scales • If you had a vertical scale, no scale score subtraction in the transition year • Standards • Transition between achievement levels will have different meaning
  11. REVIEW: THREATS TO CALCULATION • Growth metrics requiring a vertical scale for calculation will not be possible in the transition year without some type of scale equating. • Most growth analyses (e.g., SGPs, VAM) don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A signi cant ceiling/ oor associated with a test (either SBAC/PARCC or a state test used as a prior) can lead to bad model t. These growth metrics often have different data requirements. • Barring signi cant ceiling/ oor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • Changing achievement levels will require states using value-tables to rede ne values 3 times in the coming years. fi fi fl fl fi fi
  12. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH • Annual growth? Bi-annual growth? Tri-annual growth? • Annual growth requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area). • Annual targets (Projections/Targets/Trajectories) require (at a minimum) two consecutive annual measures of student achievement from the same assessment (i.e., an assessment with a time-invariant scale) or scales that are equated such that they can be treated as equivalent.
  13. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH • States primarily focus on annual growth, particularly for accountability (exceptions include, for example, student growth from 8th to 10th grade). • PARCC eld testing will impact a state’s ability to calculate growth for students due to the simple fact that student will not receive back scale scores on the PARCC eld test. • Absent double testing, students participating in the eld test cannot receive annual student growth in BOTH 2013-2014 (no current year score) AND 2014-2015 (no prior year score). • If a student has a scale score in 2012-2013, one could, for example, calculate a two year growth (e.g., a two year SGP) for a students participating in eld testing between 2012-2013 and 2014-2015. • In the years after the transition, absent double testing, students will have different numbers of priors available for growth analyses. fi fi fi fi
  14. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SUMMARY • Easier to conduct norm-referenced analyses during transition than criterion- referenced growth-to-standard analyses. • Criterion-referenced projections require linking analyses done exclusively on the PARCC/SBAC assessment in order to extrapolate forward how much growth is required to, for example catch-up/keep-up/move-up/stay-up. • Analogously growth-to-standard (e.g., career and college-readiness) will require at least two years data on the new scale as well as xed achievement levels/ standards de ned on the new scale. • Establishing baseline referenced growth will likely require even more time (e.g., 4 years) to establish a stable baseline from which to compare going forward. fi fi
  15. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate student growth doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating growth use will require de ning the purpose. • Current uses include: • Teacher and leader evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school contracts. • Improvement planning fi
  16. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Without equating of state test to consortium test, projection/trajectory based criterion-referenced growth not available until year 2 of PARCC/ SBAC. • Assessment results from different assessments (state and PARCC/SBAC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC/SBAC eld testing. fi
  17. REQUIREMENTS FOR THE USE OF GROWTH • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Most accountability systems use an observational pre-post design to hold teachers/schools/districts accountable. • Often, the pre-test occurs the spring prior to enrollment in the unit of accountability and the post-test the spring during enrollment. • An extended growth time span (e.g., two-year) undermines the attribution of responsibility implicit within the accountability determination.
  18. Questions?