Growth During Transition

CCSSO TILSA · February 2014

Growth During Transition — CCSSO TILSA, February 2014.
Published

February 5, 2014

Event
Conference
Slides
26
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GROWTH DURING TRANSITION Damian W Betebenner National Center for the Improvement of Educational Assessment Technical Issues in Large Scale Assessment (TILSA) February 5, 2014 Louisville, Kentucky Wednesday, February 5, 14
  2. ABOUT ME • Senior Associate Center for Assessment • Developed Student Growth Percentiles with the Colorado Department of Education to help them (and other states) make sense of their longitudinal data • Developer of open source tools to help analyze and visualize longitudinal student assessment data. • Proud first time father of a 9 month old daughter Wednesday, February 5, 14
  3. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher & Leader evaluation • School/District/State evaluation • Other diagnostic purposes Wednesday, February 5, 14
  4. A PLETHORA OF ACCOUNTABILITY METRICS TRANSITION IMPACTS THEM ALL • Status/Achievement: Percent proficient • Improvement: Change in percent proficient from year to year • Student Growth: Change in individual student achievement from year to year Wednesday, February 5, 14
  5. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Wednesday, February 5, 14
  6. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Before: 2013-2014 Wednesday, February 5, 14
  7. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1: 2014-2015 Wednesday, February 5, 14
  8. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1I: 2015-2016 Wednesday, February 5, 14
  9. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth After: 2016-2017 Wednesday, February 5, 14
  10. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August Wednesday, February 5, 14
  11. September October November December January February March April May June July August SBAC Field Testing & Scoring SBAC Preliminary Standard Setting Assessments and Digital Summative Assessment Final Standard Setting Library Available Administration Summative Assessment Administration Summative Assessment Administration September October November December January February March April May June July August Wednesday, February 5, 14
  12. THE ASSESSMENT TRANSITION WHAT GETS “BROKEN” • Scales • If you had a vertical scale, no scale score subtraction in the transition year • Standards • Transition between achievement levels will have different meaning Wednesday, February 5, 14
  13. REVIEW: FLAVORS OF GROWTH • Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs, Value-Added models • Criterion-Referenced: Value tables, baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • Based upon state preference, policy, and data, different states have created different combinations of norm- and criterion-referenced growth metrics. • These growth metrics often have different data requirements. • In general, norm-referenced growth analyses tolerate assessment change whereas criterion-referenced growth analyses require. Wednesday, February 5, 14
  14. Advanced Reading ● Move ●● Up Achievement Target DEMO Reading ● Scale Score ● Keep Up Target Catch Up ● Growth Proficient ● Target Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (592) Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Catch Up (60) Keep Up (76)/Move Up (99) Growth Wednesday, February 5, 14
  15. Demonstration: 2012−2013 Mathematics Norm & Criterion Referenced Growth & Achievement Baseline Referenced Percentile Growth Trajectory Scale Score 2012−2013 Achievement Percentile Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Wednesday, February 5, 14
  16. REVIEW: THREATS TO CALCULATION • Growth metrics requiring a vertical scale for calculation will not be possible in the transition year without some type of scale equating. • Most growth analyses (e.g., SGPs, VAM) don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A significant ceiling/floor associated with a test (either SBAC/PARCC or a state test used as a prior) can lead to bad model fit. These growth metrics often have different data requirements. • Barring significant ceiling/floor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • Changing achievement levels will require states using value-tables to redefine values 3 times in the coming years. Wednesday, February 5, 14
  17. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH • Annual growth? Bi-annual growth? Tri-annual growth? • Annual growth requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area). • Annual targets (Projections/Targets/Trajectories) require (at a minimum) two consecutive annual measures of student achievement from the same assessment (i.e., an assessment with a time-invariant scale) or scales that are equated such that they can be treated as equivalent. Wednesday, February 5, 14
  18. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH • States primarily focus on annual growth, particularly for accountability (exceptions include, for example, student growth from 8th to 10th grade). • PARCC field testing will impact a state’s ability to calculate growth for students due to the simple fact that student will not receive back scale scores on the PARCC field test. • Absent double testing, students participating in the field test cannot receive annual student growth in BOTH 2013-2014 (no current year score) AND 2014-2015 (no prior year score). • If a student has a scale score in 2012-2013, one could, for example, calculate a two year growth (e.g., a two year SGP) for a students participating in field testing between 2012-2013 and 2014-2015. • In the years after the transition, absent double testing, students will have different numbers of priors available for growth analyses. Wednesday, February 5, 14
  19. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SUMMARY • Easier to conduct norm-referenced analyses during transition than criterion- referenced growth-to-standard analyses. • Criterion-referenced projections require linking analyses done exclusively on the PARCC/SBAC assessment in order to extrapolate forward how much growth is required to, for example catch-up/keep-up/move-up/stay-up. • Analogously growth-to-standard (e.g., career and college-readiness) will require at least two years data on the new scale as well as fixed achievement levels/ standards defined on the new scale. • Establishing baseline referenced growth will likely require even more time (e.g., 4 years) to establish a stable baseline from which to compare going forward. Wednesday, February 5, 14
  20. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August Wednesday, February 5, 14
  21. September October November December January February March April May June July August SBAC Field Testing & Scoring SBAC Preliminary Standard Setting Assessments and Digital Summative Assessment Final Standard Setting Library Available Administration Summative Assessment Administration Summative Assessment Administration September October November December January February March April May June July August Wednesday, February 5, 14
  22. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate student growth doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating growth use will require defining the purpose. • Current uses include: • Teacher and leader evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school contracts. • Improvement planning Wednesday, February 5, 14
  23. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Without equating of state test to consortium test, projection/ trajectory based criterion-referenced growth not available until year 2 of PARCC/SBAC. • Assessment results from different assessments (state and PARCC/SBAC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC/SBAC field testing. Wednesday, February 5, 14
  24. REQUIREMENTS FOR THE USE OF GROWTH • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Most accountability systems use an observational pre-post design to hold teachers/schools/districts accountable. • Often, the pre-test occurs the spring prior to enrollment in the unit of accountability and the post-test the spring during enrollment. • An extended growth time span (e.g., two-year) undermines the attribution of responsibility implicit within the accountability determination. Wednesday, February 5, 14
  25. Wednesday, February 5, 14
  26. Questions? Wednesday, February 5, 14