Growth During Transition

CCSSO · December 2013

Growth During Transition — CCSSO, December 2013.
Published

December 17, 2013

Event
Conference
Slides
20
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GROWTH DURING TRANSITION Damian W Betebenner National Center for the Improvement of Educational Assessment CCSSO Colloquium on Growth During Transition of the PARCC Assessment States
  2. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher evaluation • School/District/State evaluation • Other diagnostic purposes
  3. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth
  4. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Before: 2013-2014
  5. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1: 2014-2015
  6. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1I: 2015-2016
  7. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth After: 2016-2017
  8. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  9. REVIEW: FLAVORS OF GROWTH • SGP accommodates both Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs • Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • Based upon state preference, policy, and data, different states have created different combinations of norm- and criterion-referenced growth metrics. • These growth metrics often have different data requirements. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.
  10. Advanced Reading ● Move ●● Up Achievement Target DEMO Reading ● Scale Score ● Keep Up Target Catch Up ● ● Growth Proficient ● Target Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Unsatisfactory Very Low 1st − 19th Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (592) Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Catch Up (60) Keep Up (76)/Move Up (99) Growth
  11. REVIEW: THREATS TO CALCULATION • SGPs don’t require a vertical scale for calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be calculated using both vertical and non-vertical scales. • A signi cant ceiling/ oor associated with a test (either PARCC or a state test used as a prior) can lead to bad model t. These growth metrics often have different data requirements. • Barring signi cant ceiling/ oor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a PARCC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis. fi fl fl fi fi
  12. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • Annual growth? Bi-annual growth? Tri-annual growth? • Annual growth (norm-referenced SGPs based upon a single cohort) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area). • Annual growth (anchored/baseline-referenced SGPs) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area) from the same assessment (i.e., an assessment with a time-invariant scale). • Annual targets (SGP Projections/Targets/Trajectories) require (at a minimum) two consecutive annual measures of student achievement from the same assessment (i.e., an assessment with a time-invariant scale).
  13. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • States primarily focus on annual growth, particularly for accountability (exceptions include, for example, student growth from 8th to 10th grade). • PARCC eld testing will impact a state’s ability to calculate growth for students due to the simple fact that student will not receive back scale scores on the PARCC eld test. • Absent double testing, students participating in the eld test cannot receive annual SGPs in BOTH 2013-2014 (no current year score) AND 2014-2015 (no prior year score). • If a student has a scale score in 2012-2013, one could, for example, calculate a two year SGP for a students participating in eld testing between 2012-2013 and 2014-2015. • In the years after the transition, absent double testing, students will have different numbers of priors available for SGP analyses. fi fi fi fi
  14. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SUMMARY • Easier to conduct norm-referenced SGP analyses during transition than criterion-referenced growth to standard analyses. • Criterion-referenced projections require linking SGP analyses done exclusively on the PARCC assessment in order to extrapolate forward how much growth is required to, for example catch-up/keep-up/move-up/stay-up. • Analogously growth-to-standard (e.g., career and college-readiness) will require at least two years data on the new scale as well as xed achievement levels/ standards de ned on the new scale. • Establishing baseline referenced SGPs will likely require even more time (e.g., 4 years) to establish a stable baseline from which to compare going forward. fi fi
  15. September October November December January February March April May June July August PARCC PBA PARCC EOY Field Testing Field Testing PBA EOY Standard Window Window Setting 2016 Scores & 2015 Scores PBA EOY Data Files Returned Window Window Returned PBA EOY 2017 Scores & Window Window Data Files Returned September October November December January February March April May June July August
  16. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate an SGP doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating SGP use will require de ning the purpose. • Current state uses include: • Teacher evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school accreditation. fi
  17. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Criterion-referenced growth not available until year 2 of PARCC. • Assessment results from different assessments (state and PARCC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to PARCC eld testing. fi
  18. REQUIREMENTS FOR THE USE OF GROWTH • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Most accountability systems use an observational pre-post design to hold teachers/schools/districts accountable. • Often, the pre-test occurs the spring prior to enrollment in the unit of accountability and the post-test the spring during enrollment. • An extended growth time span (e.g., two-year) undermines the attribution of responsibility implicit within the accountability determination. • Mis t between accountability and growth time span impact school and district accountability as well, but likely will have less impact because eld testing does not occur school wide. fi fi
  20. Questions?