Growth During Transition

CCSSO · 2014

Growth During Transition — CCSSO, 2014.
Published

January 1, 2014

Event
Conference
Slides
20
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. GROWTH DURING TRANSITION Damian W Betebenner National Center for the Improvement of Educational Assessment CCSSO Colloquium on Growth During Transition of the SBAC Assessment States
  2. GROWTH DURING TRANSITION: TWO DISTINCT CHALLENGES • Challenges to the calculation of student growth. • Challenges to the use of student growth for: • Teacher evaluation • School/District/State evaluation • Other diagnostic purposes
  3. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth
  4. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth Before: 2013-2014
  5. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1: 2014-2015
  6. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth During 1I: 2015-2016
  7. THE TRANSITION “SPEED BUMP” (At least) 4 stages to traverse when considering growth After: 2016-2017
  8. September October November December January February March April May June July August Field Testing & Scoring of Preliminary Standard Setting summative and interim items Assessments and Digital Summative Assessment Final Standard Setting Library Available Administration Summative Assessment Administration Summative Assessment Administration September October November December January February March April May June July August
  9. REVIEW: FLAVORS OF GROWTH • SGP accommodates both Norm- and Criterion-Referenced growth • Norm-Referenced: Cohort referenced SGPs • Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up. • Based upon state preference, policy, and data, different states have created different combinations of norm- and criterion-referenced growth metrics. • These growth metrics often have different data requirements. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a SBAC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.
  11. REVIEW: THREATS TO CALCULATION • SGPs don’t require a vertical scale for calculation (which SBAC has by design). SGPs can be calculated using both vertical and non-vertical scales. • A significant ceiling/floor associated with a test (either SBAC or a state test used as a prior) can lead to bad model fit. These growth metrics often have different data requirements. • Barring significant ceiling/floor, norm-referenced analyses will very likely yield a uniform distribution of percentiles. • SGP calculation is robust to scale transformations. It unlikely that the transition from a state assessment scale to a SBAC scale will prevent the calculation of an SGP. However, this has to be validated on a state by state, content area by content area, and grade by grade basis.
  12. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • Annual growth? Bi-annual growth? Tri-annual growth? • Annual growth (norm-referenced SGPs based upon a single cohort) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area). • Annual growth (anchored/baseline-referenced SGPs) requires (at a minimum) two consecutive annual measures (i.e., scale scores) of student achievement (usually taken from the same content area) from the same assessment (i.e., an assessment with a time-invariant scale). • Annual targets (SGP Projections/Targets/Trajectories) require (at a minimum) two consecutive annual measures of student achievement from the same assessment (i.e., an assessment with a time-invariant scale).
  13. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SGP • States primarily focus on annual growth, particularly for accountability (exceptions include, for example, student growth from 8th to 10th grade). • SBAC field testing will impact a state’s ability to calculate growth for students due to the simple fact that student will not receive back scale scores on the SBAC field test. • Absent double testing, students participating in the field test cannot receive annual SGPs in BOTH 2013-2014 (no current year score) AND 2014-2015 (no prior year score). • If a student has a scale score in 2012-2013, one could, for example, calculate a two year SGP for a students participating in field testing between 2012-2013 and • In the years after the transition, absent double testing, students will have different numbers of priors available for SGP analyses.
  14. REVIEW: REQUIREMENTS FOR CALCULATING GROWTH: SUMMARY • Easier to conduct norm-referenced SGP analyses during transition than criterion-referenced growth to standard analyses. • Criterion-referenced projections require linking SGP analyses done exclusively on the SBAC assessment in order to extrapolate forward how much growth is required to, for example catch-up/keep-up/move-up/stay-up. • Analogously growth-to-standard (e.g., career and college-readiness) will require at least two years data on the new scale as well as fixed achievement levels/ standards defined on the new scale. • Establishing baseline referenced SGPs will likely require even more time (e.g., 4 years) to establish a stable baseline from which to compare going forward.
  15. September October November December January February March April May June July August Field Testing & Scoring of Preliminary Standard Setting summative and interim items Assessments and Digital Summative Assessment Final Standard Setting Library Available Administration Summative Assessment Administration Summative Assessment Administration September October November December January February March April May June July August
  16. REQUIREMENTS FOR THE USE OF GROWTH • Two distinct challenges: Calculation and Use • Just because we can calculate an SGP doesn’t mean it is defensible to use it for previously existing accountability purposes. • Validating SGP use will require defining the purpose. • Current state uses include: • Teacher evaluation (required by RttT, ESEA waiver, state policy/rule making, ...) • School and District accountability (ESEA waiver, state policy, ...) • Charter school accreditation.
  17. REQUIREMENTS FOR THE USE OF GROWTH • During the assessment transition, a number of challenges confront states using growth for accountability purposes. • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Changes in types of growth analyses done for accountability during transition. Criterion-referenced growth not available until year 2 of SBAC. • Assessment results from different assessments (state and SBAC). Depending upon the number of priors used by the state, this will be an ongoing issue for many years. • Missing data due to SBAC field testing.
  18. REQUIREMENTS FOR THE USE OF GROWTH • Data time span (e.g., two-year growth) not aligned with accountability time span (annual) • Most accountability systems use an observational pre-post design to hold teachers/schools/districts accountable. • Often, the pre-test occurs the spring prior to enrollment in the unit of accountability and the post-test the spring during enrollment. • An extended growth time span (e.g., two-year) undermines the attribution of responsibility implicit within the accountability determination. • Misfit between accountability and growth time span impact school and district accountability as well, but likely will have less impact because field testing does not occur school wide.
  20. Questions?