GROWTH DURING TRANSITION
Damian W Betebenner
National Center for the Improvement of Educational Assessment
CCSSO National Conference on Student Assessment
June 27, 2014 New Orleans, Louisiana
GROWTH DURING TRANSITION:
TWO DISTINCT CHALLENGES
• Challenges to the calculation of student growth.
• Challenges to the use of student growth for:
• Teacher & Leader evaluation
• School/District/State evaluation
• Other diagnostic purposes
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
Before: 2013-2014
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1: 2014-2015
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1I: 2015-2016
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
After: 2016-2017
September October November December January February March April May June July August
PARCC PBA PARCC EOY
Field Testing Field Testing
PBA EOY Standard
Window Window Setting
2016 Scores &
2015 Scores PBA EOY
Data Files
Returned Window Window
Returned
PBA EOY 2017 Scores &
Window Window Data Files
Returned
September October November December January February March April May June July August
September October November December January February March April May June July August
SBAC Field Testing & Scoring SBAC Preliminary Standard
Setting
Assessments and Digital Summative Assessment Final Standard Setting
Library Available Administration
Summative Assessment
Administration
Summative Assessment
Administration
September October November December January February March April May June July August
THE ASSESSMENT TRANSITION
WHAT GETS “BROKEN”
• Scales
• If you had a vertical scale, no scale score subtraction in the
transition year
• Standards
• Transition between achievement levels will have different
meaning
REVIEW: THREATS TO CALCULATION
• Growth metrics requiring a vertical scale for calculation will not be possible in
the transition year without some type of scale equating.
• Most growth analyses (e.g., SGPs, VAM) don’t require a vertical scale for
calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be
calculated using both vertical and non-vertical scales.
• A signi cant ceiling/ oor associated with a test (either SBAC/PARCC or a state
test used as a prior) can lead to bad model t. These growth metrics often
have different data requirements.
• Barring signi cant ceiling/ oor, norm-referenced analyses will very likely yield a
uniform distribution of percentiles.
• Changing achievement levels will require states using value-tables to rede ne
values 3 times in the coming years.
fi
fi
fl
fl
fi
fi
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH
• Annual growth? Bi-annual growth? Tri-annual growth?
• Annual growth requires (at a minimum) two consecutive annual measures (i.e.,
scale scores) of student achievement (usually taken from the same content
area).
• Annual targets (Projections/Targets/Trajectories) require (at a minimum) two
consecutive annual measures of student achievement from the same
assessment (i.e., an assessment with a time-invariant scale) or scales that are
equated such that they can be treated as equivalent.
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH
• States primarily focus on annual growth, particularly for accountability
(exceptions include, for example, student growth from 8th to 10th grade).
• PARCC eld testing will impact a state’s ability to calculate growth for students
due to the simple fact that student will not receive back scale scores on the
PARCC eld test.
• Absent double testing, students participating in the
eld test cannot receive
annual student growth in BOTH 2013-2014 (no current year score) AND
2014-2015 (no prior year score).
• If a student has a scale score in 2012-2013, one could, for example, calculate a
two year growth (e.g., a two year SGP) for a students participating in eld
testing between 2012-2013 and 2014-2015.
• In the years after the transition, absent double testing, students will have
different numbers of priors available for growth analyses.
fi
fi
fi
fi
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH: SUMMARY
• Easier to conduct norm-referenced analyses during transition than criterion-
referenced growth-to-standard analyses.
• Criterion-referenced projections require linking analyses done exclusively on
the PARCC/SBAC assessment in order to extrapolate forward how much
growth is required to, for example catch-up/keep-up/move-up/stay-up.
• Analogously growth-to-standard (e.g., career and college-readiness) will require
at least two years data on the new scale as well as xed achievement levels/
standards de ned on the new scale.
• Establishing baseline referenced growth will likely require even more time (e.g.,
4 years) to establish a stable baseline from which to compare going forward.
fi
fi
REQUIREMENTS FOR THE USE OF GROWTH
• Two distinct challenges: Calculation and Use
• Just because we can calculate student growth doesn’t mean it is defensible to
use it for previously existing accountability purposes.
• Validating growth use will require de ning the purpose.
• Current uses include:
• Teacher and leader evaluation (required by RttT, ESEA waiver, state policy/rule making, ...)
• School and District accountability (ESEA waiver, state policy, ...)
• Charter school contracts.
• Improvement planning
fi
REQUIREMENTS FOR THE USE OF GROWTH
• During the assessment transition, a number of challenges confront states using
growth for accountability purposes.
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Changes in types of growth analyses done for accountability during transition.
Without equating of state test to consortium test, projection/trajectory
based criterion-referenced growth not available until year 2 of PARCC/
SBAC.
• Assessment results from different assessments (state and PARCC/SBAC).
Depending upon the number of priors used by the state, this will be an
ongoing issue for many years.
• Missing data due to PARCC/SBAC eld testing.
fi
REQUIREMENTS FOR THE USE OF GROWTH
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Most accountability systems use an observational pre-post design to hold
teachers/schools/districts accountable.
• Often, the pre-test occurs the spring prior to enrollment in the unit of
accountability and the post-test the spring during enrollment.
• An extended growth time span (e.g., two-year) undermines the attribution
of responsibility implicit within the accountability determination.
About this copy
Growth During Transition — CCSSO, June 2014.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.