GROWTH DURING TRANSITION
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Technical Issues in Large Scale Assessment (TILSA)
February 5, 2014 Louisville, Kentucky
Wednesday, February 5, 14
ABOUT ME
• Senior Associate Center for Assessment
• Developed Student Growth Percentiles with the Colorado
Department of Education to help them (and other states)
make sense of their longitudinal data
• Developer of open source tools to help analyze and
visualize longitudinal student assessment data.
• Proud first time father of a 9 month old daughter
Wednesday, February 5, 14
GROWTH DURING TRANSITION:
TWO DISTINCT CHALLENGES
• Challenges to the calculation of student growth.
• Challenges to the use of student growth for:
• Teacher & Leader evaluation
• School/District/State evaluation
• Other diagnostic purposes
Wednesday, February 5, 14
A PLETHORA OF ACCOUNTABILITY METRICS
TRANSITION IMPACTS THEM ALL
• Status/Achievement: Percent proficient
• Improvement: Change in percent proficient from year to year
• Student Growth: Change in individual student achievement
from year to year
Wednesday, February 5, 14
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
Wednesday, February 5, 14
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
Before: 2013-2014
Wednesday, February 5, 14
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1: 2014-2015
Wednesday, February 5, 14
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1I: 2015-2016
Wednesday, February 5, 14
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
After: 2016-2017
Wednesday, February 5, 14
September October November December January February March April May June July August
PARCC PBA PARCC EOY
Field Testing Field Testing
PBA EOY Standard
Window Window Setting
2016 Scores &
2015 Scores PBA EOY
Data Files
Returned Window Window
Returned
PBA EOY 2017 Scores &
Window Window Data Files
Returned
September October November December January February March April May June July August
Wednesday, February 5, 14
September October November December January February March April May June July August
SBAC Field Testing & Scoring SBAC Preliminary Standard
Setting
Assessments and Digital Summative Assessment Final Standard Setting
Library Available Administration
Summative Assessment
Administration
Summative Assessment
Administration
September October November December January February March April May June July August
Wednesday, February 5, 14
THE ASSESSMENT TRANSITION
WHAT GETS “BROKEN”
• Scales
• If you had a vertical scale, no scale score subtraction in the
transition year
• Standards
• Transition between achievement levels will have different
meaning
Wednesday, February 5, 14
REVIEW: FLAVORS OF GROWTH
• Norm- and Criterion-Referenced growth
• Norm-Referenced: Cohort referenced SGPs, Value-Added models
• Criterion-Referenced: Value tables, baseline referenced/anchored SGPs, student growth
projections/targets (i.e., growth to standard), catch-up/keep-up/move-up/stay-up.
• Based upon state preference, policy, and data, different states have created
different combinations of norm- and criterion-referenced growth metrics.
• These growth metrics often have different data requirements.
• In general, norm-referenced growth analyses tolerate assessment change
whereas criterion-referenced growth analyses require.
Wednesday, February 5, 14
Advanced Reading
● Move
●● Up Achievement
Target
DEMO Reading
● Scale Score
● Keep Up
Target
Catch Up ●
Growth
Proficient ● Target
Level Percentiles
● Very High 81st − 99th
High 61st − 80th
Part Proficient Typical 40th − 60th
Low 20th − 39th
Very Low 1st − 19th
Unsatisfactory
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7
Scale Score 449 533 579 602
Achievement Level
Achievement Target
Unsatisfactory Part Proficient Part Proficient Proficient
Proficient (592) Proficient (620)/Advanced (661)
Achievement
Growth Percentile 69 85 74
Growth Level
Growth Target
High Very High High
Catch Up (60) Keep Up (76)/Move Up (99)
Growth
Wednesday, February 5, 14
Demonstration: 2012−2013 Mathematics
Norm & Criterion Referenced Growth & Achievement
Baseline Referenced Percentile Growth Trajectory
Scale Score
2012−2013 Achievement Percentile
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10
Wednesday, February 5, 14
REVIEW: THREATS TO CALCULATION
• Growth metrics requiring a vertical scale for calculation will not be possible in
the transition year without some type of scale equating.
• Most growth analyses (e.g., SGPs, VAM) don’t require a vertical scale for
calculation. Regardless of whether PARCC uses a vertical scale, SGPs can be
calculated using both vertical and non-vertical scales.
• A significant ceiling/floor associated with a test (either SBAC/PARCC or a state
test used as a prior) can lead to bad model fit. These growth metrics often
have different data requirements.
• Barring significant ceiling/floor, norm-referenced analyses will very likely yield a
uniform distribution of percentiles.
• Changing achievement levels will require states using value-tables to redefine
values 3 times in the coming years.
Wednesday, February 5, 14
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH
• Annual growth? Bi-annual growth? Tri-annual growth?
• Annual growth requires (at a minimum) two consecutive annual measures (i.e.,
scale scores) of student achievement (usually taken from the same content
area).
• Annual targets (Projections/Targets/Trajectories) require (at a minimum) two
consecutive annual measures of student achievement from the same
assessment (i.e., an assessment with a time-invariant scale) or scales that are
equated such that they can be treated as equivalent.
Wednesday, February 5, 14
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH
• States primarily focus on annual growth, particularly for accountability
(exceptions include, for example, student growth from 8th to 10th grade).
• PARCC field testing will impact a state’s ability to calculate growth for students
due to the simple fact that student will not receive back scale scores on the
PARCC field test.
• Absent double testing, students participating in the field test cannot receive
annual student growth in BOTH 2013-2014 (no current year score) AND
2014-2015 (no prior year score).
• If a student has a scale score in 2012-2013, one could, for example, calculate a
two year growth (e.g., a two year SGP) for a students participating in field
testing between 2012-2013 and 2014-2015.
• In the years after the transition, absent double testing, students will have
different numbers of priors available for growth analyses.
Wednesday, February 5, 14
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH: SUMMARY
• Easier to conduct norm-referenced analyses during transition than criterion-
referenced growth-to-standard analyses.
• Criterion-referenced projections require linking analyses done exclusively on
the PARCC/SBAC assessment in order to extrapolate forward how much
growth is required to, for example catch-up/keep-up/move-up/stay-up.
• Analogously growth-to-standard (e.g., career and college-readiness) will require
at least two years data on the new scale as well as fixed achievement levels/
standards defined on the new scale.
• Establishing baseline referenced growth will likely require even more time (e.g.,
4 years) to establish a stable baseline from which to compare going forward.
Wednesday, February 5, 14
September October November December January February March April May June July August
PARCC PBA PARCC EOY
Field Testing Field Testing
PBA EOY Standard
Window Window Setting
2016 Scores &
2015 Scores PBA EOY
Data Files
Returned Window Window
Returned
PBA EOY 2017 Scores &
Window Window Data Files
Returned
September October November December January February March April May June July August
Wednesday, February 5, 14
September October November December January February March April May June July August
SBAC Field Testing & Scoring SBAC Preliminary Standard
Setting
Assessments and Digital Summative Assessment Final Standard Setting
Library Available Administration
Summative Assessment
Administration
Summative Assessment
Administration
September October November December January February March April May June July August
Wednesday, February 5, 14
REQUIREMENTS FOR THE USE OF GROWTH
• Two distinct challenges: Calculation and Use
• Just because we can calculate student growth doesn’t mean it is defensible to
use it for previously existing accountability purposes.
• Validating growth use will require defining the purpose.
• Current uses include:
• Teacher and leader evaluation (required by RttT, ESEA waiver, state policy/rule making, ...)
• School and District accountability (ESEA waiver, state policy, ...)
• Charter school contracts.
• Improvement planning
Wednesday, February 5, 14
REQUIREMENTS FOR THE USE OF GROWTH
• During the assessment transition, a number of challenges confront states using
growth for accountability purposes.
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Changes in types of growth analyses done for accountability during
transition. Without equating of state test to consortium test, projection/
trajectory based criterion-referenced growth not available until year 2 of
PARCC/SBAC.
• Assessment results from different assessments (state and PARCC/SBAC).
Depending upon the number of priors used by the state, this will be an
ongoing issue for many years.
• Missing data due to PARCC/SBAC field testing.
Wednesday, February 5, 14
REQUIREMENTS FOR THE USE OF GROWTH
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Most accountability systems use an observational pre-post design to hold
teachers/schools/districts accountable.
• Often, the pre-test occurs the spring prior to enrollment in the unit of
accountability and the post-test the spring during enrollment.
• An extended growth time span (e.g., two-year) undermines the attribution
of responsibility implicit within the accountability determination.
Wednesday, February 5, 14
Wednesday, February 5, 14
Questions?
Wednesday, February 5, 14
About this copy
Growth During Transition — CCSSO TILSA, February 2014.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.