GROWTH DURING TRANSITION
Damian W Betebenner
National Center for the Improvement of Educational Assessment
New Jersey Technical Advisory Committee
June 1, 2015
GROWTH DURING TRANSITION:
TWO DISTINCT CHALLENGES
• Challenges to the calculation of student growth.
• Challenges to the use of student growth for:
• Teacher evaluation
• School/District/State evaluation
• Other diagnostic purposes
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
Before: 2013-2014
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1: 2014-2015
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
During 1I: 2015-2016
THE TRANSITION “SPEED BUMP”
(At least) 4 stages to traverse when considering growth
After: 2016-2017
September October November December January February March April May June July August
PARCC PBA PARCC EOY
Field Testing Field Testing
PBA EOY Standard
Window Window Setting
2016 Scores &
2015 Scores PBA EOY
Data Files
Returned Window Window
Returned
PBA EOY 2017 Scores &
Window Window Data Files
Returned
September October November December January February March April May June July August
REVIEW: FLAVORS OF GROWTH
• SGP accommodates both Norm- and Criterion-Referenced growth
• Norm-Referenced: Cohort referenced SGPs
• Criterion-Referenced: Baseline referenced/anchored SGPs, student growth projections/targets
(i.e., growth to standard), catch-up/keep-up/move-up/stay-up.
• Based upon state preference, policy, and data, different states have created
different combinations of norm- and criterion-referenced growth metrics.
• These growth metrics often have different data requirements.
• SGP calculation is robust to scale transformations. It unlikely that the transition
from a state assessment scale to a PARCC scale will prevent the calculation of
an SGP. However, this has to be validated on a state by state, content area by
content area, and grade by grade basis.
Level 5
Advanced Level 4
Move Up
Target ●●
Keep Up ● ● ● Catch Up
Target Level 3
Target
Proficient ●
Level 2
Part Proficient
Level 1
Unsatisfactory
−− Grade 4 Grade 5 Grade 6 Grade 7
Scale Score −− 491 506 1499
Achievement Level
Achievement Target
−− Proficient Proficient Level 2
Level 4 (1586)/Level 5 (1586) Level 4 (1593)
Achievement
Growth Percentile −− −− 21
Growth Level
Growth Target
−− −− Low
Keep Up (99)/Move Up (99) Catch Up (99)
Growth
REVIEW: THREATS TO CALCULATION
• SGPs don’t require a vertical scale for calculation. Regardless of whether
PARCC uses a vertical scale, SGPs can be calculated using both vertical and
non-vertical scales.
• A signi cant ceiling/ oor associated with a test (either PARCC or a state test
used as a prior) can lead to bad model t. These growth metrics often have
different data requirements.
• Barring signi cant ceiling/ oor, norm-referenced analyses will very likely yield a
uniform distribution of percentiles.
• SGP calculation is robust to scale transformations. It unlikely that the transition
from a state assessment scale to a PARCC scale will prevent the calculation of
an SGP. However, this has to be validated on a state by state, content area by
content area, and grade by grade basis.
fi
fl
fl
fi
fi
REVIEW: THREATS TO CALCULATION
Student Growth Percentile Goodness−of−Fit Descriptives Student Growth Percentile Goodness−of−Fit Descriptives
2014 ELA SGP, Grade 6 (N = 95,825) 2014 Mathematics SGP, Grade 6 (N = 96,072)
SGP Deciles by Prior Achievement Level SGP Deciles by Prior Achievement Level
Advanced (8.8%)
Prior Achievement Level (Mathematics)
Advanced (43.6%)
Prior Achievement Level (ELA)
Proficient (53.2%)
Proficient (37.1%)
Part Proficient (38.1%)
Part Proficient (19.3%)
Median Student Growth Percentile Median Student Growth Percentile
Student Growth Percentile Range QQ−Plot: Student Growth Percentiles Student Growth Percentile Range QQ−Plot: Student Growth Percentiles
19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution 19 29 39 49 59 69 79 89 99 Theoretical SGP Distribution
9 to to to to to to to to to 9 to to to to to to to to to
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100 1
to 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 100
1st/[100,169) 9.89 9.94 9.85 10.26 9.62 9.83 10.15 9.89 10.10 10.48 (9.6%) 100 1st/[100,179) 9.61 10.41 10.13 9.87 9.81 10.21 9.51 9.69 10.07 10.70 (9.6%) 100
2nd/[169,181) 9.93 10.46 10.07 9.94 10.16 9.95 9.76 10.00 9.40 10.33 (8.7%) 2nd/[179,200) 9.40 9.66 9.97 10.12 9.85 9.85 10.32 10.31 9.68 10.83 (9.7%)
Prior Scale Score Decile/Range * Prior Scale Score Decile/Range *
3rd/[181,190) 9.65 10.02 10.27 10.21 10.01 9.48 10.46 10.26 10.16 9.48 (8.9%) 3rd/[200,214) 9.89 10.18 9.82 9.82 10.17 10.00 9.71 9.54 10.46 10.42 (10.7%)
Empirical SGP Distribution Empirical SGP Distribution
4th/[190,200) 9.77 10.19 10.01 9.44 10.02 9.90 9.75 9.79 10.13 11.00 (10.8%) 4th/[214,226) 9.41 9.86 10.38 10.35 9.43 9.83 9.41 10.05 10.71 10.57 (7.6%)
5th/[200,205) 9.96 10.10 10.01 10.38 10.19 9.88 9.60 9.76 9.66 10.45 (8.4%) 5th/[226,241) 9.22 10.33 9.77 9.47 10.37 10.07 10.51 10.21 8.74 11.31 (12.1%)
6th/[205,214) 9.77 10.04 10.15 9.90 9.73 10.28 10.10 10.20 9.68 10.15 (12.9%) 6th/[241,253) 9.96 9.72 10.01 10.41 9.75 9.74 9.68 9.75 10.82 10.14 (10.2%)
7th/[214,221) 9.26 10.09 9.77 9.79 9.74 9.60 9.86 10.50 10.99 10.39 (8.6%) 7th/[253,263) 9.86 9.83 10.12 9.94 9.93 9.80 9.14 9.84 11.62 9.93 (7.4%)
8th/[221,232) 9.79 10.24 9.87 9.89 10.11 10.38 10.06 9.66 9.60 10.39 (11.5%) 8th/[263,280) 9.31 10.22 9.95 9.51 10.07 9.77 10.54 10.38 9.15 11.11 (11.4%)
9th/[232,243) 9.88 9.92 10.14 10.06 10.18 9.66 10.01 9.65 9.66 10.83 (9.4%) 9th/[280,295) 8.80 9.63 9.78 10.03 10.40 10.63 10.38 9.28 4.30 16.76 (7.8%)
10th/[243,300] 9.23 10.16 9.75 10.00 9.94 10.15 9.94 10.23 10.16 10.45 (11.2%) 10th/[295,300] 9.80 10.23 10.40 9.41 5.96 5.18 3.59 2.58 0.95 41.90 (13.5%)
*Prior score deciles can be uneven depending upon the prior score distribution *Prior score deciles can be uneven depending upon the prior score distribution
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH: SGP
• Annual growth? Bi-annual growth? Tri-annual growth?
• Annual growth (norm-referenced SGPs based upon a single cohort) requires
(at a minimum) two consecutive annual measures (i.e., scale scores) of student
achievement (usually taken from the same content area).
• Annual growth (anchored/baseline-referenced SGPs) requires (at a minimum)
two consecutive annual measures (i.e., scale scores) of student achievement
(usually taken from the same content area) from the same assessment (i.e., an
assessment with a time-invariant scale).
• Annual targets (SGP Projections/Targets/Trajectories) require (at a minimum)
two consecutive annual measures of student achievement from the same
assessment (i.e., an assessment with a time-invariant scale).
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH: SGP
• States primarily focus on annual growth, particularly for accountability
(exceptions include, for example, student growth from 8th to 10th grade).
• PARCC eld testing will impact a state’s ability to calculate growth for students
due to the simple fact that student will not receive back scale scores on the
PARCC eld test.
• Absent double testing, students participating in the
eld test cannot receive
annual SGPs in BOTH 2013-2014 (no current year score) AND 2014-2015
(no prior year score).
• If a student has a scale score in 2012-2013, one could, for example, calculate a
two year SGP for a students participating in eld testing between 2012-2013
and 2014-2015.
• In the years after the transition, absent double testing, students will have
different numbers of priors available for SGP analyses.
fi
fi
fi
fi
REVIEW: REQUIREMENTS FOR CALCULATING
GROWTH: SUMMARY
• Easier to conduct norm-referenced SGP analyses during transition than
criterion-referenced growth to standard analyses.
• Criterion-referenced projections require linking SGP analyses done exclusively
on the PARCC assessment in order to extrapolate forward how much growth
is required to, for example catch-up/keep-up/move-up/stay-up.
• Analogously growth-to-standard (e.g., career and college-readiness) will require
at least two years data on the new scale as well as xed achievement levels/
standards de ned on the new scale.
• Establishing baseline referenced SGPs will likely require even more time (e.g., 4
years) to establish a stable baseline from which to compare going forward.
fi
fi
September October November December January February March April May June July August
PARCC PBA PARCC EOY
Field Testing Field Testing
PBA EOY Standard
Window Window Setting
2016 Scores &
2015 Scores PBA EOY
Data Files
Returned Window Window
Returned
PBA EOY 2017 Scores &
Window Window Data Files
Returned
September October November December January February March April May June July August
REQUIREMENTS FOR THE USE OF GROWTH
• Two distinct challenges: Calculation and Use
• Just because we can calculate an SGP doesn’t mean it is defensible to use it for
previously existing accountability purposes.
• Validating SGP use will require de ning the purpose.
• Current state uses include:
• Teacher evaluation (required by RttT, ESEA waiver, state policy/rule
making, ...)
• School and District accountability (ESEA waiver, state policy, ...)
• Charter school accreditation.
fi
REQUIREMENTS FOR THE USE OF GROWTH
• During the assessment transition, a number of challenges confront states using
growth for accountability purposes.
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Changes in types of growth analyses done for accountability during transition.
Criterion-referenced growth not available until year 2 of PARCC.
• Assessment results from different assessments (state and PARCC).
Depending upon the number of priors used by the state, this will be an
ongoing issue for many years.
• Missing data due to PARCC eld testing.
fi
REQUIREMENTS FOR THE USE OF GROWTH
• Data time span (e.g., two-year growth) not aligned with accountability time
span (annual)
• Most accountability systems use an observational pre-post design to hold
teachers/schools/districts accountable.
• Often, the pre-test occurs the spring prior to enrollment in the unit of
accountability and the post-test the spring during enrollment.
• An extended growth time span (e.g., two-year) undermines the attribution
of responsibility implicit within the accountability determination.
• Mis t between accountability and growth time span impact school and
district accountability as well, but likely will have less impact because eld
testing does not occur school wide.
fi
fi
About this copy
Growth During Transition — New Jersey Department of Education, May 2015.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.