Damian W Betebenner
National Center for the Improvement of Educational Assessment
Atlanta, Georgia February 27th, 2018
GROWTH-TO-STANDARD USING GEORGIA MILESTONES
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Atlanta, Georgia February 27th, 2018
BACKGROUND
• Since their development, SGP analyses have supported both norm- and
criterion-referenced student growth analyses.
• Norm-referenced results are more prominently used than the
criterion-referenced analyses, particularly with respect to accountability.
• Criterion-referenced SGP analyses were submitted and approved as
part of the Growth Model Pilot Program under Secretary Spellings
during the Bush Administration.
• There is renewed interest in looking at growth-to standard with ESSA
for state summative and ELP assessment with several states using
growth-to-standard as part of their accountability plans.
BACKGROUND
• The norm- and criterion-referenced component of SGP analyses are
the SGP (student growth percentile) and the AGP (adequate growth
percentile).
• AGP calculation requires establishing an ending achievement target
and a time frame to reach the target.
• The most common achievement goal is proficiency/career & college
readiness.
• The most common time frame is 3 years.
• As part of AGP calculation, AGPs are calculated for all possible time
frames (1, 2, …) and all possible achievement level cutscores.
STUDENT GROWTH PERCENTILES/
PROJECTIONS
• SGP analyses produce coefficient matrices for each of 100 quantile values:
• Using these matrices, 100 quantile cuts are established for each student based upon
their scale score history (independent variables) and used to associate a student’s scale
score (dependent variable) with an integer from 0 to 100 (inclusive). 0s and 100s are
usually converted to 1s and 99s.
• The coefficient matrices allow for the calculation of 100 percentile growth trajectories
for a student: Student Growth Projections
• AGPs (Adequate Growth Percentiles) are determined by examining the 100 trajectories
and finding the minimal trajectory that allows the student to reach the desired
destination in the given time-frame.
• AGP are like scale-score trajectories but they smooth “difficulty” (equal percentile
growth for the duration of the trajectory).
Georgia: 2017 ELA
Norm & Criterion Referenced Growth & Achievement
No Grade 10 Assessment
Percentile Growth Trajectory
2017 Achievement Percentile
SGP trajectories for a 3rd grade student starting from the Level 1/Level 2 cut.
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 EOC EOC
ELA ELA ELA ELA ELA ELA Grade 9 Lit American Lit
Georgia: 2017 ELA
Norm & Criterion Referenced Growth & Achievement
No Grade 10 Assessment
Percentile Growth Trajectory
2017 Achievement Percentile
SGP
SGP trajectories
trajectories for
for aa 3rd
3rd grade
grade student
student starting
starting from
from the
the 10th
10th achievement
achievement percentile.
percentile.
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 EOC EOC
ELA ELA ELA ELA ELA ELA Grade 9 Lit American Lit
STUDENT GROWTH PERCENTILES/
PROJECTIONS: DETAILS
• For current Georgia Milestones SGP analyses we use up to two priors
scores.
• Taking ELA 3 to 8/Grade 9 Lit/American Lit as an example content area
sequnce, we calculate coefficient matrices using the following
independent/dependent variable groups:
Grade/Content Area Dependent Variable Independent Variable
Grade 4/ELA Grade 4 ELA Scale Score Grade 3 ELA Scale Score
Grade 5/ELA Grade 5 ELA Scale Score Grade 3/4 & Grade 4 ELA Scale Score
Grade 6/ELA Grade 6 ELA Scale Score Grade 4/5 & Grade 5 ELA Scale Score
Grade 7/ELA Grade 7 ELA Scale Score Grade 5/6 & Grade 6 ELA Scale Score
Grade 8/ELA Grade 8 ELA Scale Score Grade 6/7 & Grade 7 ELA Scale Score
EOCT/Grade 9 Lit Grade 9 Lit Scale Score Grade 7/8 & Grade 8 ELA Scale Score
Grade 8 ELA/Grade 9 Lit & Grade 9 Lit
EOCT/American Lit American Lit Scale Score
Scale Score
STUDENT GROWTH PERCENTILES/
PROJECTIONS: DETAILS
• For example, consider how we construct 100 percentile trajectories for a 3rd Grade student starting at the
Beginning/Developing ELA cutscore of 475:
• Step 1: To derive 100 grade 4 projections we use the grade 3 to grade 4 coefficient matrix with a grade
3 scaled score of 475.
• Step 2: To derive the 100 grade 5 projections we use the grade 3/4 to grade 5 coefficient matrix with
the grade 3 scaled score of 475 and the 100 grade 4 projections from step 1.
• Step 3: To derive the 100 grade 6 projections we use the grade 4/5 to grade 6 coefficient matrix with
the 100 grade 4 projections and 100 grade 5 projections.
• Step 4: To derive the 100 grade 7 projections we use the grade 5/6 to grade 7 coefficient matrix with
the 100 grade 5 projections and 100 grade 6 projections.
• Step 5: To derive the 100 grade 8 projections we use the grade 6/7 to grade 8 coefficient matrix with
the 100 grade 6 projections and 100 grade 7 projections.
• Step 6: To derive the 100 grade 9 Lit projections we use the Grade 7/8 to grade 9 lit coefficient matrix
with the 100 grade 7 projections and the 100 grade 8 projections.
• Step 7: To derive the 100 american lit projections we use the Grade 8/9 lit to American lit coefficient
matrix with the 100 grade 8 projections and the 100 grade 9 lit projections.
STUDENT GROWTH PERCENTILES/
PROJECTIONS: DETAILS
• Coefficient matrices are “daisy chained” together to create the 100
percentile trajectories.
• There are only a few “canonical” content area “daisy chains” (e.g.,
ELA 3 to 8 to Grade 9 Lit to American Lit).
• However, given the different EOCT test taking patterns that
students follow, several less common trajectories are possible (but
not currently calculated).
• The list of all content area sequences along which projections are
derived is as follows:
FROM PROJECTIONS TO AN AGP
• Percentile trajectories (projections) need to be converted to a single
target based upon a time frame and an achievement goal.
• Most states use a 3 year time frame to proficiency. Georgia is
considering a 3 year time frame to each of the top 3 achievement
levels (3 cutscores).
WHAT DOES “ON TRACK” MEAN
• Common parlance is often used to try and put a sophisticated idea into “layman’s
terms”. “On track to reach proficiency” or “On track to catch-up to proficiency”
are two common phrases in this regard.
• There is general confusion about what the term “on track” means (e.g., the student
is “on track to reach proficiency”). E.g., “The student will reach proficiency.”
• My Take: “On Track” has one of two meanings depending upon whether a pre-
specified “pathway to proficiency” plan is established:
• No plan: Assuming “typical” rates of progress continue, the student will reach
proficiency.
• Yes plan: The student is hitting interim targets and assuming they continue to hit
these targets will reach proficiency.
ON TRACK
• A student is considered to be "on track" if their SGP >= AGP (that is,
they grew more than their target).
• An AGP can be converted to a sequence of 1, 2, 3, … Tmax scale
scores such that the student can compare their scale score to a target
scale score to determine whether they are "on track”
• The scale score trajectory will be curvi-linear such that each segment
has the same “difficulty”. This differs from a linear scale score
trajectory.
ANOMALIES
• In a very few cases (generally < 1%) a student will not reach their
achievement goal but their AGP >= SGP
• This situation occurs when consecutive quantile cuts encompass both
the cutscore as well as an adjacent cut.
• For example, a student scores 474 on the Grade 5 ELA. Their score is
in the (.415, .425] quantile interval (473.2, 475] and receives an SGP of
42. Their 1-year AGP is also 42 (since both 474 and 475 are in that
interval). They hit their 1-year catch-up target but didn’t catch-up.
• If AGP are converted to scaled score targets, this situation can be
avoided
OTHER CONSIDERATIONS
• Georgia currently uses/reports measurement error corrected
(SIMEX) SGPs, projections are calculated using observed score
coefficient matrices.
• This is justified (and preferred) because (Carroll, 2009):
• “… if future observation of X are also measured with an error then
it is rarely necessary to address the measurement error as
prediction of Y can be achieved via estimate by standard non-
parametric regression estimators from the sample.”
BIBLIOGRAPHY
• Carroll, R. J., Delaigle, A., & Hall, P., Nonparametric prediction in
measurement error models. J Am Stat Association. 2009 Sep 1;
LAGGED VERSUS CURRENT PROJECTIONS
• Two types of projections are calculated: Lagged and Current projections.
• Lagged projections provide the AGP targets used for growth-to-
standard.
• Lagged projections apply the most recent coefficient matrices (those
use to produce the student’s current SGP) to calculate a target. Thus
SGP and AGP are based upon the same norms.
• Current projections are used to provide forward looking targets.
• When baseline analyses are used, lagged targets and current targets
don’t change. Annual re-norming can cause some discrepancies between
current and lagged targets.
TIME FRAME TO REACH GOAL
• Times frames used for growth-to-standard can either be fixed in time
or fixed in duration (“rolled forward”).
• Fixed time frames mandate the student reach proficiency at a
particular time/grade.
• Fixed in duration time frames (e.g., 3 years) are usually extended each
year to yield new targets.
• AGP calculations can support either approach to setting a time-frame.
• Fixed in time approaches usually yield a “balloon payment” for students
who have failed to make intermediate targets. This causes problems for
accountability systems.
About this copy
Georgia Growth to Standard Introduction — Georgia Department of Education, March 2018.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.