Background
Approach
References
Assessment Changes/Transitions using SGPs
Damian W. Betebenner & Adam Van Iwaarden
National Center for the Improvement of Educational Assessment
New Meridian TAC Meeting — Austin, Texas
October 3rd, 2019
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
SGP Calculations with Pearson/PARCC/New Meridian
History of SGP Calculations
SGPs [Betebenner, 2009] calculated for 2015-2016, 2016-2017,
2017-2018, and 2018-2019
SGPs, Measurement error corrected SGPs, SGP standard
errors, and student growth projections are calculated at both the
state and consortium level.
For 2018-2019, SGPs were calculated for NM, IL, BIE, NJ,
DoDEA, DC, MD
Several states/organizations utilize consortium level SGPs and
were interested in investigating whether changing consortium
membership and/or changes in test format impacts the validity
of SGP calculated.
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
SGP Calculations with Pearson/PARCC/New Meridian
History of SGP Calculations
In general, because SGPs are norm referenced, they are scale
agnostic only requiring students to take the same test at the
same time.
Given changing consortium state membership together with
different test/form types, there are questions regarding the
comparability of SGPs both within and across years.
Issues related to mode of administration effect (within year,
different groups) have been investigated by PARCC previously.
Issues related to across year differences in growth due to
changes in consortium membership or test form change (e.g.,
2019 short-form use) have not been investigated.
For 2019 spring analyses, NM, IL, NJ, BIE utilized an
abbreviated form of the PARCC test. DC, MD, DoDEA utilized
the flagship (long) form. NJ also post-equated its ELA test.
We investigated whether changing state membership as well as
test form differences led to appreciable differences in SGP
results.
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
SGP Properties
SGP Properties
SGPs are uniformly distributed and uncorrelated with prior
achievement at the individual level.
State and consortium level SGPs (i.e., growth norms) are
calculated.
For large groups, entire states (state norms), or the consortium,
mean(SGP ) ≈ 50.
“Large” deviations from 50 or between groups usually indicates
something is going on and worth investigating further.
A state’s recent transition to different assessments at the high
school level led to 10 to 15 point mean SGP disparities between
males and females (prior tests were near parity between
males/females)
Within year mean SGP differences between groups taking
paper-pencil versus computer administered have been
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
Investigation of mean SGP differences
Two approaches
Probabilistic Difference
Given sufficient historical data, mean SGPs can
be investigated relative to the distribution of
previous observations.
A t-distribution or similar sample distribution can
be used to formally calculate a p-values.
In general, too few observations (3 years of mean
SGPs for these analyses) were available for
significance testing.
Also, statistical significance doesn’t imply
practical significance.
Practical Difference
Cohen’s D can be easily calculated for groups
using SGPs.
Because SGPs are uniform, the pooled within
group standard deviation that forms the
denominator is approximately 28.8 for most
groups investigated.
A mean(SGP ) difference of 5 (which is often
used as a rule of thumb for practical differences)
corresponds to a Cohen’s D of 0.17.
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
Investigation of mean SGP differences
Results
Historical consortium SGP results were recalculated using 2019
state membership to examine year-to-year variation.
Year-to-year differences uniformly negligible using 5 as the
practical significance threshold.
For state continuing with dual mode administration, mode
effects continue to be detected.
Betebenner & Van Iwaarden Assessment Changes & SGPs
Background
Approach
References
References
Betebenner, D. W. (2009).
Norm- and criterion-referenced student growth.
Educational Measurement: Issues and Practice, 28(4):42–51.
Betebenner & Van Iwaarden Assessment Changes & SGPs
About this copy
Assessment Changes/Transitions using SGPs — New Meridian TAC, October 3, 2019.
This is an image-based copy of the original beamer slides; builds and animations are flattened.