The Invariance of Measures of
Growth & Effectiveness to Scale Transformation
Derek Briggs & Damian Betebenner
University of Colorado, Boulder & The Center for Assessment
NCME Presentation
San Diego, California April 14th, 2009
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Study Overview
PREMISE
• Let’s assume test score scales are ordinal, not interval.
• Any monotonic transformations to the scale are fair game.
• States with vertical scales may decide for themselves what
growth “should” look like.
OUR APPROACH
• Create alternate scales from admissable transformations of
same base vertical scale.
• Compare school-level normative growth quantities using two
different models
– Layered Model (Sanders, Saxton & Horn)
– Quantile Regression (Koenker)
Colorado Data
Year
Grade Cohorts 2003 2004 2005 2006
Grade 3 Reading 3
Grade 4 Reading 4 4
Grade 5 Reading 5 5
Grade 6 Reading 6 6
Grade 7 Reading 7
•Outcome measure is Colorado Student Assessment Program (CSAP) test
scores in reading. [Items: ~70 MC, 14 CR]
•Longitudinal item responses for two cohorts of public and charter school
students in the state of Colorado.
•Each grade by year cell combination contains roughly 56,000 students.
1,379 unique schools.
•Race/Ethnicity of Students: 64% White, 26% Hispanic, 6.3% Black
Possible Vertical Scales
IRT Model Calibration θ Estimation
1. 1PLM/PCM 1. Separate 1. EAP
2. 3PLM/GPCM 2. Hybrid 2. ML
The scaling approach used in CO:
3PLM/GPCM, Separate, ML Î “SEP3ML”
What transformations to this scale might states consider if it is only
ordinal?
Scale Transformations
Approach 1: Growth Across Grades
Plausible, minor
transformations
Linear NonLinear
Constant TLC TNC
Grade to
Grade SD
Increasing TLI TNI
Approach 2: Minimize overlap by grade
Approach 3: Break the scale using Lord’s transformation
Growth in Effect Size Units:
CSAP Reading, Grades 3-6
TLC
Effect Size Units
TLI
0.3 TNC
TNI
0.2 TMO
TLORD
SEP3ML
Grade to Grade Reading Grow th
Layered Model
Yi 03 = μ03 + θ03 + ε i 03
Yi 04 = μ04 + θ03 + θ04 + ε i 04
Yi 05 = μ05 + θ03 + θ04 + θ05 + ε i 05
Yi 06 = μ06 + θ03 + θ04 + θ 05 + θ 06 + ε i 06 .
Value-Added Estimates of Interest: {θˆ 4 , θˆ 5 , θˆ 6 }
Notes: Model above assumes complete persistence. Bayesian
estimation using non-informative priors following Lockwood et al 2007
Quantile Regression/Student Growth Percentiles
We wish to estimate:
Pr(Current Achievement |Past Achievement ).
Quantile regression is used to model the conditional quantile
functions.
B-spline smoothing is applied to accommodate non-linearity,
heteroscedasticity, etc. Scatterplot Smoothing
The method is a quantile regression based generalized additive
model[Hastie and Tibshirani, 1990, Wood, 2006]
Implemented in R using SGP Package
QSSt (τ|SSt −1 , . . . , SS1 ) = φij (SSj )βij (τ),
where φi ,j , i = 1, 2, 3 and j = 1, . . . , t − 1 denote the B-spline basis
functions.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Quantile Regression/Student Growth Percentiles
Conditional Decile Regression Lines Conditional Decile Regression Curves
2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6
1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof.
Proficient Proficient
Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced
Grade 6 Scale Score Grade 6 Scale Score
Grade 5 Scale Score Grade 5 Scale Score
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Within Method Results
Are school results within each analysis method
invariant to monotonic scale transformations?
In general, yes
With the exception of the Lord transformation, correlations of VAM
effects/residuals were all high (above 0.85). With three years of data
the correlations were all above 0.98 and above.
Correlations for median student growth percentiles across the 7
scales and 3 years were 1.0
The Lord Transformation “breaks” the VAM estimates because it
radically alters the interval nature of the scale.
Standardizing scores with layered VAM models obviates most issues
associated with non-intervalness.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Across Method Results
Fixing the monotonic scale transformation, are
school results across analysis method similar?
It depends.
With three years (2006/Grade 6) of data, the correlation between
median student growth percentiles and VAM effects/residuals
(SEP3ML3) were 0.91. Standardizing the percentiles increases
correlation to 0.95.
With 1 year (2004/Grade 4) and 2 years (2005/Grade 5) data, the
correlations were 0.72 and 0.84
Outliers suggest relationship with prior achievement, especially with
1 year (2004/Grade 4)
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Figure 4a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale)
by Prior Achievement Quintile (School Size >= 50)
Prior Achievement
1st Quintile
2nd Quintile
3rd Quintile
● ●● ● ● ● 4th Quintile
Median Student Growth Percentile
● 5th Quintile
School Size
●●● ● 25 Students
50 Students
100 Students
● 250 Students
500 Students
Value−Added Estimate from Layered Model
Figure 4b: Grade 5 (2005) Layered Model versus Quantile Regression Estimates (SEP3ML Scale)
by Prior Achievement Quintile (School Size >= 50)
Prior Achievement
1st Quintile
2nd Quintile
● ● 3rd Quintile
4th Quintile
Median Student Growth Percentile
5th Quintile
● School Size
25 Students
●● ● ● 50 Students
●● 100 Students
● 250 Students
● 500 Students
Value Added−Estimate from Layered Model
Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale)
by Prior Achievement Quintile (School Size >= 50)
Prior Achievement
1st Quintile
2nd Quintile
● ● 3rd Quintile
4th Quintile
Median Student Growth Percentile
70 ●●●● 5th Quintile
●●●● School Size
25 Students
● 50 Students
100 Students
●● ● 250 Students
● ● 500 Students
Value−Added Estimate from Layered Model
Within Method Results: Redux
What is the relationship between prior
achievement and the VAM effects/median SGP?
Correlation between median SGP and prior achievement (mean
theta, SEP3ML3) were 0.34, 0.18, and 0.31 in 2004, 2005, and
2006, respectively.
Correlation between VAM effects/residuals and prior achievement
(mean theta, SEP3ML3) were -0.11, -0.09, and 0.39 in 2004, 2005,
and 2006, respectively.
Still investigating why the dramatic switch in the VAM model.
What correlation passes the smell test?
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Within Method Results: Redux
What is the relationship between school poverty
and the VAM effects/median SGP?
Correlation between median SGP and percent free/reduced lunch
were -0.42, -0.25, and -0.39 in 2004, 2005, and 2006, respectively.
Correlation between VAM effects/residuals and percent free/reduced
lunch were -0.15, -0.06, and -0.51 in 2004, 2005, and 2006,
respectively.
What correlation passes the smell test?
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
References
Hastie, T. J. and Tibshirani, R. J. (1990).
Generalized Additive Models.
Chapman & Hall/CRC, New York, NY.
Wood, S. N. (2006).
Generalized Additive Models: An Introduction with R.
Chapman & Hall/CRC, New York, NY.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
About this copy
The Invariance of Measures of Growth & Effectiveness to Scale Transformation — NCME Annual Meeting, April 2009.
This is an image-based copy of the original PowerPoint slides; builds and animations are flattened.