The Invariance of Measures of
Growth & Effectiveness to Scale Transformation
Derek Briggs & Damian Betebenner
University of Colorado, Boulder & The Center for Assessment
NCME Presentation
San Diego, California April 14th, 2009
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Quantile Regression/Student Growth Percentiles
We wish to estimate:
Pr(Current Achievement |Past Achievement ).
Quantile regression is used to model the conditional quantile
functions.
B-spline smoothing is applied to accommodate non-linearity,
heteroscedasticity, etc. Scatterplot Smoothing
The method is a quantile regression based generalized additive
model[Hastie and Tibshirani, 1990, Wood, 2006]
Implemented in R using SGP Package
QSSt (τ|SSt −1 , . . . , SS1 ) = φij (SSj )βij (τ),
where φi ,j , i = 1, 2, 3 and j = 1, . . . , t − 1 denote the B-spline basis
functions.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Quantile Regression/Student Growth Percentiles
Conditional Decile Regression Lines Conditional Decile Regression Curves
2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6
1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof.
Proficient Proficient
Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced
Grade 6 Scale Score Grade 6 Scale Score
Grade 5 Scale Score Grade 5 Scale Score
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Within Method Results
Are school results within each analysis method
invariant to monotonic scale transformations?
In general, yes
With the exception of the Lord transformation, correlations of VAM
effects/residuals were all high (above 0.85). With three years of data
the correlations were all above 0.98 and above.
Correlations for median student growth percentiles across the 7
scales and 3 years were 1.0
The Lord Transformation “breaks” the VAM estimates because it
radically alters the interval nature of the scale.
Standardizing scores with layered VAM models obviates most issues
associated with non-intervalness.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Across Method Results
Fixing the monotonic scale transformation, are
school results across analysis method similar?
It depends.
With three years (2006/Grade 6) of data, the correlation between
median student growth percentiles and VAM effects/residuals
(SEP3ML3) were 0.91. Standardizing the percentiles increases
correlation to 0.95.
With 1 year (2004/Grade 4) and 2 years (2005/Grade 5) data, the
correlations were 0.72 and 0.84
Outliers suggest relationship with prior achievement, especially with
1 year (2004/Grade 4)
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Within Method Results: Redux
What is the relationship between prior
achievement and the VAM effects/median SGP?
Correlation between median SGP and prior achievement (mean
theta, SEP3ML3) were 0.34, 0.18, and 0.31 in 2004, 2005, and
2006, respectively.
Correlation between VAM effects/residuals and prior achievement
(mean theta, SEP3ML3) were -0.11, -0.09, and 0.39 in 2004, 2005,
and 2006, respectively.
Still investigating why the dramatic switch in the VAM model.
What correlation passes the smell test?
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
Within Method Results: Redux
What is the relationship between school poverty
and the VAM effects/median SGP?
Correlation between median SGP and percent free/reduced lunch
were -0.42, -0.25, and -0.39 in 2004, 2005, and 2006, respectively.
Correlation between VAM effects/residuals and percent free/reduced
lunch were -0.15, -0.06, and -0.51 in 2004, 2005, and 2006,
respectively.
What correlation passes the smell test?
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
References
Hastie, T. J. and Tibshirani, R. J. (1990).
Generalized Additive Models.
Chapman & Hall/CRC, New York, NY.
Wood, S. N. (2006).
Generalized Additive Models: An Introduction with R.
Chapman & Hall/CRC, New York, NY.
D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
About this copy
The Invariance of Measures of Growth & Effectiveness to Scale Transformation — NCME Annual Meeting, April 2009.
This is an image-based copy of the original beamer slides; builds and animations are flattened.