The Invariance of Measures of Growth & Effectiveness to Scale Transformation

NCME Annual Meeting · April 2009

The Invariance of Measures of Growth & Effectiveness to Scale Transformation — NCME Annual Meeting, April 2009.
Published

April 14, 2009

Event
Conference
Slides
11
Format
Converted from PDF

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. The Invariance of Measures of Growth & Effectiveness to Scale Transformation Derek Briggs & Damian Betebenner University of Colorado, Boulder & The Center for Assessment NCME Presentation San Diego, California April 14th, 2009 D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  2. Quantile Regression/Student Growth Percentiles We wish to estimate: Pr(Current Achievement |Past Achievement ). Quantile regression is used to model the conditional quantile functions. B-spline smoothing is applied to accommodate non-linearity, heteroscedasticity, etc. Scatterplot Smoothing The method is a quantile regression based generalized additive model[Hastie and Tibshirani, 1990, Wood, 2006] Implemented in R using SGP Package QSSt (τ|SSt −1 , . . . , SS1 ) = φij (SSj )βij (τ), where φi ,j , i = 1, 2, 3 and j = 1, . . . , t − 1 denote the B-spline basis functions. D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  3. Quantile Regression/Student Growth Percentiles Conditional Decile Regression Lines Conditional Decile Regression Curves 2006−07 Math: Grade 5 versus 6 2006−07 Math: Grade 5 versus 6 1000 Unsatisfactory Partially Prof. 1000 Unsatisfactory Partially Prof. Proficient Proficient Partially Prof. Proficient Advanced Partially Prof. Proficient Advanced Grade 6 Scale Score Grade 6 Scale Score Grade 5 Scale Score Grade 5 Scale Score D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  4. Within Method Results Are school results within each analysis method invariant to monotonic scale transformations? In general, yes With the exception of the Lord transformation, correlations of VAM effects/residuals were all high (above 0.85). With three years of data the correlations were all above 0.98 and above. Correlations for median student growth percentiles across the 7 scales and 3 years were 1.0 The Lord Transformation “breaks” the VAM estimates because it radically alters the interval nature of the scale. Standardizing scores with layered VAM models obviates most issues associated with non-intervalness. D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  5. Across Method Results Fixing the monotonic scale transformation, are school results across analysis method similar? It depends. With three years (2006/Grade 6) of data, the correlation between median student growth percentiles and VAM effects/residuals (SEP3ML3) were 0.91. Standardizing the percentiles increases correlation to 0.95. With 1 year (2004/Grade 4) and 2 years (2005/Grade 5) data, the correlations were 0.72 and 0.84 Outliers suggest relationship with prior achievement, especially with 1 year (2004/Grade 4) D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  6. Figure 4a: Grade 4 (2004) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile 3rd Quintile ● ●● ● ● ● 4th Quintile Median Student Growth Percentile ● 5th Quintile School Size ●●● ● 25 Students 50 Students 100 Students ● 250 Students 500 Students Value−Added Estimate from Layered Model
  7. Figure 4b: Grade 5 (2005) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile 4th Quintile Median Student Growth Percentile 5th Quintile ● School Size 25 Students ●● ● ● 50 Students ●● 100 Students ● 250 Students ● 500 Students Value Added−Estimate from Layered Model
  8. Figure 4c: Grade 6 (2006) Layered Model versus Quantile Regression Estimates (SEP3ML Scale) by Prior Achievement Quintile (School Size >= 50) Prior Achievement 1st Quintile 2nd Quintile ● ● 3rd Quintile 4th Quintile Median Student Growth Percentile 70 ●●●● 5th Quintile ●●●● School Size 25 Students ● 50 Students 100 Students ●● ● 250 Students ● ● 500 Students Value−Added Estimate from Layered Model
  9. Within Method Results: Redux What is the relationship between prior achievement and the VAM effects/median SGP? Correlation between median SGP and prior achievement (mean theta, SEP3ML3) were 0.34, 0.18, and 0.31 in 2004, 2005, and 2006, respectively. Correlation between VAM effects/residuals and prior achievement (mean theta, SEP3ML3) were -0.11, -0.09, and 0.39 in 2004, 2005, and 2006, respectively. Still investigating why the dramatic switch in the VAM model. What correlation passes the smell test? D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  10. Within Method Results: Redux What is the relationship between school poverty and the VAM effects/median SGP? Correlation between median SGP and percent free/reduced lunch were -0.42, -0.25, and -0.39 in 2004, 2005, and 2006, respectively. Correlation between VAM effects/residuals and percent free/reduced lunch were -0.15, -0.06, and -0.51 in 2004, 2005, and 2006, respectively. What correlation passes the smell test? D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation
  11. References Hastie, T. J. and Tibshirani, R. J. (1990). Generalized Additive Models. Chapman & Hall/CRC, New York, NY. Wood, S. N. (2006). Generalized Additive Models: An Introduction with R. Chapman & Hall/CRC, New York, NY. D. Briggs & D. Betebenner Growth, Effectiveness, and Scale Transformation