The Big Picture Measurement/Psychometric Considerations References On Consortia Approaches to Measuring Student Growth Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH Measuring Student Growth: New Advances and Major Challenges in the State Assessment Consortia Initiatives April 9th, 2011 Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Student Growth: An Intersection of Three Topics Longitudinal Data Analysis/Applied Statistics Accountability/Education Policy/Data Use Measurement/Psychometrics Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Overview Growth in what? How much growth? Scales for measuring growth Ordinal Interval Vertical Growth: magnitude and/or norm Adequate growth? Norm- versus criterion-referencing (intersection of Accountability and Measurement) Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Growth in what? Beneath any notion of change (i.e., growth) is a construct that is changing over time Height and weight are common points of reference Constructs in education are not manifest like physical measurements Need, at a minimum, an underlying semantical referent (e.g. reading or math) Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Magnitude of Growth Are growth magnitudes possible in education? If calculable, are they interpretable? Approaches to growth magnitudes: Performance standards Vertical scale with interval properties Learning progressions (qualitative growth) Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Growth using Performance Standards Strengths Anchors reference points for discussions about performance Growth is embedded in accountability metric Common core standards will give uniform meaning across states (unlike today where standards vary greatly with stringency) Limitations Few levels, mask substantial range within levels thus masking student growth within level Currently standards vary greatly in stringency from state to state so that âĂIJproficientâĂİ performance lacks meaning Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Growth using Scale Scores Strengths Semi-continuous scores (many score points) Can be used to create vertical scales across grade levels Give the appearance of interval scales needed by some analytical models Limitations Difficult to interpret or explain to users Vertical scales are hard to defend Claims of interval measurement properties don’t hold to close scrutiny Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Vertical Scales Vertical & Interval scales required for some analytic techniques: Gain score calculation (magnitude of growth) Growth curve analysis (rate of growth) (e.g., Singer & Willett, Vertical & Interval scales required for some questions: Matthew effects: Do higher achievers grow faster than lower achievers? Growth rates relative to student age: Do students grow more in later grades than earlier grades? Invariance to monotone transformation of scale If a developmental/vertical scale exists for a sequence of assessments, the existing scale is a monotone transformation of that scale. Growth analyses that are invariant to monotone transformation of scale obviate the need for a vertical scale in many cases. Damian W. Betebenner Measuring Student Growth

The Big Picture Measurement/Psychometric Considerations References Career and College Readiness Before: Reach/Maintain proficiency. Now: Reach career/college readiness Challenge for consortia is to combine norm- and criterion-referenced growth. Criteria combined with norms provide a transparent and fair mechanism to communicate simultaneously: What is? What should be? What is reasonable? Damian W. Betebenner Measuring Student Growth

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

The Big Picture Measurement/Psychometric Considerations References References Singer, J. D. and Willett, J. B. (2003). Applied Longitudinal Data Analysis. Oxford University Press, New York. Damian W. Betebenner Measuring Student Growth

About this copy

On Consortia Approaches to Measuring Student Growth — AERA Annual Meeting, April 9, 2011.

This is an image-based copy of the original beamer slides; builds and animations are flattened.