Using Growth Norms to (in)Validate Years of Learning

NCME Annual Meeting · April 2024

Using Growth Norms to (in)Validate Years of Learning — NCME Annual Meeting, April 2024.
Published

April 11, 2024

Event
Conference
Slides
8
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Using Growth Norms to (in)Validate Years of Learning Damian Betebenner, Center for Assessment Charlie DePascale, Psychometric Confections NCME Annual Meeting Philadelphia, PA April 12, 2024
  2. “Time is an illusion” – Einstein Performance of fourth-grade students on the 2022 NAEP Reading test dropped 7 points from Educational measurement is 2018 to the lowest point it has been since arbitrary. Learning is illusive. Can we use the illusion to interpret the arbitrary so that we better understand the illusive? The 8th grade average score of 500 in 2023 is equal to the average score of 6th students in 2019. Four Perspectives • Semantics • Technical Foundations • Validation Students are two years behind, a half- • Utility year behind, six weeks behind…
  3. Semantics: Learning and Time • Learning ≠ Achievement • Learning is a rate: Learning = (Change in Achievement)/Time • Learning is not linear or uniform over time • Learning is inherently an individual activity • Time-based estimates are inherently focused on the group • Time is rarely the most important variable affecting learning • Time is out of favor as a relevant factor when discussing learning or achievement
  4. Technical Foundations Measurement is never better than the empirical operations by which it is carried out, and operations range from bad to good. – S.S. Stevens (1946) Student Vertical Time- Theta Scaled Standard Effect Test Respons Scale Based Score Score Deviation Size e Score Estimate Even accepting that each individual operation along the way is good, the conceptual and empirical distance travelled is simply too great to support technically sound, interpretable, useful time- based reporting of student test scores.
  5. Validation (or lack thereof) • We’ve travelled this road before • At their core, there is little difference between today’s time-based estimates and the grade equivalent scores soundly rejected by the measurement field and educators in the NRT era
  6. Utility • I used to say, "it is better to be complicated than wrong.” But recently I've relaxed this perspective when engaging with the public about measurement. I'd rather people be interested and learn more than tune out entirely. This comes with risks. I'll accept them. – Andrew Ho (Sept 1, 2022)
  8. Utility – Problems with Time-based Reporting 1. Time-based reporting is devoid of content. 2. Time-based reporting leads stakeholders toward time-based solutions. 3. The aggregate-level results which are at the core of time-based reporting are not representative of what is taking place at the individual level.