Mind the Gap

Center for Assessment staff meeting · 2019

Mind the Gap — Center for Assessment staff meeting, 2019.
Published

January 1, 2019

Event
Staff session
Slides
9
Format
Converted from Keynote

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. MIND THE GAP: FINDING A BETTER WAY TO PRESENT GAPS & GAP TRENDS Damian W Betebenner National Center for the Improvement of Educational Assessment Dover, New Hampshire October 3rd, 2017
  2. ACHIEVEMENT GAPS • IMHO, the most abused and misunderstood of academic indicators. • With the advent of criterion-referenced testing, gaps in percent pro cient have become ubiquitous. • Used as part of almost all state accountability systems. • Not only gaps themselves but trends in gap statistics. fi
  3. ACADEMIC LITERATURE • “The technical quality of performance assessments: Standard error of percents of pupils reaching standards” by Wendy Yen in 1997. • “Stability of school-building accountability scores and gains” by Carolyn Haug and Robert Linn in 2002. • “Two Measures of Change in the Gaps Between the CDFs of Test-Score Distributions” by Paul Holland in 2002. • “The problem with “pro ciency”: Limitations of statistics and policy under No Child Left Behind” by Andrew Ho in 2008 • “Variability in Percentage Above Cut Scores Due to Discreteness in Score Scale” by Ying Lu in 2017 fi
  4. MOUNTAINS AND MOLEHILLS • My motivation for thinking about this came from the recent initiative led by Morgan Polikoff to move away from pro ciency based cuts, especially for looking at gaps. • http://www.npr.org/sections/ed/2016/07/18/486196906/here- s-an-idea-ask-experts-how-to-rate-schools • Analogous to issue of whether to use percent above cut or mean for SGPs. An issue, yes, but NOT a big one. • After rst declining this approach by states, feds seem to be amenable to it. fi fi
  5. SCRATCH A CRITERION AND YOU FIND A NORM • It’s not clear (to me) that scale score based gap/gap-trend indicators are (much of) an improvement. • One loses the important pro ciency cut as a reference point in the gap. • Even with pro ciency based cuts, it’s not clear that they are that interpretable. • Is a 20% gap between economically and non-economically disadvantaged students the same: 0%/20%, 40%/60%, and 80%/100%. • This takes me back to the same deliberations I went through when thinking about student growth and writing Norm- and criterion-referenced student growth. fi fi
  6. Norm-referenced achievement gaps Overall Achievement Quantile White Equality African American Ethnicity Achievement Percentile
  7. NORM- AND CRITERION-REFERENCED GAP METRICS • Goal: Produce a metric-free distributional comparison for looking at gaps. • Goal: Communicate a 50th percentile student who is white lies at what percentile of the overall distribution. • Extend: Communicate an Xth percentile student who is of ethnicity Y lies at what percentile of the overall distribution. • Extend: Do this for other groups (e.g., (non)-economically disadvantaged) • Question: Would it be better to depict where an overall Xth percentile student like in each of the ethnicity speci c distributions? Problem (bigger numbers are worse). • Gaps a achievement level cuts can be depicted by locating cut graphically. fi
  8. CONCLUSIONS & NEXT STEPS • Gap and gap trends summarizations at a single point of the distribution necessarily lead to information loss. • Lots of simulation work to be done regarding comparisons of different a gap metrics. • Possibly apply to NAEP data. • Not sure whether applicable for accountability purposes. • Creating an R package and associated GitHub repo (called Accountability) for analytics and documentation around this work (if others want to collaborate