MIND THE GAP:
FINDING A BETTER WAY TO PRESENT GAPS & GAP TRENDS
Damian W Betebenner
National Center for the Improvement of Educational Assessment
Dover, New Hampshire October 3rd, 2017
ACHIEVEMENT GAPS
• IMHO, the most abused and misunderstood of academic
indicators.
• With the advent of criterion-referenced testing, gaps in
percent pro cient have become ubiquitous.
• Used as part of almost all state accountability systems.
• Not only gaps themselves but trends in gap statistics.
fi
ACADEMIC LITERATURE
• “The technical quality of performance assessments: Standard error of percents
of pupils reaching standards” by Wendy Yen in 1997.
• “Stability of school-building accountability scores and gains” by Carolyn Haug
and Robert Linn in 2002.
• “Two Measures of Change in the Gaps Between the CDFs of Test-Score
Distributions” by Paul Holland in 2002.
• “The problem with “pro ciency”: Limitations of statistics and policy under No
Child Left Behind” by Andrew Ho in 2008
• “Variability in Percentage Above Cut Scores Due to Discreteness in Score
Scale” by Ying Lu in 2017
fi
MOUNTAINS AND MOLEHILLS
• My motivation for thinking about this came from the recent
initiative led by Morgan Polikoff to move away from
pro ciency based cuts, especially for looking at gaps.
• http://www.npr.org/sections/ed/2016/07/18/486196906/here-
s-an-idea-ask-experts-how-to-rate-schools
• Analogous to issue of whether to use percent above cut or
mean for SGPs. An issue, yes, but NOT a big one.
• After rst declining this approach by states, feds seem to be
amenable to it.
fi
fi
SCRATCH A CRITERION AND YOU FIND A
NORM
• It’s not clear (to me) that scale score based gap/gap-trend indicators are (much
of) an improvement.
• One loses the important pro ciency cut as a reference point in the gap.
• Even with pro ciency based cuts, it’s not clear that they are that interpretable.
• Is a 20% gap between economically and non-economically disadvantaged
students the same: 0%/20%, 40%/60%, and 80%/100%.
• This takes me back to the same deliberations I went through when thinking
about student growth and writing Norm- and criterion-referenced student growth.
fi
fi
Norm-referenced achievement gaps
Overall Achievement Quantile
White
Equality
African American
Ethnicity Achievement Percentile
NORM- AND CRITERION-REFERENCED GAP METRICS
• Goal: Produce a metric-free distributional comparison for looking at gaps.
• Goal: Communicate a 50th percentile student who is white lies at what
percentile of the overall distribution.
• Extend: Communicate an Xth percentile student who is of ethnicity Y lies at
what percentile of the overall distribution.
• Extend: Do this for other groups (e.g., (non)-economically disadvantaged)
• Question: Would it be better to depict where an overall Xth percentile student
like in each of the ethnicity speci c distributions? Problem (bigger numbers are
worse).
• Gaps a achievement level cuts can be depicted by locating cut graphically.
fi
CONCLUSIONS & NEXT STEPS
• Gap and gap trends summarizations at a single point of the distribution
necessarily lead to information loss.
• Lots of simulation work to be done regarding comparisons of different a gap
metrics.
• Possibly apply to NAEP data.
• Not sure whether applicable for accountability purposes.
• Creating an R package and associated GitHub repo (called Accountability) for
analytics and documentation around this work (if others want to collaborate
About this copy
Mind the Gap — Center for Assessment staff meeting, 2019.
This is an image-based copy of the original Keynote slides; builds and animations are flattened.