Mind the Gap EXPLORING WAYS TO PRESENT GAPS & GAP TRENDS

Achievement Gaps • IMHO, the most abused and misunderstood of academic indicators. • With the advent of criterion-referenced testing, gaps in percent pro cient have become ubiquitous. • Used as part of almost all state accountability systems. • Not only gaps themselves but trends in gap statistics. Staff Sharing - 1/10/2019 2 fi

Academic Literature • “The technical quality of performance assessments: Standard error of percents of pupils reaching standards” by Wendy Yen in 1997. • “Stability of school-building accountability scores and gains” by Carolyn Haug and Robert Linn in 2002. • “Two Measures of Change in the Gaps Between the CDFs of Test-Score Distributions” by Paul Holland in 2002. • “The problem with “pro ciency”: Limitations of statistics and policy under No Child Left Behind” by Andrew Ho in 2008 • “Variability in Percentage Above Cut Scores Due to Discreteness in Score Scale” by Ying Lu in 2017 Staff Sharing - 1/10/2019 3 fi

Mountains & Mole Hills • My motivation for thinking about this came from the recent initiative led by Morgan Polikoff to move away from pro ciency based cuts, especially for looking at gaps. • Experts proposed looking at scale score differences. • http://www.npr.org/sections/ed/2016/07/18/486196906/here-s-an-idea-ask-experts- how-to-rate-schools • This is analogous to issue of whether to use percent above cut or mean for SGPs. • This is an issue regarding how to describe the difference in central tendency of two distributions (gaps). It’s mostly a minor technical issue. • A less technical interest relates to how to communicate gaps to be meaningful. • After rst declining this approach by states, feds seem to be amenable to it. Massachusetts included it in their ESSA application. Staff Sharing - 1/10/2019 4 fi fi

Scratch a Criterion and you find norm • It’s not clear (to me) that scale score based gap/gap-trend indicators are (much of) an improvement. • One loses the important pro ciency cut as a reference point in the gap. • Though pro ciency cuts are designed to make the scale interpretable. Even with pro ciency based cuts, it’s not clear that differences in percent pro cient are that interpretable. • Is a 20% gap between economically and non-economically disadvantaged students the same: 0%/20%, 40%/60%, and 80%/100%. • This takes me back to the same deliberations I went through when thinking about student growth and writing Norm- and criterion-referenced student growth. Staff Sharing - 1/10/2019 5 fi fi fi fi

Norm- and Criterion-Referenced Gap Metrics • Goal: Produce a metric-free distributional comparison for looking at gaps. • Goal: Communicate a 50th percentile student who is white lies at what percentile of the overall distribution. • Extend: Communicate an Xth percentile student who is of ethnicity Y lies at what percentile of the overall distribution. • Extend: Do this for other groups (e.g., (non)-economically disadvantaged) • Question: Would it be better to depict where an overall Xth percentile student like in each of the ethnicity speci c distributions? Problem (bigger numbers are worse). • Gaps a achievement level cuts can be depicted by locating cut graphically. Staff Sharing - 1/10/2019 6 fi

Distribution-Free Gap Metrics • Bene t: Gaps are more interpretable. If 50th percentile is used for gap, then one is looking at the difference in percentage of students below “grade level”. • Bene t: Gaps can defensibly be compared in different grades. E.g., Are achievement gaps larger in Grade 4 Math or Grade 8 Math? • Bene t: Gaps can defensibly be compared in different content areas. E.g., Are achievement gaps larger in Grade 7 ELA or Grade 7 Math? • Bene t: Gaps can defensibly compared in different states using different tests. E.g., Are achievement gaps in Grade 4 ELA larger in Colorado than Georgia Staff Sharing - 1/10/2019 7 fi fi fi fi

Gap Representation: Colorado Grade 4 Mathematics Gap Representation: Mathematics Grade 4, Colorado African American White Group Specific Achievement Percentile Mean Scale Score Difference = 24.3 Percent Proficient Difference = 26.7 Level 1 Level 2 Level 3 Level 4 Level 5 Overall Achievement Percentile Staff Sharing - 1/10/2019 8

Gap Representation: Colorado Grade 4 Mathematics Gap Representation: Mathematics Grade 8, Colorado African American White Group Specific Achievement Percentile Mean Scale Score Difference = 24.1 Percent Proficient Difference = 21 Level 1 Level 2 Level 3 Level 4 Level 5 Overall Achievement Percentile Staff Sharing - 1/10/2019 9

Gap Representation: Colorado Grade 4 Mathematics Gap Representation: ELA Grade 8, Colorado African American White Group Specific Achievement Percentile Mean Scale Score Difference = 22.9 Percent Proficient Difference = 24 Level 1 Level 2 Level 3 Level 4 Level 5 Overall Achievement Percentile Staff Sharing - 1/10/2019 10

Gap Representation: Colorado Grade 4 Mathematics Gap Representation: ELA Grade 8, Georgia African American White Group Specific Achievement Percentile Mean Scale Score Difference = 30.1 Percent Proficient Difference = 25.4 Level 1 Level 2 Level 3 Level 4 Overall Achievement Percentile Staff Sharing - 1/10/2019 11

Gap Representation: Colorado Grade 4 Mathematics Gap Representation: Math Grade 8, Georgia African American White Group Specific Achievement Percentile Mean Scale Score Difference = 34.5 34.4 Percent Proficient Difference = 30.9 Level 1 Level 2 Level 3 Level 4 Overall Achievement Percentile Staff Sharing - 1/10/2019 12

Questions/Observations • The differences between approaches aren’t dramatic. • Comparison of gaps across grades using percent pro cient are questionable based upon where the cut is placed (Andrew Ho). This would also apply to cross state comparisons. • Comparison of gaps across grades using scale scores requires some though. Cross state comparisons wouldn’t be possible. Staff Sharing - 1/10/2019 13 fi

Next Steps • This seems like a useful analysis/representation that any state could run to better understand gaps. • The current graphical representation suffers from the disadvantaged group being above the advantaged group. Probably needs to be reversed but not sure how. • Create an R package (e.g., CenterforAssessment_Tools) that contains data tools a state could use on its data based upon an assumed format provided. • Staff with analyses/diagnostics that states could run on their data are welcome to contribute and build out the package. • The package could be used with our work with states or used by a test vendor in conjunction with their data validation. Staff Sharing - 1/10/2019 14

For more information: Center for Assessment www.nciea.org

About this copy

Mind the Gap — Center for Assessment staff meeting, January 9, 2019.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.