A BRIEF HISTORY OF STUDENT GROWTH PROMISES FULFILLED & OPPORTUNITIES MISSED Damian W Betebenner National Center for the Improvement of Educational Assessment CCSSO/NCSA Orlando, FL June 25th, 2019 dbetebenner@nciea.org @dbetebenner

My Background • Began investigating large scale longitudinal analyses in 1999 as a graduate student at the University of Colorado, Boulder. • Investigated/replicated EVAAS and other VAMs, Markov-chain/ transition matrices, growth curve/HLM models in early 2000s. • Developed Student Growth Percentile (SGP) methodology in 2005-2007 while a professor at Boston College. • Joined the Center for Assessment in 2007 and over the last 12 years have worked with over 40 states and several consortia and testing companies on the calculation and use of student growth.

Promises Fulfilled • All states have investigated and nearly every state has adopted/ utilized an individual student growth model. • States and testing vendors are very interested in calculating student growth in almost all contexts in which they have longitudinal student achievement data. • There is MUCH greater understanding of student growth amongst state level personnel (far exceeding what academics know). State level personnel, in general, know what they are talking about. • States recognize that a vertical scale doesn’t solve all the growth issues and are taking advantage of both norm- and criterion- referenced growth. • States recognize and are taking advantage of both norm- and criterion-referenced growth.

Opportunities Missed (thus far) • Broadly, we are much better at using data as a hammer than as a flashlight. • Student growth has primarily been used as part of state accountability initiatives. For example, school and teacher evaluations. • Student growth has, in general, not been used to investigate: • Issues related to equity: For example, growth gaps between demographic subgroups. • Issues related to assessment bias: For example, test mode of administration effects. • Issues related to program evaluation: For example, investigating whether programs or curricula are associated with exemplary student growth. • Issues related to diagnosis of individual strengths/weaknesses. • Policy makers don’t understand the implications of what growth-to- standard analyses imply.

Growth Gaps • The analysis of student growth can enable powerful discussions about equal educational opportunity. • Even after accounting for where students start, students from disadvantaged subgroups demonstrate less academic growth than students from advantaged subgroups. • Median/mean SGPs for disadvantaged subgroups are almost always in the mid to low 40s where as advantaged groups are usually in mid 50s. • Using growth analyses to identify districts and schools with extreme growth gaps or reverse growth gaps supports powerful discussions about equity. • Conversely, including demographic adjustments in growth analyses themselves adjusts out these differences and sweeps the problem under the rug.

Assessment Bias • Regression based growth analyses (e.g. SGP) which utilize students’ prior scores act like a propensity score based analysis. • SGP analyses in the past have detected mode effects between paper and pencil and computer administered tests that appeared inconsequential when looked at using only current attainment. • SGP analyses have also detected substantial differences in the growth of subgroups when states have transitioned to different tests. • Beyond accountability, disparate growth results can quickly and easily highlight issues related to to the testing system itself — but it is rarely used in this fashion.

Program Evaluation • Again, regression based growth analyses (e.g. SGP) which utilize students’ prior scores act like a propensity score analysis. • States, districts, schools often have local knowledge about student group membership that can be used to quickly and easily investigate impact. • A school district in MA noticed unusually low student growth amongst 4th graders. This district had K-3 schools followed by a transition to 4-6. The transition was likely impacting student growth. • With growth results over multiple years, the impact of curricular changes can be investigated to see whether they lead to increased student growth.

Education is not the filling of a pail, but the lighting of a fire. William Butler Years

Gifted & Talented • As the definition suggests, gifted & talented is often associated with extraordinary levels of attainment. • Gifted in the common parlance refers to students whose pails are “full”. • How do we square this definition within a growth based view of the world. One that focuses on fire.

Explosive growth! Advanced Reading Move Up Achievement ●● Target DEMO Reading ● Scale Score Keep Up Catch Up ● Target Proficient Target ● ● Growth Level Percentiles ● Very High 81st − 99th High 61st − 80th Part Proficient Typical 40th − 60th Low 20th − 39th Very Low 1st − 19th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Scale Score 449 533 579 602 Achievement Level Achievement Target Unsatisfactory Part Proficient Part Proficient Proficient Proficient (592) Proficient (620)/Advanced (661) Achievement Growth Percentile 69 85 74 Growth Level Growth Target High Very High High Catch Up (60) Keep Up (76)/Move Up (99) Growth

Demonstration: 2018−2019 Mathematics Norm & Criterion Referenced Growth & Achievement College Ready Percentile Growth Trajectory to College Readiness Growth to Career/College 550 Scale Score 2018−2019 Achievement Percentile Readiness Not College Ready SGP trajectories for SGP trajectories for aa 3rd 3rd grade grade student student starting starting from from the the Level Level 1/Level 1/Level 22 cut. cut. Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Damian Betebenner dbetebenner@nciea.org @dbetebenner

About this copy

A Brief History of Student Growth — CCSSO National Conference on Student Assessment, June 25, 2019.

This is an image-based copy of the original Keynote slides; builds and animations are flattened.